LinuxLists.cc - possible bug in nfs-kernel-server

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

On Fri, Nov 20, 2015 at 12:04:49PM +0100, Omar Walid Llorente wrote:
>
> Hi, I'm Omar Walid Llorente and I am a systems administrator at the
> Politechnical University of Madrid (UPM), Spain. I write you in the
> hope you can help us manage a problem that have discovered recently
> about our new datastore architecture in our teaching labs. We have
> created a gluster distributed volume that we reexport with NFS to
> our lab clients via intermediate servers.
>
> First of all thanks for all your work and sorry if this isn't
> related with your package, but I think it has a good chance. I'll
> try explain myself as short as possible.
>
> As introduced previously, we have a problem exporting with
> nfs-kernel-server-1.2.8-6 (ubuntu based) a directory previously
> mounted with gluster-3.7.4 via fuse mount.
>
> The problem is quite simple to reproduce and always repeatable: if a
> file has read-only permissions for owner and user wants to copy it,
> permissions problem arises:
> cdc@client:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod
> 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
> cp: failed to close ‘kk.txt’: Permission denied
> -r--r--r-- 1 cdc admincdc 7 nov 3 2015 444.txt
> -r--r--r-- 1 cdc admincdc 0 nov 3 2015 kk.txt
> cdc@client:~$
>
> If the file permissions are not read-only, there is no problem:
> cdc@client:~$ rm -f kk.txt 644.txt; echo "prueba" > 644.txt; chmod
> 644 644.txt; cp -p 644.txt kk.txt; ls -ld 644.txt kk.txt
> -rw-r--r-- 1 cdc admincdc 7 nov 3 2015 644.txt
> -rw-r--r-- 1 cdc admincdc 7 nov 3 2015 kk.txt
> cdc@client:~$
>
> If we track it down with strace, the problem arises exactly when
> fsync() is called from cp.

So, cp is calling fsync on kk.txt before closing it?

> Of course, if we try this combination of commands in other
> directories not mounted by nfs (local ones) or mounted with
> samba/cifs or even mounted with nfs-ganesha (both fuse mounted with
> gluster), this doen't happen. This problem doesn't happen either if
> the nfs-kernel-server exports a directory not mounted with fuse (any
> local one).
>
> Please, tell me if this is the right place to post the probem and
> where is it if this is not. Let me know if we can help you any way
> to solve or test it (we've developed a small program in c that shows
> exactly the same behaviour).

It'd be interesting to see exactly which NFS operation is failing. You
should be able to do this by running wireshark to sniff the NFS
client/server traffic while running the reproducer, then reading through
the traffic to find the failing operation.

I'm also curious:

> LOGS ON SERVER SIDE (glusterfs mount logs):
> [2015-11-20 10:51:53.872656] I [io-stats.c:1014:io_stats_dump_fd]
> 0-home-lab-3: --- fd stats ---
> [2015-11-20 10:51:53.872692] I [io-stats.c:1019:io_stats_dump_fd]
> 0-home-lab-3: Filename : /cdc/444.txt
> [2015-11-20 10:51:53.872704] I [io-stats.c:1034:io_stats_dump_fd]
> 0-home-lab-3: BytesWritten : 7 bytes
> [2015-11-20 10:51:53.872714] I [io-stats.c:1046:io_stats_dump_fd]
> 0-home-lab-3: Write 000004b+ : 1
> [2015-11-20 10:51:53.874917] W [MSGID: 114031]
> [client-rpc-fops.c:1298:client3_3_removexattr_cbk]
> 0-home-lab-3-client-0: remote operation failed [Permission denied]
> [2015-11-20 10:51:53.874976] W [fuse-bridge.c:1230:fuse_err_cbk]
> 0-glusterfs-fuse: 63459954: REMOVEXATTR() /cdc/444.txt => -1
> (Permission denied)
> [2015-11-20 10:51:53.881389] W [MSGID: 114031]
> [client-rpc-fops.c:1298:client3_3_removexattr_cbk]
> 0-home-lab-3-client-3: remote operation failed [Permission denied]
> [2015-11-20 10:51:53.881434] W [fuse-bridge.c:1230:fuse_err_cbk]
> 0-glusterfs-fuse: 63459961: REMOVEXATTR() /cdc/kk.txt => -1
> (Permission denied)

I wonder what this is about? Why would an fsync on the NFS client
result in a REMOVEXATTR on the server side?

--b.

> [2015-11-20 10:51:53.883072] W [fuse-bridge.c:1230:fuse_err_cbk]
> 0-glusterfs-fuse: 63459964: REMOVEXATTR() /cdc/kk.txt => -1
> (Permission denied)
> [2015-11-20 10:51:53.883057] W [MSGID: 114031]
> [client-rpc-fops.c:1298:client3_3_removexattr_cbk]
> 0-home-lab-3-client-3: remote operation failed [Permission denied]
> [2015-11-20 10:51:53.884003] E [MSGID: 114031]
> [client-rpc-fops.c:466:client3_3_open_cbk] 0-home-lab-3-client-3:
> remote operation failed. Path: /cdc/kk.txt
> (3175e0cd-8308-45b8-a4b0-699f6f8cf37f) [Permission denied]
> [2015-11-20 10:51:53.884056] W [fuse-bridge.c:969:fuse_fd_cbk]
> 0-glusterfs-fuse: 63459965: OPEN() /cdc/kk.txt => -1 (Permission
> denied)
> [2015-11-20 10:51:53.885619] W [MSGID: 114031]
> [client-rpc-fops.c:1298:client3_3_removexattr_cbk]
> 0-home-lab-3-client-3: remote operation failed [Permission denied]
> [2015-11-20 10:51:53.885664] W [fuse-bridge.c:1230:fuse_err_cbk]
> 0-glusterfs-fuse: 63459967: REMOVEXATTR() /cdc/kk.txt => -1
> (Permission denied)
> [2015-11-20 10:51:53.887908] W [fuse-bridge.c:1230:fuse_err_cbk]
> 0-glusterfs-fuse: 63459971: REMOVEXATTR() /cdc/kk.txt => -1
> (Permission denied)
> [2015-11-20 10:51:53.887891] W [MSGID: 114031]
> [client-rpc-fops.c:1298:client3_3_removexattr_cbk]
> 0-home-lab-3-client-3: remote operation failed [Permission denied]
>
> (NOTE: We have more gluster brick logs but we don't know if are relevant)
>
> --
> ----------------------------------------------------------------
> Centro de Cálculo Depto. Ingeniería Sistemas Telemáticos
> E-mail: [email protected] Universidad Politécnica de Madrid
> Fax:(+34) 913367333 E.T.S. Ing. Telecomunicación
> Tel:(+34) 915495700-Ext.3005 28040 Madrid (Spain)
> Tel:(+34) 915495762-Ext.3005
> Tel:(+34) 913367366-Ext.3005
> ----------------------------------------------------------------
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html

2015-11-25 13:50:55

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

Hi, Jeff, thanks for the answer. I'm out of the office until next week,
but when I come back, I'll try to do the tests and send you the info.

Thank you very much,

Omar

El 2015-11-21 14:18, Jeff Layton escribió:
> On Fri, 20 Nov 2015 12:04:49 +0100
> Omar Walid Llorente <[email protected]> wrote:
>
>>
>> Hi, I'm Omar Walid Llorente and I am a systems administrator at the
>> Politechnical University of Madrid (UPM), Spain. I write you in the
>> hope
>> you can help us manage a problem that have discovered recently about
>> our
>> new datastore architecture in our teaching labs. We have created a
>> gluster distributed volume that we reexport with NFS to our lab
>> clients
>> via intermediate servers.
>>
>> First of all thanks for all your work and sorry if this isn't
>> related
>> with your package, but I think it has a good chance. I'll try
>> explain
>> myself as short as possible.
>>
>> As introduced previously, we have a problem exporting with
>> nfs-kernel-server-1.2.8-6 (ubuntu based) a directory previously
>> mounted
>> with gluster-3.7.4 via fuse mount.
>>
>
> What's important here (for the nfs server) is the kernel version.
> What
> kernel version are you running on the server? Also, what NFS version
> is
> the client using? If you grab the mount's line out of /proc/mounts on
> the client then that would be helpful.
>
> Also, does the NFS version matter here? If you're using NFSv4 then
> maybe try with NFSv3, or with v4 or so if you're already using v3?
>
>> The problem is quite simple to reproduce and always repeatable: if a
>> file has read-only permissions for owner and user wants to copy it,
>> permissions problem arises:
>> cdc@client:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod
>> 444
>> 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
>> cp: failed to close ‘kk.txt’: Permission denied
>> -r--r--r-- 1 cdc admincdc 7 nov 3 2015 444.txt
>> -r--r--r-- 1 cdc admincdc 0 nov 3 2015 kk.txt
>> cdc@client:~$
>>
>> If the file permissions are not read-only, there is no problem:
>> cdc@client:~$ rm -f kk.txt 644.txt; echo "prueba" > 644.txt; chmod
>> 644
>> 644.txt; cp -p 644.txt kk.txt; ls -ld 644.txt kk.txt
>> -rw-r--r-- 1 cdc admincdc 7 nov 3 2015 644.txt
>> -rw-r--r-- 1 cdc admincdc 7 nov 3 2015 kk.txt
>> cdc@client:~$
>>
>> If we track it down with strace, the problem arises exactly when
>> fsync()
>> is called from cp.
>>
>> Of course, if we try this combination of commands in other
>> directories
>> not mounted by nfs (local ones) or mounted with samba/cifs or even
>> mounted with nfs-ganesha (both fuse mounted with gluster), this
>> doen't
>> happen. This problem doesn't happen either if the nfs-kernel-server
>> exports a directory not mounted with fuse (any local one).
>>
>
> Ok, that's good info, but when dealing with a problem like this, it'd
> be best to get a capture of the network traffic between client and
> server while you're reproducing this. We can then look at it to
> figure
> out which RPC call is getting the actual error. That will help narrow
> down the problem a bit more.
>
> You can do that with tcpdump. Something like this should do it:
>
> # tcpdump -i eth0 -w /tmp/nfs.pcap -s 512 port 2049
>
> ...reproduce the problem and then stop the capture. Then you can
> open /tmp/nfs.pcap with wireshark to analyze it (or send it to me and
> I'll take a look).
>
>> Please, tell me if this is the right place to post the probem and
>> where is it if this is not. Let me know if we can help you any way
>> to
>> solve or test it (we've developed a small program in c that shows
>> exactly the same behaviour).
>>
>> Thanks again.
>>
>> Omar
>>
>> PS: Pointer to this email address came from:
>> http://wiki.linux-nfs.org/wiki/index.php/Reporting_bugs
>>
>> ADDITIONAL INFO:
>>
>> cdc@client:~$ uname -a
>> Linux l056 3.13.0-63-generic #103-Ubuntu SMP Fri Aug 14 21:43:30 UTC
>> 2015 i686 i686 i686 GNU/Linux
>> cdc@client:~$
>> cdc@client:~$ mount | grep home
>> cuentas02:/home-3/cdc on /home/cdc type nfs
>>
>> (rw,noatime,intr,fsc,nolock,rsize=262140,wsize=262140,addr=138.4.30.15)
>> cdc@client:~$
>>
>> root@server-lab:~# uname -a
>> Linux cuentas02-lab.lab.dit.upm.es 3.13.0-63-generic #103-Ubuntu SMP
>> Fri Aug 14 21:42:59 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>> root@server-lab:~#
>> root@server-lab:~# dpkg -l | grep nfs
>> ii libnfsidmap2:amd64 0.25-5 amd64
>> NFS idmapping library
>> ii nfs-common 1:1.2.8-6ubuntu1.1 amd64 NFS
>> support files common to client and server
>> ii nfs-kernel-server 1:1.2.8-6ubuntu1.1 amd64
>> support for NFS kernel server
>> root@server-lab:~#
>> root@server-lab:~# exportfs -v
>> /home-3
>>
>> 138.4.30.0/23(rw,async,wdelay,insecure,no_root_squash,no_subtree_check,fsid=3,sec=sys,rw,no_root_squash,no_all_squash)
>> root@server-lab:~#
>>
>> LOGS ON SERVER SIDE (glusterfs mount logs):
>> [2015-11-20 10:51:53.872656] I [io-stats.c:1014:io_stats_dump_fd]
>> 0-home-lab-3: --- fd stats ---
>> [2015-11-20 10:51:53.872692] I [io-stats.c:1019:io_stats_dump_fd]
>> 0-home-lab-3: Filename : /cdc/444.txt
>> [2015-11-20 10:51:53.872704] I [io-stats.c:1034:io_stats_dump_fd]
>> 0-home-lab-3: BytesWritten : 7 bytes
>> [2015-11-20 10:51:53.872714] I [io-stats.c:1046:io_stats_dump_fd]
>> 0-home-lab-3: Write 000004b+ : 1
>> [2015-11-20 10:51:53.874917] W [MSGID: 114031]
>> [client-rpc-fops.c:1298:client3_3_removexattr_cbk]
>> 0-home-lab-3-client-0: remote operation failed [Permission denied]
>> [2015-11-20 10:51:53.874976] W [fuse-bridge.c:1230:fuse_err_cbk]
>> 0-glusterfs-fuse: 63459954: REMOVEXATTR() /cdc/444.txt => -1
>> (Permission denied)
>> [2015-11-20 10:51:53.881389] W [MSGID: 114031]
>> [client-rpc-fops.c:1298:client3_3_removexattr_cbk]
>> 0-home-lab-3-client-3: remote operation failed [Permission denied]
>> [2015-11-20 10:51:53.881434] W [fuse-bridge.c:1230:fuse_err_cbk]
>> 0-glusterfs-fuse: 63459961: REMOVEXATTR() /cdc/kk.txt => -1
>> (Permission denied)
>> [2015-11-20 10:51:53.883072] W [fuse-bridge.c:1230:fuse_err_cbk]
>> 0-glusterfs-fuse: 63459964: REMOVEXATTR() /cdc/kk.txt => -1
>> (Permission denied)
>> [2015-11-20 10:51:53.883057] W [MSGID: 114031]
>> [client-rpc-fops.c:1298:client3_3_removexattr_cbk]
>> 0-home-lab-3-client-3: remote operation failed [Permission denied]
>> [2015-11-20 10:51:53.884003] E [MSGID: 114031]
>> [client-rpc-fops.c:466:client3_3_open_cbk] 0-home-lab-3-client-3:
>> remote operation failed. Path: /cdc/kk.txt
>> (3175e0cd-8308-45b8-a4b0-699f6f8cf37f) [Permission denied]
>> [2015-11-20 10:51:53.884056] W [fuse-bridge.c:969:fuse_fd_cbk]
>> 0-glusterfs-fuse: 63459965: OPEN() /cdc/kk.txt => -1 (Permission
>> denied)
>
> The above message is interesting and might be related to the problem.
> That said, we generally set the NFSD_MAY_OWNER_OVERRIDE bit on opens
> of
> regular files, which allows the nfsd_permission check to pass
> regardless when the owner matches.
>
> My guess would be that the dentry_open call in nfsd_open is failing
> here as the concept of "owner override" doesn't really get passed
> down
> to it. Still, it'd be good to confirm that...
>
>> [2015-11-20 10:51:53.885619] W [MSGID: 114031]
>> [client-rpc-fops.c:1298:client3_3_removexattr_cbk]
>> 0-home-lab-3-client-3: remote operation failed [Permission denied]
>> [2015-11-20 10:51:53.885664] W [fuse-bridge.c:1230:fuse_err_cbk]
>> 0-glusterfs-fuse: 63459967: REMOVEXATTR() /cdc/kk.txt => -1
>> (Permission
>> denied)
>> [2015-11-20 10:51:53.887908] W [fuse-bridge.c:1230:fuse_err_cbk]
>> 0-glusterfs-fuse: 63459971: REMOVEXATTR() /cdc/kk.txt => -1
>> (Permission
>> denied)
>> [2015-11-20 10:51:53.887891] W [MSGID: 114031]
>> [client-rpc-fops.c:1298:client3_3_removexattr_cbk]
>> 0-home-lab-3-client-3: remote operation failed [Permission denied]
>>
>> (NOTE: We have more gluster brick logs but we don't know if are
>> relevant)
>>

2015-11-25 16:24:00

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

Hi, thanks for the answer. I'm out of the office until next week, but
when I come back, I'll try to do the tests and send you the info.

Thank you very much,

Omar

El 2015-11-23 21:18, [email protected] escribió:
> On Fri, Nov 20, 2015 at 12:04:49PM +0100, Omar Walid Llorente wrote:
>>
>> Hi, I'm Omar Walid Llorente and I am a systems administrator at the
>> Politechnical University of Madrid (UPM), Spain. I write you in the
>> hope you can help us manage a problem that have discovered recently
>> about our new datastore architecture in our teaching labs. We have
>> created a gluster distributed volume that we reexport with NFS to
>> our lab clients via intermediate servers.
>>
>> First of all thanks for all your work and sorry if this isn't
>> related with your package, but I think it has a good chance. I'll
>> try explain myself as short as possible.
>>
>> As introduced previously, we have a problem exporting with
>> nfs-kernel-server-1.2.8-6 (ubuntu based) a directory previously
>> mounted with gluster-3.7.4 via fuse mount.
>>
>> The problem is quite simple to reproduce and always repeatable: if a
>> file has read-only permissions for owner and user wants to copy it,
>> permissions problem arises:
>> cdc@client:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod
>> 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
>> cp: failed to close ‘kk.txt’: Permission denied
>> -r--r--r-- 1 cdc admincdc 7 nov 3 2015 444.txt
>> -r--r--r-- 1 cdc admincdc 0 nov 3 2015 kk.txt
>> cdc@client:~$
>>
>> If the file permissions are not read-only, there is no problem:
>> cdc@client:~$ rm -f kk.txt 644.txt; echo "prueba" > 644.txt; chmod
>> 644 644.txt; cp -p 644.txt kk.txt; ls -ld 644.txt kk.txt
>> -rw-r--r-- 1 cdc admincdc 7 nov 3 2015 644.txt
>> -rw-r--r-- 1 cdc admincdc 7 nov 3 2015 kk.txt
>> cdc@client:~$
>>
>> If we track it down with strace, the problem arises exactly when
>> fsync() is called from cp.
>
> So, cp is calling fsync on kk.txt before closing it?
>
>> Of course, if we try this combination of commands in other
>> directories not mounted by nfs (local ones) or mounted with
>> samba/cifs or even mounted with nfs-ganesha (both fuse mounted with
>> gluster), this doen't happen. This problem doesn't happen either if
>> the nfs-kernel-server exports a directory not mounted with fuse (any
>> local one).
>>
>> Please, tell me if this is the right place to post the probem and
>> where is it if this is not. Let me know if we can help you any way
>> to solve or test it (we've developed a small program in c that shows
>> exactly the same behaviour).
>
> It'd be interesting to see exactly which NFS operation is failing.
> You
> should be able to do this by running wireshark to sniff the NFS
> client/server traffic while running the reproducer, then reading
> through
> the traffic to find the failing operation.
>
> I'm also curious:
>
>> LOGS ON SERVER SIDE (glusterfs mount logs):
>> [2015-11-20 10:51:53.872656] I [io-stats.c:1014:io_stats_dump_fd]
>> 0-home-lab-3: --- fd stats ---
>> [2015-11-20 10:51:53.872692] I [io-stats.c:1019:io_stats_dump_fd]
>> 0-home-lab-3: Filename : /cdc/444.txt
>> [2015-11-20 10:51:53.872704] I [io-stats.c:1034:io_stats_dump_fd]
>> 0-home-lab-3: BytesWritten : 7 bytes
>> [2015-11-20 10:51:53.872714] I [io-stats.c:1046:io_stats_dump_fd]
>> 0-home-lab-3: Write 000004b+ : 1
>> [2015-11-20 10:51:53.874917] W [MSGID: 114031]
>> [client-rpc-fops.c:1298:client3_3_removexattr_cbk]
>> 0-home-lab-3-client-0: remote operation failed [Permission denied]
>> [2015-11-20 10:51:53.874976] W [fuse-bridge.c:1230:fuse_err_cbk]
>> 0-glusterfs-fuse: 63459954: REMOVEXATTR() /cdc/444.txt => -1
>> (Permission denied)
>> [2015-11-20 10:51:53.881389] W [MSGID: 114031]
>> [client-rpc-fops.c:1298:client3_3_removexattr_cbk]
>> 0-home-lab-3-client-3: remote operation failed [Permission denied]
>> [2015-11-20 10:51:53.881434] W [fuse-bridge.c:1230:fuse_err_cbk]
>> 0-glusterfs-fuse: 63459961: REMOVEXATTR() /cdc/kk.txt => -1
>> (Permission denied)
>
> I wonder what this is about? Why would an fsync on the NFS client
> result in a REMOVEXATTR on the server side?
>
> --b.
>
>> [2015-11-20 10:51:53.883072] W [fuse-bridge.c:1230:fuse_err_cbk]
>> 0-glusterfs-fuse: 63459964: REMOVEXATTR() /cdc/kk.txt => -1
>> (Permission denied)
>> [2015-11-20 10:51:53.883057] W [MSGID: 114031]
>> [client-rpc-fops.c:1298:client3_3_removexattr_cbk]
>> 0-home-lab-3-client-3: remote operation failed [Permission denied]
>> [2015-11-20 10:51:53.884003] E [MSGID: 114031]
>> [client-rpc-fops.c:466:client3_3_open_cbk] 0-home-lab-3-client-3:
>> remote operation failed. Path: /cdc/kk.txt
>> (3175e0cd-8308-45b8-a4b0-699f6f8cf37f) [Permission denied]
>> [2015-11-20 10:51:53.884056] W [fuse-bridge.c:969:fuse_fd_cbk]
>> 0-glusterfs-fuse: 63459965: OPEN() /cdc/kk.txt => -1 (Permission
>> denied)
>> [2015-11-20 10:51:53.885619] W [MSGID: 114031]
>> [client-rpc-fops.c:1298:client3_3_removexattr_cbk]
>> 0-home-lab-3-client-3: remote operation failed [Permission denied]
>> [2015-11-20 10:51:53.885664] W [fuse-bridge.c:1230:fuse_err_cbk]
>> 0-glusterfs-fuse: 63459967: REMOVEXATTR() /cdc/kk.txt => -1
>> (Permission denied)
>> [2015-11-20 10:51:53.887908] W [fuse-bridge.c:1230:fuse_err_cbk]
>> 0-glusterfs-fuse: 63459971: REMOVEXATTR() /cdc/kk.txt => -1
>> (Permission denied)
>> [2015-11-20 10:51:53.887891] W [MSGID: 114031]
>> [client-rpc-fops.c:1298:client3_3_removexattr_cbk]
>> 0-home-lab-3-client-3: remote operation failed [Permission denied]
>>
>> (NOTE: We have more gluster brick logs but we don't know if are
>> relevant)
>>
>> --
>> ----------------------------------------------------------------
>> Centro de Cálculo Depto. Ingeniería Sistemas Telemáticos
>> E-mail: [email protected] Universidad Politécnica de Madrid
>> Fax:(+34) 913367333 E.T.S. Ing. Telecomunicación
>> Tel:(+34) 915495700-Ext.3005 28040 Madrid (Spain)
>> Tel:(+34) 915495762-Ext.3005
>> Tel:(+34) 913367366-Ext.3005
>> ----------------------------------------------------------------
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs"
>> in
>> the body of a message to [email protected]
>> More majordomo info at http://vger.kernel.org/majordomo-info.html

2015-12-10 10:32:59

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs
packets (you can find them in attached file nfs-problem-nks.pcap.zip).
Sorry for being so late.

What I did was the following:

1st) Create the RO file:
cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt;

2nd) Init the capture:
root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
512 bytes

3rd) Try to copy the RO file and get the error:
cdc@l056:~/prueba-git$ cp -p 444.txt kk.txt;
cp: failed to close ‘kk.txt’: Permission denied
cdc@l056:~/prueba-git$

4th) Close the capture:
^C26 packets captured
26 packets received by filter
0 packets dropped by kernel
root@l056:~#

Hope you can send us some clue about it. Do you need me to do any other
test? Thanks in advance!

Omar

El 25/11/15 a las 14:50, omar escribió:
>
>
> Hi, Jeff, thanks for the answer. I'm out of the office until next
> week, but when I come back, I'll try to do the tests and send you the
> info.
>
> Thank you very much,
>
> Omar
>
> El 2015-11-21 14:18, Jeff Layton escribió:
>> On Fri, 20 Nov 2015 12:04:49 +0100
>> Omar Walid Llorente <[email protected]> wrote:
>>
>>>
>>> Hi, I'm Omar Walid Llorente and I am a systems administrator at the
>>> Politechnical University of Madrid (UPM), Spain. I write you in the
>>> hope
>>> you can help us manage a problem that have discovered recently about
>>> our
>>> new datastore architecture in our teaching labs. We have created a
>>> gluster distributed volume that we reexport with NFS to our lab clients
>>> via intermediate servers.
>>>
>>> First of all thanks for all your work and sorry if this isn't related
>>> with your package, but I think it has a good chance. I'll try explain
>>> myself as short as possible.
>>>
>>> As introduced previously, we have a problem exporting with
>>> nfs-kernel-server-1.2.8-6 (ubuntu based) a directory previously mounted
>>> with gluster-3.7.4 via fuse mount.
>>>
>>
>> What's important here (for the nfs server) is the kernel version. What
>> kernel version are you running on the server? Also, what NFS version is
>> the client using? If you grab the mount's line out of /proc/mounts on
>> the client then that would be helpful.
>>
>> Also, does the NFS version matter here? If you're using NFSv4 then
>> maybe try with NFSv3, or with v4 or so if you're already using v3?
>>
>>> The problem is quite simple to reproduce and always repeatable: if a
>>> file has read-only permissions for owner and user wants to copy it,
>>> permissions problem arises:
>>> cdc@client:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444
>>> 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
>>> cp: failed to close ‘kk.txt’: Permission denied
>>> -r--r--r-- 1 cdc admincdc 7 nov 3 2015 444.txt
>>> -r--r--r-- 1 cdc admincdc 0 nov 3 2015 kk.txt
>>> cdc@client:~$
>>>
>>> If the file permissions are not read-only, there is no problem:
>>> cdc@client:~$ rm -f kk.txt 644.txt; echo "prueba" > 644.txt; chmod 644
>>> 644.txt; cp -p 644.txt kk.txt; ls -ld 644.txt kk.txt
>>> -rw-r--r-- 1 cdc admincdc 7 nov 3 2015 644.txt
>>> -rw-r--r-- 1 cdc admincdc 7 nov 3 2015 kk.txt
>>> cdc@client:~$
>>>
>>> If we track it down with strace, the problem arises exactly when
>>> fsync()
>>> is called from cp.
>>>
>>> Of course, if we try this combination of commands in other directories
>>> not mounted by nfs (local ones) or mounted with samba/cifs or even
>>> mounted with nfs-ganesha (both fuse mounted with gluster), this doen't
>>> happen. This problem doesn't happen either if the nfs-kernel-server
>>> exports a directory not mounted with fuse (any local one).
>>>
>>
>> Ok, that's good info, but when dealing with a problem like this, it'd
>> be best to get a capture of the network traffic between client and
>> server while you're reproducing this. We can then look at it to figure
>> out which RPC call is getting the actual error. That will help narrow
>> down the problem a bit more.
>>
>> You can do that with tcpdump. Something like this should do it:
>>
>> # tcpdump -i eth0 -w /tmp/nfs.pcap -s 512 port 2049
>>
>> ...reproduce the problem and then stop the capture. Then you can
>> open /tmp/nfs.pcap with wireshark to analyze it (or send it to me and
>> I'll take a look).
>>
>>> Please, tell me if this is the right place to post the probem and
>>> where is it if this is not. Let me know if we can help you any way to
>>> solve or test it (we've developed a small program in c that shows
>>> exactly the same behaviour).
>>>
>>> Thanks again.
>>>
>>> Omar
>>>
>>> PS: Pointer to this email address came from:
>>> http://wiki.linux-nfs.org/wiki/index.php/Reporting_bugs
>>>
>>> ADDITIONAL INFO:
>>>
>>> cdc@client:~$ uname -a
>>> Linux l056 3.13.0-63-generic #103-Ubuntu SMP Fri Aug 14 21:43:30 UTC
>>> 2015 i686 i686 i686 GNU/Linux
>>> cdc@client:~$
>>> cdc@client:~$ mount | grep home
>>> cuentas02:/home-3/cdc on /home/cdc type nfs
>>>
>>> (rw,noatime,intr,fsc,nolock,rsize=262140,wsize=262140,addr=138.4.30.15)
>>> cdc@client:~$
>>>
>>> root@server-lab:~# uname -a
>>> Linux cuentas02-lab.lab.dit.upm.es 3.13.0-63-generic #103-Ubuntu SMP
>>> Fri Aug 14 21:42:59 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>>> root@server-lab:~#
>>> root@server-lab:~# dpkg -l | grep nfs
>>> ii libnfsidmap2:amd64 0.25-5 amd64
>>> NFS idmapping library
>>> ii nfs-common 1:1.2.8-6ubuntu1.1 amd64 NFS
>>> support files common to client and server
>>> ii nfs-kernel-server 1:1.2.8-6ubuntu1.1 amd64
>>> support for NFS kernel server
>>> root@server-lab:~#
>>> root@server-lab:~# exportfs -v
>>> /home-3
>>>
>>> 138.4.30.0/23(rw,async,wdelay,insecure,no_root_squash,no_subtree_check,fsid=3,sec=sys,rw,no_root_squash,no_all_squash)
>>>
>>> root@server-lab:~#
>>>
>>> LOGS ON SERVER SIDE (glusterfs mount logs):
>>> [2015-11-20 10:51:53.872656] I [io-stats.c:1014:io_stats_dump_fd]
>>> 0-home-lab-3: --- fd stats ---
>>> [2015-11-20 10:51:53.872692] I [io-stats.c:1019:io_stats_dump_fd]
>>> 0-home-lab-3: Filename : /cdc/444.txt
>>> [2015-11-20 10:51:53.872704] I [io-stats.c:1034:io_stats_dump_fd]
>>> 0-home-lab-3: BytesWritten : 7 bytes
>>> [2015-11-20 10:51:53.872714] I [io-stats.c:1046:io_stats_dump_fd]
>>> 0-home-lab-3: Write 000004b+ : 1
>>> [2015-11-20 10:51:53.874917] W [MSGID: 114031]
>>> [client-rpc-fops.c:1298:client3_3_removexattr_cbk]
>>> 0-home-lab-3-client-0: remote operation failed [Permission denied]
>>> [2015-11-20 10:51:53.874976] W [fuse-bridge.c:1230:fuse_err_cbk]
>>> 0-glusterfs-fuse: 63459954: REMOVEXATTR() /cdc/444.txt => -1
>>> (Permission denied)
>>> [2015-11-20 10:51:53.881389] W [MSGID: 114031]
>>> [client-rpc-fops.c:1298:client3_3_removexattr_cbk]
>>> 0-home-lab-3-client-3: remote operation failed [Permission denied]
>>> [2015-11-20 10:51:53.881434] W [fuse-bridge.c:1230:fuse_err_cbk]
>>> 0-glusterfs-fuse: 63459961: REMOVEXATTR() /cdc/kk.txt => -1
>>> (Permission denied)
>>> [2015-11-20 10:51:53.883072] W [fuse-bridge.c:1230:fuse_err_cbk]
>>> 0-glusterfs-fuse: 63459964: REMOVEXATTR() /cdc/kk.txt => -1
>>> (Permission denied)
>>> [2015-11-20 10:51:53.883057] W [MSGID: 114031]
>>> [client-rpc-fops.c:1298:client3_3_removexattr_cbk]
>>> 0-home-lab-3-client-3: remote operation failed [Permission denied]
>>> [2015-11-20 10:51:53.884003] E [MSGID: 114031]
>>> [client-rpc-fops.c:466:client3_3_open_cbk] 0-home-lab-3-client-3:
>>> remote operation failed. Path: /cdc/kk.txt
>>> (3175e0cd-8308-45b8-a4b0-699f6f8cf37f) [Permission denied]
>>> [2015-11-20 10:51:53.884056] W [fuse-bridge.c:969:fuse_fd_cbk]
>>> 0-glusterfs-fuse: 63459965: OPEN() /cdc/kk.txt => -1 (Permission
>>> denied)
>>
>> The above message is interesting and might be related to the problem.
>> That said, we generally set the NFSD_MAY_OWNER_OVERRIDE bit on opens of
>> regular files, which allows the nfsd_permission check to pass
>> regardless when the owner matches.
>>
>> My guess would be that the dentry_open call in nfsd_open is failing
>> here as the concept of "owner override" doesn't really get passed down
>> to it. Still, it'd be good to confirm that...
>>
>>> [2015-11-20 10:51:53.885619] W [MSGID: 114031]
>>> [client-rpc-fops.c:1298:client3_3_removexattr_cbk]
>>> 0-home-lab-3-client-3: remote operation failed [Permission denied]
>>> [2015-11-20 10:51:53.885664] W [fuse-bridge.c:1230:fuse_err_cbk]
>>> 0-glusterfs-fuse: 63459967: REMOVEXATTR() /cdc/kk.txt => -1 (Permission
>>> denied)
>>> [2015-11-20 10:51:53.887908] W [fuse-bridge.c:1230:fuse_err_cbk]
>>> 0-glusterfs-fuse: 63459971: REMOVEXATTR() /cdc/kk.txt => -1 (Permission
>>> denied)
>>> [2015-11-20 10:51:53.887891] W [MSGID: 114031]
>>> [client-rpc-fops.c:1298:client3_3_removexattr_cbk]
>>> 0-home-lab-3-client-3: remote operation failed [Permission denied]
>>>
>>> (NOTE: We have more gluster brick logs but we don't know if are
>>> relevant)
>>>

--
----------------------------------------------------------------
Centro de Cálculo Depto. Ingeniería Sistemas Telemáticos
E-mail: [email protected] Universidad Politécnica de Madrid
Fax:(+34) 913367333 E.T.S. Ing. Telecomunicación
Tel:(+34) 915495700-Ext.3005 28040 Madrid (Spain)
Tel:(+34) 915495762-Ext.3005
Tel:(+34) 913367366-Ext.3005
----------------------------------------------------------------

Attachments:

nfs-problem-nks.pcap.zip (3.16 kB)

2015-12-10 12:29:37

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

On 12/10/2015 04:02 PM, Omar Walid Llorente wrote:
>
> Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs
> packets (you can find them in attached file nfs-problem-nks.pcap.zip).
> Sorry for being so late.
>
> What I did was the following:
>
> 1st) Create the RO file:
> cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
> chmod 444 444.txt;
>
> 2nd) Init the capture:
> root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
> tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
> 512 bytes
>
GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It may
be helpful to see what GlusterFS operations are being processed as part
of NFS WRITE call (which has failed in this case).

Could you please try taking the packet trace on the machine where NFS
server is running (without filtering out based on the port number).

Also I tried out the same test on Fedora22 machine, but haven't run into
any issue. What are the fuse mount options you have used to mount
gluster volume?

Thanks,
Soumya

[1] https://www.wireshark.org/docs/dfref/g/glusterfs.html

> 3rd) Try to copy the RO file and get the error:
> cdc@l056:~/prueba-git$ cp -p 444.txt kk.txt;
> cp: failed to close ‘kk.txt’: Permission denied
> cdc@l056:~/prueba-git$
>
> 4th) Close the capture:
> ^C26 packets captured
> 26 packets received by filter
> 0 packets dropped by kernel
> root@l056:~#
>
> Hope you can send us some clue about it. Do you need me to do any other
> test? Thanks in advance!
>
> Omar
>
> El 25/11/15 a las 14:50, omar escribió:
>>
>>
>> Hi, Jeff, thanks for the answer. I'm out of the office until next
>> week, but when I come back, I'll try to do the tests and send you the
>> info.
>>
>> Thank you very much,
>>
>> Omar
>>
>> El 2015-11-21 14:18, Jeff Layton escribió:
>>> On Fri, 20 Nov 2015 12:04:49 +0100
>>> Omar Walid Llorente <[email protected]> wrote:
>>>
>>>>
>>>> Hi, I'm Omar Walid Llorente and I am a systems administrator at the
>>>> Politechnical University of Madrid (UPM), Spain. I write you in the
>>>> hope
>>>> you can help us manage a problem that have discovered recently about
>>>> our
>>>> new datastore architecture in our teaching labs. We have created a
>>>> gluster distributed volume that we reexport with NFS to our lab clients
>>>> via intermediate servers.
>>>>
>>>> First of all thanks for all your work and sorry if this isn't related
>>>> with your package, but I think it has a good chance. I'll try explain
>>>> myself as short as possible.
>>>>
>>>> As introduced previously, we have a problem exporting with
>>>> nfs-kernel-server-1.2.8-6 (ubuntu based) a directory previously mounted
>>>> with gluster-3.7.4 via fuse mount.
>>>>
>>>
>>> What's important here (for the nfs server) is the kernel version. What
>>> kernel version are you running on the server? Also, what NFS version is
>>> the client using? If you grab the mount's line out of /proc/mounts on
>>> the client then that would be helpful.
>>>
>>> Also, does the NFS version matter here? If you're using NFSv4 then
>>> maybe try with NFSv3, or with v4 or so if you're already using v3?
>>>
>>>> The problem is quite simple to reproduce and always repeatable: if a
>>>> file has read-only permissions for owner and user wants to copy it,
>>>> permissions problem arises:
>>>> cdc@client:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444
>>>> 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
>>>> cp: failed to close ‘kk.txt’: Permission denied
>>>> -r--r--r-- 1 cdc admincdc 7 nov 3 2015 444.txt
>>>> -r--r--r-- 1 cdc admincdc 0 nov 3 2015 kk.txt
>>>> cdc@client:~$
>>>>
>>>> If the file permissions are not read-only, there is no problem:
>>>> cdc@client:~$ rm -f kk.txt 644.txt; echo "prueba" > 644.txt; chmod 644
>>>> 644.txt; cp -p 644.txt kk.txt; ls -ld 644.txt kk.txt
>>>> -rw-r--r-- 1 cdc admincdc 7 nov 3 2015 644.txt
>>>> -rw-r--r-- 1 cdc admincdc 7 nov 3 2015 kk.txt
>>>> cdc@client:~$
>>>>
>>>> If we track it down with strace, the problem arises exactly when
>>>> fsync()
>>>> is called from cp.
>>>>
>>>> Of course, if we try this combination of commands in other directories
>>>> not mounted by nfs (local ones) or mounted with samba/cifs or even
>>>> mounted with nfs-ganesha (both fuse mounted with gluster), this doen't
>>>> happen. This problem doesn't happen either if the nfs-kernel-server
>>>> exports a directory not mounted with fuse (any local one).
>>>>
>>>
>>> Ok, that's good info, but when dealing with a problem like this, it'd
>>> be best to get a capture of the network traffic between client and
>>> server while you're reproducing this. We can then look at it to figure
>>> out which RPC call is getting the actual error. That will help narrow
>>> down the problem a bit more.
>>>
>>> You can do that with tcpdump. Something like this should do it:
>>>
>>> # tcpdump -i eth0 -w /tmp/nfs.pcap -s 512 port 2049
>>>
>>> ...reproduce the problem and then stop the capture. Then you can
>>> open /tmp/nfs.pcap with wireshark to analyze it (or send it to me and
>>> I'll take a look).
>>>
>>>> Please, tell me if this is the right place to post the probem and
>>>> where is it if this is not. Let me know if we can help you any way to
>>>> solve or test it (we've developed a small program in c that shows
>>>> exactly the same behaviour).
>>>>
>>>> Thanks again.
>>>>
>>>> Omar
>>>>
>>>> PS: Pointer to this email address came from:
>>>> http://wiki.linux-nfs.org/wiki/index.php/Reporting_bugs
>>>>
>>>> ADDITIONAL INFO:
>>>>
>>>> cdc@client:~$ uname -a
>>>> Linux l056 3.13.0-63-generic #103-Ubuntu SMP Fri Aug 14 21:43:30 UTC
>>>> 2015 i686 i686 i686 GNU/Linux
>>>> cdc@client:~$
>>>> cdc@client:~$ mount | grep home
>>>> cuentas02:/home-3/cdc on /home/cdc type nfs
>>>>
>>>> (rw,noatime,intr,fsc,nolock,rsize=262140,wsize=262140,addr=138.4.30.15)
>>>> cdc@client:~$
>>>>
>>>> root@server-lab:~# uname -a
>>>> Linux cuentas02-lab.lab.dit.upm.es 3.13.0-63-generic #103-Ubuntu SMP
>>>> Fri Aug 14 21:42:59 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>>>> root@server-lab:~#
>>>> root@server-lab:~# dpkg -l | grep nfs
>>>> ii libnfsidmap2:amd64 0.25-5 amd64
>>>> NFS idmapping library
>>>> ii nfs-common 1:1.2.8-6ubuntu1.1 amd64 NFS
>>>> support files common to client and server
>>>> ii nfs-kernel-server 1:1.2.8-6ubuntu1.1 amd64
>>>> support for NFS kernel server
>>>> root@server-lab:~#
>>>> root@server-lab:~# exportfs -v
>>>> /home-3
>>>>
>>>> 138.4.30.0/23(rw,async,wdelay,insecure,no_root_squash,no_subtree_check,fsid=3,sec=sys,rw,no_root_squash,no_all_squash)
>>>>
>>>> root@server-lab:~#
>>>>
>>>> LOGS ON SERVER SIDE (glusterfs mount logs):
>>>> [2015-11-20 10:51:53.872656] I [io-stats.c:1014:io_stats_dump_fd]
>>>> 0-home-lab-3: --- fd stats ---
>>>> [2015-11-20 10:51:53.872692] I [io-stats.c:1019:io_stats_dump_fd]
>>>> 0-home-lab-3: Filename : /cdc/444.txt
>>>> [2015-11-20 10:51:53.872704] I [io-stats.c:1034:io_stats_dump_fd]
>>>> 0-home-lab-3: BytesWritten : 7 bytes
>>>> [2015-11-20 10:51:53.872714] I [io-stats.c:1046:io_stats_dump_fd]
>>>> 0-home-lab-3: Write 000004b+ : 1
>>>> [2015-11-20 10:51:53.874917] W [MSGID: 114031]
>>>> [client-rpc-fops.c:1298:client3_3_removexattr_cbk]
>>>> 0-home-lab-3-client-0: remote operation failed [Permission denied]
>>>> [2015-11-20 10:51:53.874976] W [fuse-bridge.c:1230:fuse_err_cbk]
>>>> 0-glusterfs-fuse: 63459954: REMOVEXATTR() /cdc/444.txt => -1
>>>> (Permission denied)
>>>> [2015-11-20 10:51:53.881389] W [MSGID: 114031]
>>>> [client-rpc-fops.c:1298:client3_3_removexattr_cbk]
>>>> 0-home-lab-3-client-3: remote operation failed [Permission denied]
>>>> [2015-11-20 10:51:53.881434] W [fuse-bridge.c:1230:fuse_err_cbk]
>>>> 0-glusterfs-fuse: 63459961: REMOVEXATTR() /cdc/kk.txt => -1
>>>> (Permission denied)
>>>> [2015-11-20 10:51:53.883072] W [fuse-bridge.c:1230:fuse_err_cbk]
>>>> 0-glusterfs-fuse: 63459964: REMOVEXATTR() /cdc/kk.txt => -1
>>>> (Permission denied)
>>>> [2015-11-20 10:51:53.883057] W [MSGID: 114031]
>>>> [client-rpc-fops.c:1298:client3_3_removexattr_cbk]
>>>> 0-home-lab-3-client-3: remote operation failed [Permission denied]
>>>> [2015-11-20 10:51:53.884003] E [MSGID: 114031]
>>>> [client-rpc-fops.c:466:client3_3_open_cbk] 0-home-lab-3-client-3:
>>>> remote operation failed. Path: /cdc/kk.txt
>>>> (3175e0cd-8308-45b8-a4b0-699f6f8cf37f) [Permission denied]
>>>> [2015-11-20 10:51:53.884056] W [fuse-bridge.c:969:fuse_fd_cbk]
>>>> 0-glusterfs-fuse: 63459965: OPEN() /cdc/kk.txt => -1 (Permission
>>>> denied)
>>>
>>> The above message is interesting and might be related to the problem.
>>> That said, we generally set the NFSD_MAY_OWNER_OVERRIDE bit on opens of
>>> regular files, which allows the nfsd_permission check to pass
>>> regardless when the owner matches.
>>>
>>> My guess would be that the dentry_open call in nfsd_open is failing
>>> here as the concept of "owner override" doesn't really get passed down
>>> to it. Still, it'd be good to confirm that...
>>>
>>>> [2015-11-20 10:51:53.885619] W [MSGID: 114031]
>>>> [client-rpc-fops.c:1298:client3_3_removexattr_cbk]
>>>> 0-home-lab-3-client-3: remote operation failed [Permission denied]
>>>> [2015-11-20 10:51:53.885664] W [fuse-bridge.c:1230:fuse_err_cbk]
>>>> 0-glusterfs-fuse: 63459967: REMOVEXATTR() /cdc/kk.txt => -1 (Permission
>>>> denied)
>>>> [2015-11-20 10:51:53.887908] W [fuse-bridge.c:1230:fuse_err_cbk]
>>>> 0-glusterfs-fuse: 63459971: REMOVEXATTR() /cdc/kk.txt => -1 (Permission
>>>> denied)
>>>> [2015-11-20 10:51:53.887891] W [MSGID: 114031]
>>>> [client-rpc-fops.c:1298:client3_3_removexattr_cbk]
>>>> 0-home-lab-3-client-3: remote operation failed [Permission denied]
>>>>
>>>> (NOTE: We have more gluster brick logs but we don't know if are
>>>> relevant)
>>>>
>

2015-12-10 14:44:36

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

On Thu, Dec 10, 2015 at 05:59:33PM +0530, Soumya Koduri wrote:
>
>
> On 12/10/2015 04:02 PM, Omar Walid Llorente wrote:
> >
> >Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs
> >packets (you can find them in attached file nfs-problem-nks.pcap.zip).
> >Sorry for being so late.
> >
> >What I did was the following:
> >
> >1st) Create the RO file:
> >cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
> >chmod 444 444.txt;
> >
> >2nd) Init the capture:
> >root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
> >tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
> >512 bytes
> >
> GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It
> may be helpful to see what GlusterFS operations are being processed
> as part of NFS WRITE call (which has failed in this case).
>
> Could you please try taking the packet trace on the machine where
> NFS server is running (without filtering out based on the port
> number).
>
> Also I tried out the same test on Fedora22 machine, but haven't run
> into any issue. What are the fuse mount options you have used to
> mount gluster volume?

Oh, I think this is a simple problem (but maybe hard to fix). The
capture shows NFSv3 traffic like:

CREATE -> OK
SETATTR (mode set to 0400) -> OK
WRITE -> NFS3ERR_ACCES

That write would succeed locally (because the mode doesn't matter to a
local application that already holds the file open). It would fail over
NFSv3, which doesn't know about the open--except that there's a hack for
this case: NFSv3 servers allow IO operations to ignore the mode, if the
operation comes from the owner of the file. NFSv3 clients are then
careful to perform necessary access checks on open to ensure that this
owner-override rule doesn't grant too many permissions.

That allows NFSv3 applications to see behavior that's mostly like a
local filesystem, without opening much of a security hole (since the
owner could always chmod anyway).

So, knfsd is making this special exception--but gluster (which I believe
it's exporting in this case, via fuse?)--probably doesn't.... I'm not
sure what you can do about that.

--b.

2015-12-14 16:57:22

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

Thank you Bruce, others, for the responses. I send attached a complete
capture of the issue, including the glusterfs transactions.

Hope this helps to clear where may it be...

Omar

El 10/12/15 a las 15:44, J. Bruce Fields escribi?:
> On Thu, Dec 10, 2015 at 05:59:33PM +0530, Soumya Koduri wrote:
>>
>> On 12/10/2015 04:02 PM, Omar Walid Llorente wrote:
>>> Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs
>>> packets (you can find them in attached file nfs-problem-nks.pcap.zip).
>>> Sorry for being so late.
>>>
>>> What I did was the following:
>>>
>>> 1st) Create the RO file:
>>> cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
>>> chmod 444 444.txt;
>>>
>>> 2nd) Init the capture:
>>> root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
>>> tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
>>> 512 bytes
>>>
>> GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It
>> may be helpful to see what GlusterFS operations are being processed
>> as part of NFS WRITE call (which has failed in this case).
>>
>> Could you please try taking the packet trace on the machine where
>> NFS server is running (without filtering out based on the port
>> number).
>>
>> Also I tried out the same test on Fedora22 machine, but haven't run
>> into any issue. What are the fuse mount options you have used to
>> mount gluster volume?
> Oh, I think this is a simple problem (but maybe hard to fix). The
> capture shows NFSv3 traffic like:
>
> CREATE -> OK
> SETATTR (mode set to 0400) -> OK
> WRITE -> NFS3ERR_ACCES
>
> That write would succeed locally (because the mode doesn't matter to a
> local application that already holds the file open). It would fail over
> NFSv3, which doesn't know about the open--except that there's a hack for
> this case: NFSv3 servers allow IO operations to ignore the mode, if the
> operation comes from the owner of the file. NFSv3 clients are then
> careful to perform necessary access checks on open to ensure that this
> owner-override rule doesn't grant too many permissions.
>
> That allows NFSv3 applications to see behavior that's mostly like a
> local filesystem, without opening much of a security hole (since the
> owner could always chmod anyway).
>
> So, knfsd is making this special exception--but gluster (which I believe
> it's exporting in this case, via fuse?)--probably doesn't.... I'm not
> sure what you can do about that.
>
> --b.

--
----------------------------------------------------------------
Centro de C?lculo Depto. Ingenier?a Sistemas Telem?ticos
E-mail: [email protected] Universidad Polit?cnica de Madrid
Fax:(+34) 913367333 E.T.S. Ing. Telecomunicaci?n
Tel:(+34) 915495700-Ext.3005 28040 Madrid (Spain)
Tel:(+34) 915495762-Ext.3005
Tel:(+34) 913367366-Ext.3005
----------------------------------------------------------------

Attachments:

nfs-problem-nks+glusterfs.pcap.zip (4.62 kB)

2015-12-17 12:16:18

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

As mentioned by Bruce, GlusterFS doesn't have owner-override rule except
for setattr.

I did few experiments to check why this test case passes on plain
glusterfs fuse mount & NFS-Ganesha but fails with kernel-NFS.

NFS-Ganesha (for most of the FSALs) seem to be passing the actual
request credentials to the back-end filesystem only for CREATE(-like)
and UNLINK fops. For all the remaining fops, it does the access check at
its end and then perform the operation with root credentials. That's the
reason WRITE succeeded in your case as NFS-Ganesha (like kernel-NFS)
skipped the access check if the request caller_uid proved to be the
file's owner.

In case of native GlusterFS FUSE mount, there is no OPEN fop involved.
WRITE is performed on the fd returned by CREATE. And strangely GlusterFS
seem to be doing certain access checks only during OPEN but not for
WRITE (this seems like a bug and probably needs to be fixed in Gluster).

Thanks,
Soumya

On 12/14/2015 10:27 PM, Omar Walid Llorente wrote:
>
> Thank you Bruce, others, for the responses. I send attached a complete
> capture of the issue, including the glusterfs transactions.
>
> Hope this helps to clear where may it be...
>
> Omar
>
> El 10/12/15 a las 15:44, J. Bruce Fields escribi?:
>> On Thu, Dec 10, 2015 at 05:59:33PM +0530, Soumya Koduri wrote:
>>>
>>> On 12/10/2015 04:02 PM, Omar Walid Llorente wrote:
>>>> Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs
>>>> packets (you can find them in attached file nfs-problem-nks.pcap.zip).
>>>> Sorry for being so late.
>>>>
>>>> What I did was the following:
>>>>
>>>> 1st) Create the RO file:
>>>> cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
>>>> chmod 444 444.txt;
>>>>
>>>> 2nd) Init the capture:
>>>> root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
>>>> tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
>>>> 512 bytes
>>>>
>>> GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It
>>> may be helpful to see what GlusterFS operations are being processed
>>> as part of NFS WRITE call (which has failed in this case).
>>>
>>> Could you please try taking the packet trace on the machine where
>>> NFS server is running (without filtering out based on the port
>>> number).
>>>
>>> Also I tried out the same test on Fedora22 machine, but haven't run
>>> into any issue. What are the fuse mount options you have used to
>>> mount gluster volume?
>> Oh, I think this is a simple problem (but maybe hard to fix). The
>> capture shows NFSv3 traffic like:
>>
>> CREATE -> OK
>> SETATTR (mode set to 0400) -> OK
>> WRITE -> NFS3ERR_ACCES
>>
>> That write would succeed locally (because the mode doesn't matter to a
>> local application that already holds the file open). It would fail over
>> NFSv3, which doesn't know about the open--except that there's a hack for
>> this case: NFSv3 servers allow IO operations to ignore the mode, if the
>> operation comes from the owner of the file. NFSv3 clients are then
>> careful to perform necessary access checks on open to ensure that this
>> owner-override rule doesn't grant too many permissions.
>>
>> That allows NFSv3 applications to see behavior that's mostly like a
>> local filesystem, without opening much of a security hole (since the
>> owner could always chmod anyway).
>>
>> So, knfsd is making this special exception--but gluster (which I believe
>> it's exporting in this case, via fuse?)--probably doesn't.... I'm not
>> sure what you can do about that.
>>
>> --b.
>

2015-12-18 00:37:30

by Malahal Naineni

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

IIRC, permission checks are done in open(). write/read syscalls should
NOT do much access checks (at least based on POSIX). This is why once an
open is done, you remove permissions for that process, but it should
still be able to read/write based on the open flags it did when it
opened the file.

I don't know all the details of this defect, but gluster seems to be
doing what it is supposed to do.

Regards, Malahal.

Soumya Koduri [[email protected]] wrote:
> As mentioned by Bruce, GlusterFS doesn't have owner-override rule
> except for setattr.
>
> I did few experiments to check why this test case passes on plain
> glusterfs fuse mount & NFS-Ganesha but fails with kernel-NFS.
>
> NFS-Ganesha (for most of the FSALs) seem to be passing the actual
> request credentials to the back-end filesystem only for
> CREATE(-like) and UNLINK fops. For all the remaining fops, it does
> the access check at its end and then perform the operation with root
> credentials. That's the reason WRITE succeeded in your case as
> NFS-Ganesha (like kernel-NFS) skipped the access check if the
> request caller_uid proved to be the file's owner.
>
> In case of native GlusterFS FUSE mount, there is no OPEN fop
> involved. WRITE is performed on the fd returned by CREATE. And
> strangely GlusterFS seem to be doing certain access checks only
> during OPEN but not for WRITE (this seems like a bug and probably
> needs to be fixed in Gluster).
>
> Thanks,
> Soumya
>
> On 12/14/2015 10:27 PM, Omar Walid Llorente wrote:
> >
> >Thank you Bruce, others, for the responses. I send attached a complete
> >capture of the issue, including the glusterfs transactions.
> >
> >Hope this helps to clear where may it be...
> >
> >Omar
> >
> >El 10/12/15 a las 15:44, J. Bruce Fields escribi?:
> >>On Thu, Dec 10, 2015 at 05:59:33PM +0530, Soumya Koduri wrote:
> >>>
> >>>On 12/10/2015 04:02 PM, Omar Walid Llorente wrote:
> >>>>Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs
> >>>>packets (you can find them in attached file nfs-problem-nks.pcap.zip).
> >>>>Sorry for being so late.
> >>>>
> >>>>What I did was the following:
> >>>>
> >>>>1st) Create the RO file:
> >>>>cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
> >>>>chmod 444 444.txt;
> >>>>
> >>>>2nd) Init the capture:
> >>>>root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
> >>>>tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
> >>>>512 bytes
> >>>>
> >>>GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It
> >>>may be helpful to see what GlusterFS operations are being processed
> >>>as part of NFS WRITE call (which has failed in this case).
> >>>
> >>>Could you please try taking the packet trace on the machine where
> >>>NFS server is running (without filtering out based on the port
> >>>number).
> >>>
> >>>Also I tried out the same test on Fedora22 machine, but haven't run
> >>>into any issue. What are the fuse mount options you have used to
> >>>mount gluster volume?
> >>Oh, I think this is a simple problem (but maybe hard to fix). The
> >>capture shows NFSv3 traffic like:
> >>
> >> CREATE -> OK
> >> SETATTR (mode set to 0400) -> OK
> >> WRITE -> NFS3ERR_ACCES
> >>
> >>That write would succeed locally (because the mode doesn't matter to a
> >>local application that already holds the file open). It would fail over
> >>NFSv3, which doesn't know about the open--except that there's a hack for
> >>this case: NFSv3 servers allow IO operations to ignore the mode, if the
> >>operation comes from the owner of the file. NFSv3 clients are then
> >>careful to perform necessary access checks on open to ensure that this
> >>owner-override rule doesn't grant too many permissions.
> >>
> >>That allows NFSv3 applications to see behavior that's mostly like a
> >>local filesystem, without opening much of a security hole (since the
> >>owner could always chmod anyway).
> >>
> >>So, knfsd is making this special exception--but gluster (which I believe
> >>it's exporting in this case, via fuse?)--probably doesn't.... I'm not
> >>sure what you can do about that.
> >>
> >>--b.
> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>

2015-12-18 08:43:45

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

On 12/18/2015 06:07 AM, Malahal Naineni wrote:
> IIRC, permission checks are done in open(). write/read syscalls should
> NOT do much access checks (at least based on POSIX). This is why once an
> open is done, you remove permissions for that process, but it should
> still be able to read/write based on the open flags it did when it
> opened the file.
>
> I don't know all the details of this defect, but gluster seems to be
> doing what it is supposed to do.
>
Right. Thanks for the correction. I assumed the behavior should be same
for both OPEN+WRITE vs CREATE+WRITE in the below scenario. But looks
like (from 'man creat') the open() call that creates a read-only file
may well return a read/write file descriptor, which is the reason the
following WRITE can succeed.

Thanks,
Soumya

> Regards, Malahal.
>
> Soumya Koduri [[email protected]] wrote:
>> As mentioned by Bruce, GlusterFS doesn't have owner-override rule
>> except for setattr.
>>
>> I did few experiments to check why this test case passes on plain
>> glusterfs fuse mount & NFS-Ganesha but fails with kernel-NFS.
>>
>> NFS-Ganesha (for most of the FSALs) seem to be passing the actual
>> request credentials to the back-end filesystem only for
>> CREATE(-like) and UNLINK fops. For all the remaining fops, it does
>> the access check at its end and then perform the operation with root
>> credentials. That's the reason WRITE succeeded in your case as
>> NFS-Ganesha (like kernel-NFS) skipped the access check if the
>> request caller_uid proved to be the file's owner.
>>
>> In case of native GlusterFS FUSE mount, there is no OPEN fop
>> involved. WRITE is performed on the fd returned by CREATE. And
>> strangely GlusterFS seem to be doing certain access checks only
>> during OPEN but not for WRITE (this seems like a bug and probably
>> needs to be fixed in Gluster).
>>
>> Thanks,
>> Soumya
>>
>> On 12/14/2015 10:27 PM, Omar Walid Llorente wrote:
>>>
>>> Thank you Bruce, others, for the responses. I send attached a complete
>>> capture of the issue, including the glusterfs transactions.
>>>
>>> Hope this helps to clear where may it be...
>>>
>>> Omar
>>>
>>> El 10/12/15 a las 15:44, J. Bruce Fields escribi?:
>>>> On Thu, Dec 10, 2015 at 05:59:33PM +0530, Soumya Koduri wrote:
>>>>>
>>>>> On 12/10/2015 04:02 PM, Omar Walid Llorente wrote:
>>>>>> Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs
>>>>>> packets (you can find them in attached file nfs-problem-nks.pcap.zip).
>>>>>> Sorry for being so late.
>>>>>>
>>>>>> What I did was the following:
>>>>>>
>>>>>> 1st) Create the RO file:
>>>>>> cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
>>>>>> chmod 444 444.txt;
>>>>>>
>>>>>> 2nd) Init the capture:
>>>>>> root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
>>>>>> tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
>>>>>> 512 bytes
>>>>>>
>>>>> GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It
>>>>> may be helpful to see what GlusterFS operations are being processed
>>>>> as part of NFS WRITE call (which has failed in this case).
>>>>>
>>>>> Could you please try taking the packet trace on the machine where
>>>>> NFS server is running (without filtering out based on the port
>>>>> number).
>>>>>
>>>>> Also I tried out the same test on Fedora22 machine, but haven't run
>>>>> into any issue. What are the fuse mount options you have used to
>>>>> mount gluster volume?
>>>> Oh, I think this is a simple problem (but maybe hard to fix). The
>>>> capture shows NFSv3 traffic like:
>>>>
>>>> CREATE -> OK
>>>> SETATTR (mode set to 0400) -> OK
>>>> WRITE -> NFS3ERR_ACCES
>>>>
>>>> That write would succeed locally (because the mode doesn't matter to a
>>>> local application that already holds the file open). It would fail over
>>>> NFSv3, which doesn't know about the open--except that there's a hack for
>>>> this case: NFSv3 servers allow IO operations to ignore the mode, if the
>>>> operation comes from the owner of the file. NFSv3 clients are then
>>>> careful to perform necessary access checks on open to ensure that this
>>>> owner-override rule doesn't grant too many permissions.
>>>>
>>>> That allows NFSv3 applications to see behavior that's mostly like a
>>>> local filesystem, without opening much of a security hole (since the
>>>> owner could always chmod anyway).
>>>>
>>>> So, knfsd is making this special exception--but gluster (which I believe
>>>> it's exporting in this case, via fuse?)--probably doesn't.... I'm not
>>>> sure what you can do about that.
>>>>
>>>> --b.
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to [email protected]
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>

2015-12-18 15:20:40

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

On Fri, Dec 18, 2015 at 02:13:40PM +0530, Soumya Koduri wrote:
>
>
> On 12/18/2015 06:07 AM, Malahal Naineni wrote:
> >IIRC, permission checks are done in open(). write/read syscalls should
> >NOT do much access checks (at least based on POSIX). This is why once an
> >open is done, you remove permissions for that process, but it should
> >still be able to read/write based on the open flags it did when it
> >opened the file.
> >
> >I don't know all the details of this defect, but gluster seems to be
> >doing what it is supposed to do.
> >
> Right. Thanks for the correction. I assumed the behavior should be
> same for both OPEN+WRITE vs CREATE+WRITE in the below scenario. But
> looks like (from 'man creat') the open() call that creates a
> read-only file may well return a read/write file descriptor, which
> is the reason the following WRITE can succeed.

I forgot another complication, which is that knsfd actually does a
temporary open before each read or write--I assume that's getting
translated into fuse and gluster open operations?

In which case it might be worth experimenting with NFSv4 or with Jeff
Layton's filehandle-caching patches. Neither's a real fix, but that
could help confirm whether it's the temporary opens that are a problem.

--b.

>
> Thanks,
> Soumya
>
>
> >Regards, Malahal.
> >
> >Soumya Koduri [[email protected]] wrote:
> >>As mentioned by Bruce, GlusterFS doesn't have owner-override rule
> >>except for setattr.
> >>
> >>I did few experiments to check why this test case passes on plain
> >>glusterfs fuse mount & NFS-Ganesha but fails with kernel-NFS.
> >>
> >>NFS-Ganesha (for most of the FSALs) seem to be passing the actual
> >>request credentials to the back-end filesystem only for
> >>CREATE(-like) and UNLINK fops. For all the remaining fops, it does
> >>the access check at its end and then perform the operation with root
> >>credentials. That's the reason WRITE succeeded in your case as
> >>NFS-Ganesha (like kernel-NFS) skipped the access check if the
> >>request caller_uid proved to be the file's owner.
> >>
> >>In case of native GlusterFS FUSE mount, there is no OPEN fop
> >>involved. WRITE is performed on the fd returned by CREATE. And
> >>strangely GlusterFS seem to be doing certain access checks only
> >>during OPEN but not for WRITE (this seems like a bug and probably
> >>needs to be fixed in Gluster).
> >>
> >>Thanks,
> >>Soumya
> >>
> >>On 12/14/2015 10:27 PM, Omar Walid Llorente wrote:
> >>>
> >>>Thank you Bruce, others, for the responses. I send attached a complete
> >>>capture of the issue, including the glusterfs transactions.
> >>>
> >>>Hope this helps to clear where may it be...
> >>>
> >>>Omar
> >>>
> >>>El 10/12/15 a las 15:44, J. Bruce Fields escribió:
> >>>>On Thu, Dec 10, 2015 at 05:59:33PM +0530, Soumya Koduri wrote:
> >>>>>
> >>>>>On 12/10/2015 04:02 PM, Omar Walid Llorente wrote:
> >>>>>>Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs
> >>>>>>packets (you can find them in attached file nfs-problem-nks.pcap.zip).
> >>>>>>Sorry for being so late.
> >>>>>>
> >>>>>>What I did was the following:
> >>>>>>
> >>>>>>1st) Create the RO file:
> >>>>>>cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
> >>>>>>chmod 444 444.txt;
> >>>>>>
> >>>>>>2nd) Init the capture:
> >>>>>>root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
> >>>>>>tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
> >>>>>>512 bytes
> >>>>>>
> >>>>>GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It
> >>>>>may be helpful to see what GlusterFS operations are being processed
> >>>>>as part of NFS WRITE call (which has failed in this case).
> >>>>>
> >>>>>Could you please try taking the packet trace on the machine where
> >>>>>NFS server is running (without filtering out based on the port
> >>>>>number).
> >>>>>
> >>>>>Also I tried out the same test on Fedora22 machine, but haven't run
> >>>>>into any issue. What are the fuse mount options you have used to
> >>>>>mount gluster volume?
> >>>>Oh, I think this is a simple problem (but maybe hard to fix). The
> >>>>capture shows NFSv3 traffic like:
> >>>>
> >>>> CREATE -> OK
> >>>> SETATTR (mode set to 0400) -> OK
> >>>> WRITE -> NFS3ERR_ACCES
> >>>>
> >>>>That write would succeed locally (because the mode doesn't matter to a
> >>>>local application that already holds the file open). It would fail over
> >>>>NFSv3, which doesn't know about the open--except that there's a hack for
> >>>>this case: NFSv3 servers allow IO operations to ignore the mode, if the
> >>>>operation comes from the owner of the file. NFSv3 clients are then
> >>>>careful to perform necessary access checks on open to ensure that this
> >>>>owner-override rule doesn't grant too many permissions.
> >>>>
> >>>>That allows NFSv3 applications to see behavior that's mostly like a
> >>>>local filesystem, without opening much of a security hole (since the
> >>>>owner could always chmod anyway).
> >>>>
> >>>>So, knfsd is making this special exception--but gluster (which I believe
> >>>>it's exporting in this case, via fuse?)--probably doesn't.... I'm not
> >>>>sure what you can do about that.
> >>>>
> >>>>--b.
> >>>
> >>--
> >>To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> >>the body of a message to [email protected]
> >>More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>
> >

2015-12-18 17:17:47

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

On 12/18/2015 08:50 PM, J. Bruce Fields wrote:
> On Fri, Dec 18, 2015 at 02:13:40PM +0530, Soumya Koduri wrote:
>>
>>
>> On 12/18/2015 06:07 AM, Malahal Naineni wrote:
>>> IIRC, permission checks are done in open(). write/read syscalls should
>>> NOT do much access checks (at least based on POSIX). This is why once an
>>> open is done, you remove permissions for that process, but it should
>>> still be able to read/write based on the open flags it did when it
>>> opened the file.
>>>
>>> I don't know all the details of this defect, but gluster seems to be
>>> doing what it is supposed to do.
>>>
>> Right. Thanks for the correction. I assumed the behavior should be
>> same for both OPEN+WRITE vs CREATE+WRITE in the below scenario. But
>> looks like (from 'man creat') the open() call that creates a
>> read-only file may well return a read/write file descriptor, which
>> is the reason the following WRITE can succeed.
>
> I forgot another complication, which is that knsfd actually does a
> temporary open before each read or write--I assume that's getting
> translated into fuse and gluster open operations?
>
yes. It is the OPEN done as part of NFS WRITE which fails with EACCESS
error (with both NFSv3 and NFSv4 mounts).

63 16:59:09.278651000 ::1 -> ::1 NFS 232 V3 WRITE
Call, FH: 0x49a35e54 Offset: 0 Len: 7 FILE_SYNC
64 16:59:09.278926000 192.168.122.1 -> 192.168.122.202 GlusterFS 164
V330 OPEN Call
65 16:59:09.278937000 192.168.122.1 -> 192.168.122.202 GlusterFS 164
[RPC retransmission of #64][TCP Retransmission] V330 OPEN Call
66 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS 116
V330 OPEN Reply (Call In 64)
67 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS 116
[RPC duplicate of #66][TCP Retransmission] V330 OPEN Reply (Call In 64)
68 16:59:09.279733000 ::1 -> ::1 NFS 212 V3 WRITE
Reply (Call In 63) Error: NFS3ERR_ACCES

Thanks,
Soumya

> In which case it might be worth experimenting with NFSv4 or with Jeff
> Layton's filehandle-caching patches. Neither's a real fix, but that
> could help confirm whether it's the temporary opens that are a problem.
>
> --b.
>
>>
>> Thanks,
>> Soumya
>>
>>
>>> Regards, Malahal.
>>>
>>> Soumya Koduri [[email protected]] wrote:
>>>> As mentioned by Bruce, GlusterFS doesn't have owner-override rule
>>>> except for setattr.
>>>>
>>>> I did few experiments to check why this test case passes on plain
>>>> glusterfs fuse mount & NFS-Ganesha but fails with kernel-NFS.
>>>>
>>>> NFS-Ganesha (for most of the FSALs) seem to be passing the actual
>>>> request credentials to the back-end filesystem only for
>>>> CREATE(-like) and UNLINK fops. For all the remaining fops, it does
>>>> the access check at its end and then perform the operation with root
>>>> credentials. That's the reason WRITE succeeded in your case as
>>>> NFS-Ganesha (like kernel-NFS) skipped the access check if the
>>>> request caller_uid proved to be the file's owner.
>>>>
>>>> In case of native GlusterFS FUSE mount, there is no OPEN fop
>>>> involved. WRITE is performed on the fd returned by CREATE. And
>>>> strangely GlusterFS seem to be doing certain access checks only
>>>> during OPEN but not for WRITE (this seems like a bug and probably
>>>> needs to be fixed in Gluster).
>>>>
>>>> Thanks,
>>>> Soumya
>>>>
>>>> On 12/14/2015 10:27 PM, Omar Walid Llorente wrote:
>>>>>
>>>>> Thank you Bruce, others, for the responses. I send attached a complete
>>>>> capture of the issue, including the glusterfs transactions.
>>>>>
>>>>> Hope this helps to clear where may it be...
>>>>>
>>>>> Omar
>>>>>
>>>>> El 10/12/15 a las 15:44, J. Bruce Fields escribió:
>>>>>> On Thu, Dec 10, 2015 at 05:59:33PM +0530, Soumya Koduri wrote:
>>>>>>>
>>>>>>> On 12/10/2015 04:02 PM, Omar Walid Llorente wrote:
>>>>>>>> Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs
>>>>>>>> packets (you can find them in attached file nfs-problem-nks.pcap.zip).
>>>>>>>> Sorry for being so late.
>>>>>>>>
>>>>>>>> What I did was the following:
>>>>>>>>
>>>>>>>> 1st) Create the RO file:
>>>>>>>> cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
>>>>>>>> chmod 444 444.txt;
>>>>>>>>
>>>>>>>> 2nd) Init the capture:
>>>>>>>> root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
>>>>>>>> tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
>>>>>>>> 512 bytes
>>>>>>>>
>>>>>>> GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It
>>>>>>> may be helpful to see what GlusterFS operations are being processed
>>>>>>> as part of NFS WRITE call (which has failed in this case).
>>>>>>>
>>>>>>> Could you please try taking the packet trace on the machine where
>>>>>>> NFS server is running (without filtering out based on the port
>>>>>>> number).
>>>>>>>
>>>>>>> Also I tried out the same test on Fedora22 machine, but haven't run
>>>>>>> into any issue. What are the fuse mount options you have used to
>>>>>>> mount gluster volume?
>>>>>> Oh, I think this is a simple problem (but maybe hard to fix). The
>>>>>> capture shows NFSv3 traffic like:
>>>>>>
>>>>>> CREATE -> OK
>>>>>> SETATTR (mode set to 0400) -> OK
>>>>>> WRITE -> NFS3ERR_ACCES
>>>>>>
>>>>>> That write would succeed locally (because the mode doesn't matter to a
>>>>>> local application that already holds the file open). It would fail over
>>>>>> NFSv3, which doesn't know about the open--except that there's a hack for
>>>>>> this case: NFSv3 servers allow IO operations to ignore the mode, if the
>>>>>> operation comes from the owner of the file. NFSv3 clients are then
>>>>>> careful to perform necessary access checks on open to ensure that this
>>>>>> owner-override rule doesn't grant too many permissions.
>>>>>>
>>>>>> That allows NFSv3 applications to see behavior that's mostly like a
>>>>>> local filesystem, without opening much of a security hole (since the
>>>>>> owner could always chmod anyway).
>>>>>>
>>>>>> So, knfsd is making this special exception--but gluster (which I believe
>>>>>> it's exporting in this case, via fuse?)--probably doesn't.... I'm not
>>>>>> sure what you can do about that.
>>>>>>
>>>>>> --b.
>>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>>> the body of a message to [email protected]
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>
>>>

2015-12-18 20:08:41

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

On Fri, Dec 18, 2015 at 10:47:42PM +0530, Soumya Koduri wrote:
>
>
> On 12/18/2015 08:50 PM, J. Bruce Fields wrote:
> >On Fri, Dec 18, 2015 at 02:13:40PM +0530, Soumya Koduri wrote:
> >>
> >>
> >>On 12/18/2015 06:07 AM, Malahal Naineni wrote:
> >>>IIRC, permission checks are done in open(). write/read syscalls should
> >>>NOT do much access checks (at least based on POSIX). This is why once an
> >>>open is done, you remove permissions for that process, but it should
> >>>still be able to read/write based on the open flags it did when it
> >>>opened the file.
> >>>
> >>>I don't know all the details of this defect, but gluster seems to be
> >>>doing what it is supposed to do.
> >>>
> >>Right. Thanks for the correction. I assumed the behavior should be
> >>same for both OPEN+WRITE vs CREATE+WRITE in the below scenario. But
> >>looks like (from 'man creat') the open() call that creates a
> >>read-only file may well return a read/write file descriptor, which
> >>is the reason the following WRITE can succeed.
> >
> >I forgot another complication, which is that knsfd actually does a
> >temporary open before each read or write--I assume that's getting
> >translated into fuse and gluster open operations?
> >
> yes. It is the OPEN done as part of NFS WRITE which fails with
> EACCESS error (with both NFSv3 and NFSv4 mounts).

Makes sense for v3, but I wouldn't normally expect the extra temporary
open on v4 WRITEs. Could you share any details?

--b.

>
> 63 16:59:09.278651000 ::1 -> ::1 NFS 232 V3 WRITE
> Call, FH: 0x49a35e54 Offset: 0 Len: 7 FILE_SYNC
> 64 16:59:09.278926000 192.168.122.1 -> 192.168.122.202 GlusterFS
> 164 V330 OPEN Call
> 65 16:59:09.278937000 192.168.122.1 -> 192.168.122.202 GlusterFS
> 164 [RPC retransmission of #64][TCP Retransmission] V330 OPEN Call
> 66 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
> 116 V330 OPEN Reply (Call In 64)
> 67 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
> 116 [RPC duplicate of #66][TCP Retransmission] V330 OPEN Reply (Call
> In 64)
> 68 16:59:09.279733000 ::1 -> ::1 NFS 212 V3 WRITE
> Reply (Call In 63) Error: NFS3ERR_ACCES
>
>
> Thanks,
> Soumya
>
> >In which case it might be worth experimenting with NFSv4 or with Jeff
> >Layton's filehandle-caching patches. Neither's a real fix, but that
> >could help confirm whether it's the temporary opens that are a problem.
> >
> >--b.
> >
> >>
> >>Thanks,
> >>Soumya
> >>
> >>
> >>>Regards, Malahal.
> >>>
> >>>Soumya Koduri [[email protected]] wrote:
> >>>>As mentioned by Bruce, GlusterFS doesn't have owner-override rule
> >>>>except for setattr.
> >>>>
> >>>>I did few experiments to check why this test case passes on plain
> >>>>glusterfs fuse mount & NFS-Ganesha but fails with kernel-NFS.
> >>>>
> >>>>NFS-Ganesha (for most of the FSALs) seem to be passing the actual
> >>>>request credentials to the back-end filesystem only for
> >>>>CREATE(-like) and UNLINK fops. For all the remaining fops, it does
> >>>>the access check at its end and then perform the operation with root
> >>>>credentials. That's the reason WRITE succeeded in your case as
> >>>>NFS-Ganesha (like kernel-NFS) skipped the access check if the
> >>>>request caller_uid proved to be the file's owner.
> >>>>
> >>>>In case of native GlusterFS FUSE mount, there is no OPEN fop
> >>>>involved. WRITE is performed on the fd returned by CREATE. And
> >>>>strangely GlusterFS seem to be doing certain access checks only
> >>>>during OPEN but not for WRITE (this seems like a bug and probably
> >>>>needs to be fixed in Gluster).
> >>>>
> >>>>Thanks,
> >>>>Soumya
> >>>>
> >>>>On 12/14/2015 10:27 PM, Omar Walid Llorente wrote:
> >>>>>
> >>>>>Thank you Bruce, others, for the responses. I send attached a complete
> >>>>>capture of the issue, including the glusterfs transactions.
> >>>>>
> >>>>>Hope this helps to clear where may it be...
> >>>>>
> >>>>>Omar
> >>>>>
> >>>>>El 10/12/15 a las 15:44, J. Bruce Fields escribió:
> >>>>>>On Thu, Dec 10, 2015 at 05:59:33PM +0530, Soumya Koduri wrote:
> >>>>>>>
> >>>>>>>On 12/10/2015 04:02 PM, Omar Walid Llorente wrote:
> >>>>>>>>Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs
> >>>>>>>>packets (you can find them in attached file nfs-problem-nks.pcap.zip).
> >>>>>>>>Sorry for being so late.
> >>>>>>>>
> >>>>>>>>What I did was the following:
> >>>>>>>>
> >>>>>>>>1st) Create the RO file:
> >>>>>>>>cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
> >>>>>>>>chmod 444 444.txt;
> >>>>>>>>
> >>>>>>>>2nd) Init the capture:
> >>>>>>>>root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
> >>>>>>>>tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
> >>>>>>>>512 bytes
> >>>>>>>>
> >>>>>>>GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It
> >>>>>>>may be helpful to see what GlusterFS operations are being processed
> >>>>>>>as part of NFS WRITE call (which has failed in this case).
> >>>>>>>
> >>>>>>>Could you please try taking the packet trace on the machine where
> >>>>>>>NFS server is running (without filtering out based on the port
> >>>>>>>number).
> >>>>>>>
> >>>>>>>Also I tried out the same test on Fedora22 machine, but haven't run
> >>>>>>>into any issue. What are the fuse mount options you have used to
> >>>>>>>mount gluster volume?
> >>>>>>Oh, I think this is a simple problem (but maybe hard to fix). The
> >>>>>>capture shows NFSv3 traffic like:
> >>>>>>
> >>>>>> CREATE -> OK
> >>>>>> SETATTR (mode set to 0400) -> OK
> >>>>>> WRITE -> NFS3ERR_ACCES
> >>>>>>
> >>>>>>That write would succeed locally (because the mode doesn't matter to a
> >>>>>>local application that already holds the file open). It would fail over
> >>>>>>NFSv3, which doesn't know about the open--except that there's a hack for
> >>>>>>this case: NFSv3 servers allow IO operations to ignore the mode, if the
> >>>>>>operation comes from the owner of the file. NFSv3 clients are then
> >>>>>>careful to perform necessary access checks on open to ensure that this
> >>>>>>owner-override rule doesn't grant too many permissions.
> >>>>>>
> >>>>>>That allows NFSv3 applications to see behavior that's mostly like a
> >>>>>>local filesystem, without opening much of a security hole (since the
> >>>>>>owner could always chmod anyway).
> >>>>>>
> >>>>>>So, knfsd is making this special exception--but gluster (which I believe
> >>>>>>it's exporting in this case, via fuse?)--probably doesn't.... I'm not
> >>>>>>sure what you can do about that.
> >>>>>>
> >>>>>>--b.
> >>>>>
> >>>>--
> >>>>To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> >>>>the body of a message to [email protected]
> >>>>More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>>>
> >>>

2015-12-21 08:48:26

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

On 12/19/2015 01:38 AM, J. Bruce Fields wrote:
> On Fri, Dec 18, 2015 at 10:47:42PM +0530, Soumya Koduri wrote:
>>
>>
>> On 12/18/2015 08:50 PM, J. Bruce Fields wrote:
>>> On Fri, Dec 18, 2015 at 02:13:40PM +0530, Soumya Koduri wrote:
>>>>
>>>>
>>>> On 12/18/2015 06:07 AM, Malahal Naineni wrote:
>>>>> IIRC, permission checks are done in open(). write/read syscalls should
>>>>> NOT do much access checks (at least based on POSIX). This is why once an
>>>>> open is done, you remove permissions for that process, but it should
>>>>> still be able to read/write based on the open flags it did when it
>>>>> opened the file.
>>>>>
>>>>> I don't know all the details of this defect, but gluster seems to be
>>>>> doing what it is supposed to do.
>>>>>
>>>> Right. Thanks for the correction. I assumed the behavior should be
>>>> same for both OPEN+WRITE vs CREATE+WRITE in the below scenario. But
>>>> looks like (from 'man creat') the open() call that creates a
>>>> read-only file may well return a read/write file descriptor, which
>>>> is the reason the following WRITE can succeed.
>>>
>>> I forgot another complication, which is that knsfd actually does a
>>> temporary open before each read or write--I assume that's getting
>>> translated into fuse and gluster open operations?
>>>
>> yes. It is the OPEN done as part of NFS WRITE which fails with
>> EACCESS error (with both NFSv3 and NFSv4 mounts).
>
> Makes sense for v3, but I wouldn't normally expect the extra temporary
> open on v4 WRITEs. Could you share any details?
>
I re-tried the test on v4 mount using Fedora23 machine, acting as both
NFS server and client (Linux#4.2.3-300.fc23.x86_64). Please find the pkt
trace attached.

56 07:23:25.567134 ::1 -> ::1 NFS 288 V4 Call WRITE
StateID: 0xf934 Offset: 0 Len: 7
57 07:23:25.567233 192.168.122.17 -> 192.168.122.202 GlusterFS 188
V330 GETXATTR Call
58 07:23:25.567732 192.168.122.202 -> 192.168.122.17 GlusterFS 112
V330 GETXATTR Reply (Call In 57)
59 07:23:25.567881 192.168.122.17 -> 192.168.122.202 GlusterFS 164
V330 OPEN Call
60 07:23:25.568354 192.168.122.202 -> 192.168.122.17 GlusterFS 116
V330 OPEN Reply (Call In 59)
61 07:23:25.568570 ::1 -> ::1 NFS 144 V4 Reply (Call
In 56) WRITE Status: NFS4ERR_ACCESS

Thanks,
Soumya

> --b.
>
>>
>> 63 16:59:09.278651000 ::1 -> ::1 NFS 232 V3 WRITE
>> Call, FH: 0x49a35e54 Offset: 0 Len: 7 FILE_SYNC
>> 64 16:59:09.278926000 192.168.122.1 -> 192.168.122.202 GlusterFS
>> 164 V330 OPEN Call
>> 65 16:59:09.278937000 192.168.122.1 -> 192.168.122.202 GlusterFS
>> 164 [RPC retransmission of #64][TCP Retransmission] V330 OPEN Call
>> 66 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
>> 116 V330 OPEN Reply (Call In 64)
>> 67 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
>> 116 [RPC duplicate of #66][TCP Retransmission] V330 OPEN Reply (Call
>> In 64)
>> 68 16:59:09.279733000 ::1 -> ::1 NFS 212 V3 WRITE
>> Reply (Call In 63) Error: NFS3ERR_ACCES
>>
>>
>> Thanks,
>> Soumya
>>
>>> In which case it might be worth experimenting with NFSv4 or with Jeff
>>> Layton's filehandle-caching patches. Neither's a real fix, but that
>>> could help confirm whether it's the temporary opens that are a problem.
>>>
>>> --b.
>>>
>>>>
>>>> Thanks,
>>>> Soumya
>>>>
>>>>
>>>>> Regards, Malahal.
>>>>>
>>>>> Soumya Koduri [[email protected]] wrote:
>>>>>> As mentioned by Bruce, GlusterFS doesn't have owner-override rule
>>>>>> except for setattr.
>>>>>>
>>>>>> I did few experiments to check why this test case passes on plain
>>>>>> glusterfs fuse mount & NFS-Ganesha but fails with kernel-NFS.
>>>>>>
>>>>>> NFS-Ganesha (for most of the FSALs) seem to be passing the actual
>>>>>> request credentials to the back-end filesystem only for
>>>>>> CREATE(-like) and UNLINK fops. For all the remaining fops, it does
>>>>>> the access check at its end and then perform the operation with root
>>>>>> credentials. That's the reason WRITE succeeded in your case as
>>>>>> NFS-Ganesha (like kernel-NFS) skipped the access check if the
>>>>>> request caller_uid proved to be the file's owner.
>>>>>>
>>>>>> In case of native GlusterFS FUSE mount, there is no OPEN fop
>>>>>> involved. WRITE is performed on the fd returned by CREATE. And
>>>>>> strangely GlusterFS seem to be doing certain access checks only
>>>>>> during OPEN but not for WRITE (this seems like a bug and probably
>>>>>> needs to be fixed in Gluster).
>>>>>>
>>>>>> Thanks,
>>>>>> Soumya
>>>>>>
>>>>>> On 12/14/2015 10:27 PM, Omar Walid Llorente wrote:
>>>>>>>
>>>>>>> Thank you Bruce, others, for the responses. I send attached a complete
>>>>>>> capture of the issue, including the glusterfs transactions.
>>>>>>>
>>>>>>> Hope this helps to clear where may it be...
>>>>>>>
>>>>>>> Omar
>>>>>>>
>>>>>>> El 10/12/15 a las 15:44, J. Bruce Fields escribió:
>>>>>>>> On Thu, Dec 10, 2015 at 05:59:33PM +0530, Soumya Koduri wrote:
>>>>>>>>>
>>>>>>>>> On 12/10/2015 04:02 PM, Omar Walid Llorente wrote:
>>>>>>>>>> Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs
>>>>>>>>>> packets (you can find them in attached file nfs-problem-nks.pcap.zip).
>>>>>>>>>> Sorry for being so late.
>>>>>>>>>>
>>>>>>>>>> What I did was the following:
>>>>>>>>>>
>>>>>>>>>> 1st) Create the RO file:
>>>>>>>>>> cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
>>>>>>>>>> chmod 444 444.txt;
>>>>>>>>>>
>>>>>>>>>> 2nd) Init the capture:
>>>>>>>>>> root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
>>>>>>>>>> tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
>>>>>>>>>> 512 bytes
>>>>>>>>>>
>>>>>>>>> GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It
>>>>>>>>> may be helpful to see what GlusterFS operations are being processed
>>>>>>>>> as part of NFS WRITE call (which has failed in this case).
>>>>>>>>>
>>>>>>>>> Could you please try taking the packet trace on the machine where
>>>>>>>>> NFS server is running (without filtering out based on the port
>>>>>>>>> number).
>>>>>>>>>
>>>>>>>>> Also I tried out the same test on Fedora22 machine, but haven't run
>>>>>>>>> into any issue. What are the fuse mount options you have used to
>>>>>>>>> mount gluster volume?
>>>>>>>> Oh, I think this is a simple problem (but maybe hard to fix). The
>>>>>>>> capture shows NFSv3 traffic like:
>>>>>>>>
>>>>>>>> CREATE -> OK
>>>>>>>> SETATTR (mode set to 0400) -> OK
>>>>>>>> WRITE -> NFS3ERR_ACCES
>>>>>>>>
>>>>>>>> That write would succeed locally (because the mode doesn't matter to a
>>>>>>>> local application that already holds the file open). It would fail over
>>>>>>>> NFSv3, which doesn't know about the open--except that there's a hack for
>>>>>>>> this case: NFSv3 servers allow IO operations to ignore the mode, if the
>>>>>>>> operation comes from the owner of the file. NFSv3 clients are then
>>>>>>>> careful to perform necessary access checks on open to ensure that this
>>>>>>>> owner-override rule doesn't grant too many permissions.
>>>>>>>>
>>>>>>>> That allows NFSv3 applications to see behavior that's mostly like a
>>>>>>>> local filesystem, without opening much of a security hole (since the
>>>>>>>> owner could always chmod anyway).
>>>>>>>>
>>>>>>>> So, knfsd is making this special exception--but gluster (which I believe
>>>>>>>> it's exporting in this case, via fuse?)--probably doesn't.... I'm not
>>>>>>>> sure what you can do about that.
>>>>>>>>
>>>>>>>> --b.
>>>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>>>>> the body of a message to [email protected]
>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>

Attachments:

nfs_v4_mount_+_glusterfs.pcap (21.55 kB)

2015-12-21 16:47:54

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

On Mon, Dec 21, 2015 at 02:18:20PM +0530, Soumya Koduri wrote:
>
>
> On 12/19/2015 01:38 AM, J. Bruce Fields wrote:
> >On Fri, Dec 18, 2015 at 10:47:42PM +0530, Soumya Koduri wrote:
> >>
> >>
> >>On 12/18/2015 08:50 PM, J. Bruce Fields wrote:
> >>>On Fri, Dec 18, 2015 at 02:13:40PM +0530, Soumya Koduri wrote:
> >>>>
> >>>>
> >>>>On 12/18/2015 06:07 AM, Malahal Naineni wrote:
> >>>>>IIRC, permission checks are done in open(). write/read syscalls should
> >>>>>NOT do much access checks (at least based on POSIX). This is why once an
> >>>>>open is done, you remove permissions for that process, but it should
> >>>>>still be able to read/write based on the open flags it did when it
> >>>>>opened the file.
> >>>>>
> >>>>>I don't know all the details of this defect, but gluster seems to be
> >>>>>doing what it is supposed to do.
> >>>>>
> >>>>Right. Thanks for the correction. I assumed the behavior should be
> >>>>same for both OPEN+WRITE vs CREATE+WRITE in the below scenario. But
> >>>>looks like (from 'man creat') the open() call that creates a
> >>>>read-only file may well return a read/write file descriptor, which
> >>>>is the reason the following WRITE can succeed.
> >>>
> >>>I forgot another complication, which is that knsfd actually does a
> >>>temporary open before each read or write--I assume that's getting
> >>>translated into fuse and gluster open operations?
> >>>
> >>yes. It is the OPEN done as part of NFS WRITE which fails with
> >>EACCESS error (with both NFSv3 and NFSv4 mounts).
> >
> >Makes sense for v3, but I wouldn't normally expect the extra temporary
> >open on v4 WRITEs. Could you share any details?
> >
> I re-tried the test on v4 mount using Fedora23 machine, acting as
> both NFS server and client (Linux#4.2.3-300.fc23.x86_64). Please
> find the pkt trace attached.
>
> 56 07:23:25.567134 ::1 -> ::1 NFS 288 V4 Call
> WRITE StateID: 0xf934 Offset: 0 Len: 7
> 57 07:23:25.567233 192.168.122.17 -> 192.168.122.202 GlusterFS 188
> V330 GETXATTR Call
> 58 07:23:25.567732 192.168.122.202 -> 192.168.122.17 GlusterFS 112
> V330 GETXATTR Reply (Call In 57)
> 59 07:23:25.567881 192.168.122.17 -> 192.168.122.202 GlusterFS 164
> V330 OPEN Call

Remind me what kernel version your server is on?

--b.

> 60 07:23:25.568354 192.168.122.202 -> 192.168.122.17 GlusterFS 116
> V330 OPEN Reply (Call In 59)
> 61 07:23:25.568570 ::1 -> ::1 NFS 144 V4 Reply
> (Call In 56) WRITE Status: NFS4ERR_ACCESS
>
> Thanks,
> Soumya
>
> >--b.
> >
> >>
> >> 63 16:59:09.278651000 ::1 -> ::1 NFS 232 V3 WRITE
> >>Call, FH: 0x49a35e54 Offset: 0 Len: 7 FILE_SYNC
> >> 64 16:59:09.278926000 192.168.122.1 -> 192.168.122.202 GlusterFS
> >>164 V330 OPEN Call
> >> 65 16:59:09.278937000 192.168.122.1 -> 192.168.122.202 GlusterFS
> >>164 [RPC retransmission of #64][TCP Retransmission] V330 OPEN Call
> >> 66 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
> >>116 V330 OPEN Reply (Call In 64)
> >> 67 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
> >>116 [RPC duplicate of #66][TCP Retransmission] V330 OPEN Reply (Call
> >>In 64)
> >> 68 16:59:09.279733000 ::1 -> ::1 NFS 212 V3 WRITE
> >>Reply (Call In 63) Error: NFS3ERR_ACCES
> >>
> >>
> >>Thanks,
> >>Soumya
> >>
> >>>In which case it might be worth experimenting with NFSv4 or with Jeff
> >>>Layton's filehandle-caching patches. Neither's a real fix, but that
> >>>could help confirm whether it's the temporary opens that are a problem.
> >>>
> >>>--b.
> >>>
> >>>>
> >>>>Thanks,
> >>>>Soumya
> >>>>
> >>>>
> >>>>>Regards, Malahal.
> >>>>>
> >>>>>Soumya Koduri [[email protected]] wrote:
> >>>>>>As mentioned by Bruce, GlusterFS doesn't have owner-override rule
> >>>>>>except for setattr.
> >>>>>>
> >>>>>>I did few experiments to check why this test case passes on plain
> >>>>>>glusterfs fuse mount & NFS-Ganesha but fails with kernel-NFS.
> >>>>>>
> >>>>>>NFS-Ganesha (for most of the FSALs) seem to be passing the actual
> >>>>>>request credentials to the back-end filesystem only for
> >>>>>>CREATE(-like) and UNLINK fops. For all the remaining fops, it does
> >>>>>>the access check at its end and then perform the operation with root
> >>>>>>credentials. That's the reason WRITE succeeded in your case as
> >>>>>>NFS-Ganesha (like kernel-NFS) skipped the access check if the
> >>>>>>request caller_uid proved to be the file's owner.
> >>>>>>
> >>>>>>In case of native GlusterFS FUSE mount, there is no OPEN fop
> >>>>>>involved. WRITE is performed on the fd returned by CREATE. And
> >>>>>>strangely GlusterFS seem to be doing certain access checks only
> >>>>>>during OPEN but not for WRITE (this seems like a bug and probably
> >>>>>>needs to be fixed in Gluster).
> >>>>>>
> >>>>>>Thanks,
> >>>>>>Soumya
> >>>>>>
> >>>>>>On 12/14/2015 10:27 PM, Omar Walid Llorente wrote:
> >>>>>>>
> >>>>>>>Thank you Bruce, others, for the responses. I send attached a complete
> >>>>>>>capture of the issue, including the glusterfs transactions.
> >>>>>>>
> >>>>>>>Hope this helps to clear where may it be...
> >>>>>>>
> >>>>>>>Omar
> >>>>>>>
> >>>>>>>El 10/12/15 a las 15:44, J. Bruce Fields escribió:
> >>>>>>>>On Thu, Dec 10, 2015 at 05:59:33PM +0530, Soumya Koduri wrote:
> >>>>>>>>>
> >>>>>>>>>On 12/10/2015 04:02 PM, Omar Walid Llorente wrote:
> >>>>>>>>>>Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs
> >>>>>>>>>>packets (you can find them in attached file nfs-problem-nks.pcap.zip).
> >>>>>>>>>>Sorry for being so late.
> >>>>>>>>>>
> >>>>>>>>>>What I did was the following:
> >>>>>>>>>>
> >>>>>>>>>>1st) Create the RO file:
> >>>>>>>>>>cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
> >>>>>>>>>>chmod 444 444.txt;
> >>>>>>>>>>
> >>>>>>>>>>2nd) Init the capture:
> >>>>>>>>>>root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
> >>>>>>>>>>tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
> >>>>>>>>>>512 bytes
> >>>>>>>>>>
> >>>>>>>>>GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It
> >>>>>>>>>may be helpful to see what GlusterFS operations are being processed
> >>>>>>>>>as part of NFS WRITE call (which has failed in this case).
> >>>>>>>>>
> >>>>>>>>>Could you please try taking the packet trace on the machine where
> >>>>>>>>>NFS server is running (without filtering out based on the port
> >>>>>>>>>number).
> >>>>>>>>>
> >>>>>>>>>Also I tried out the same test on Fedora22 machine, but haven't run
> >>>>>>>>>into any issue. What are the fuse mount options you have used to
> >>>>>>>>>mount gluster volume?
> >>>>>>>>Oh, I think this is a simple problem (but maybe hard to fix). The
> >>>>>>>>capture shows NFSv3 traffic like:
> >>>>>>>>
> >>>>>>>> CREATE -> OK
> >>>>>>>> SETATTR (mode set to 0400) -> OK
> >>>>>>>> WRITE -> NFS3ERR_ACCES
> >>>>>>>>
> >>>>>>>>That write would succeed locally (because the mode doesn't matter to a
> >>>>>>>>local application that already holds the file open). It would fail over
> >>>>>>>>NFSv3, which doesn't know about the open--except that there's a hack for
> >>>>>>>>this case: NFSv3 servers allow IO operations to ignore the mode, if the
> >>>>>>>>operation comes from the owner of the file. NFSv3 clients are then
> >>>>>>>>careful to perform necessary access checks on open to ensure that this
> >>>>>>>>owner-override rule doesn't grant too many permissions.
> >>>>>>>>
> >>>>>>>>That allows NFSv3 applications to see behavior that's mostly like a
> >>>>>>>>local filesystem, without opening much of a security hole (since the
> >>>>>>>>owner could always chmod anyway).
> >>>>>>>>
> >>>>>>>>So, knfsd is making this special exception--but gluster (which I believe
> >>>>>>>>it's exporting in this case, via fuse?)--probably doesn't.... I'm not
> >>>>>>>>sure what you can do about that.
> >>>>>>>>
> >>>>>>>>--b.
> >>>>>>>
> >>>>>>--
> >>>>>>To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> >>>>>>the body of a message to [email protected]
> >>>>>>More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>>>>>
> >>>>>
> >--
> >To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> >the body of a message to [email protected]
> >More majordomo info at http://vger.kernel.org/majordomo-info.html
> >

2015-12-21 17:58:43

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

On 12/21/2015 10:17 PM, J. Bruce Fields wrote:
> On Mon, Dec 21, 2015 at 02:18:20PM +0530, Soumya Koduri wrote:
>>
>>
>> On 12/19/2015 01:38 AM, J. Bruce Fields wrote:
>>> On Fri, Dec 18, 2015 at 10:47:42PM +0530, Soumya Koduri wrote:
>>>>
>>>>
>>>> On 12/18/2015 08:50 PM, J. Bruce Fields wrote:
>>>>> On Fri, Dec 18, 2015 at 02:13:40PM +0530, Soumya Koduri wrote:
>>>>>>
>>>>>>
>>>>>> On 12/18/2015 06:07 AM, Malahal Naineni wrote:
>>>>>>> IIRC, permission checks are done in open(). write/read syscalls should
>>>>>>> NOT do much access checks (at least based on POSIX). This is why once an
>>>>>>> open is done, you remove permissions for that process, but it should
>>>>>>> still be able to read/write based on the open flags it did when it
>>>>>>> opened the file.
>>>>>>>
>>>>>>> I don't know all the details of this defect, but gluster seems to be
>>>>>>> doing what it is supposed to do.
>>>>>>>
>>>>>> Right. Thanks for the correction. I assumed the behavior should be
>>>>>> same for both OPEN+WRITE vs CREATE+WRITE in the below scenario. But
>>>>>> looks like (from 'man creat') the open() call that creates a
>>>>>> read-only file may well return a read/write file descriptor, which
>>>>>> is the reason the following WRITE can succeed.
>>>>>
>>>>> I forgot another complication, which is that knsfd actually does a
>>>>> temporary open before each read or write--I assume that's getting
>>>>> translated into fuse and gluster open operations?
>>>>>
>>>> yes. It is the OPEN done as part of NFS WRITE which fails with
>>>> EACCESS error (with both NFSv3 and NFSv4 mounts).
>>>
>>> Makes sense for v3, but I wouldn't normally expect the extra temporary
>>> open on v4 WRITEs. Could you share any details?
>>>
>> I re-tried the test on v4 mount using Fedora23 machine, acting as
>> both NFS server and client (Linux#4.2.3-300.fc23.x86_64). Please
>> find the pkt trace attached.
>>
>> 56 07:23:25.567134 ::1 -> ::1 NFS 288 V4 Call
>> WRITE StateID: 0xf934 Offset: 0 Len: 7
>> 57 07:23:25.567233 192.168.122.17 -> 192.168.122.202 GlusterFS 188
>> V330 GETXATTR Call
>> 58 07:23:25.567732 192.168.122.202 -> 192.168.122.17 GlusterFS 112
>> V330 GETXATTR Reply (Call In 57)
>> 59 07:23:25.567881 192.168.122.17 -> 192.168.122.202 GlusterFS 164
>> V330 OPEN Call
>
> Remind me what kernel version your server is on?

NFS server is on fedora23 VM - Linux version 4.2.3-300.fc23.x86_64

Thanks,
Soumya

>
> --b.
>
>
>> 60 07:23:25.568354 192.168.122.202 -> 192.168.122.17 GlusterFS 116
>> V330 OPEN Reply (Call In 59)
>> 61 07:23:25.568570 ::1 -> ::1 NFS 144 V4 Reply
>> (Call In 56) WRITE Status: NFS4ERR_ACCESS
>>
>> Thanks,
>> Soumya
>>
>>> --b.
>>>
>>>>
>>>> 63 16:59:09.278651000 ::1 -> ::1 NFS 232 V3 WRITE
>>>> Call, FH: 0x49a35e54 Offset: 0 Len: 7 FILE_SYNC
>>>> 64 16:59:09.278926000 192.168.122.1 -> 192.168.122.202 GlusterFS
>>>> 164 V330 OPEN Call
>>>> 65 16:59:09.278937000 192.168.122.1 -> 192.168.122.202 GlusterFS
>>>> 164 [RPC retransmission of #64][TCP Retransmission] V330 OPEN Call
>>>> 66 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
>>>> 116 V330 OPEN Reply (Call In 64)
>>>> 67 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
>>>> 116 [RPC duplicate of #66][TCP Retransmission] V330 OPEN Reply (Call
>>>> In 64)
>>>> 68 16:59:09.279733000 ::1 -> ::1 NFS 212 V3 WRITE
>>>> Reply (Call In 63) Error: NFS3ERR_ACCES
>>>>
>>>>
>>>> Thanks,
>>>> Soumya
>>>>
>>>>> In which case it might be worth experimenting with NFSv4 or with Jeff
>>>>> Layton's filehandle-caching patches. Neither's a real fix, but that
>>>>> could help confirm whether it's the temporary opens that are a problem.
>>>>>
>>>>> --b.
>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Soumya
>>>>>>
>>>>>>
>>>>>>> Regards, Malahal.
>>>>>>>
>>>>>>> Soumya Koduri [[email protected]] wrote:
>>>>>>>> As mentioned by Bruce, GlusterFS doesn't have owner-override rule
>>>>>>>> except for setattr.
>>>>>>>>
>>>>>>>> I did few experiments to check why this test case passes on plain
>>>>>>>> glusterfs fuse mount & NFS-Ganesha but fails with kernel-NFS.
>>>>>>>>
>>>>>>>> NFS-Ganesha (for most of the FSALs) seem to be passing the actual
>>>>>>>> request credentials to the back-end filesystem only for
>>>>>>>> CREATE(-like) and UNLINK fops. For all the remaining fops, it does
>>>>>>>> the access check at its end and then perform the operation with root
>>>>>>>> credentials. That's the reason WRITE succeeded in your case as
>>>>>>>> NFS-Ganesha (like kernel-NFS) skipped the access check if the
>>>>>>>> request caller_uid proved to be the file's owner.
>>>>>>>>
>>>>>>>> In case of native GlusterFS FUSE mount, there is no OPEN fop
>>>>>>>> involved. WRITE is performed on the fd returned by CREATE. And
>>>>>>>> strangely GlusterFS seem to be doing certain access checks only
>>>>>>>> during OPEN but not for WRITE (this seems like a bug and probably
>>>>>>>> needs to be fixed in Gluster).
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Soumya
>>>>>>>>
>>>>>>>> On 12/14/2015 10:27 PM, Omar Walid Llorente wrote:
>>>>>>>>>
>>>>>>>>> Thank you Bruce, others, for the responses. I send attached a complete
>>>>>>>>> capture of the issue, including the glusterfs transactions.
>>>>>>>>>
>>>>>>>>> Hope this helps to clear where may it be...
>>>>>>>>>
>>>>>>>>> Omar
>>>>>>>>>
>>>>>>>>> El 10/12/15 a las 15:44, J. Bruce Fields escribió:
>>>>>>>>>> On Thu, Dec 10, 2015 at 05:59:33PM +0530, Soumya Koduri wrote:
>>>>>>>>>>>
>>>>>>>>>>> On 12/10/2015 04:02 PM, Omar Walid Llorente wrote:
>>>>>>>>>>>> Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs
>>>>>>>>>>>> packets (you can find them in attached file nfs-problem-nks.pcap.zip).
>>>>>>>>>>>> Sorry for being so late.
>>>>>>>>>>>>
>>>>>>>>>>>> What I did was the following:
>>>>>>>>>>>>
>>>>>>>>>>>> 1st) Create the RO file:
>>>>>>>>>>>> cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
>>>>>>>>>>>> chmod 444 444.txt;
>>>>>>>>>>>>
>>>>>>>>>>>> 2nd) Init the capture:
>>>>>>>>>>>> root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
>>>>>>>>>>>> tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
>>>>>>>>>>>> 512 bytes
>>>>>>>>>>>>
>>>>>>>>>>> GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It
>>>>>>>>>>> may be helpful to see what GlusterFS operations are being processed
>>>>>>>>>>> as part of NFS WRITE call (which has failed in this case).
>>>>>>>>>>>
>>>>>>>>>>> Could you please try taking the packet trace on the machine where
>>>>>>>>>>> NFS server is running (without filtering out based on the port
>>>>>>>>>>> number).
>>>>>>>>>>>
>>>>>>>>>>> Also I tried out the same test on Fedora22 machine, but haven't run
>>>>>>>>>>> into any issue. What are the fuse mount options you have used to
>>>>>>>>>>> mount gluster volume?
>>>>>>>>>> Oh, I think this is a simple problem (but maybe hard to fix). The
>>>>>>>>>> capture shows NFSv3 traffic like:
>>>>>>>>>>
>>>>>>>>>> CREATE -> OK
>>>>>>>>>> SETATTR (mode set to 0400) -> OK
>>>>>>>>>> WRITE -> NFS3ERR_ACCES
>>>>>>>>>>
>>>>>>>>>> That write would succeed locally (because the mode doesn't matter to a
>>>>>>>>>> local application that already holds the file open). It would fail over
>>>>>>>>>> NFSv3, which doesn't know about the open--except that there's a hack for
>>>>>>>>>> this case: NFSv3 servers allow IO operations to ignore the mode, if the
>>>>>>>>>> operation comes from the owner of the file. NFSv3 clients are then
>>>>>>>>>> careful to perform necessary access checks on open to ensure that this
>>>>>>>>>> owner-override rule doesn't grant too many permissions.
>>>>>>>>>>
>>>>>>>>>> That allows NFSv3 applications to see behavior that's mostly like a
>>>>>>>>>> local filesystem, without opening much of a security hole (since the
>>>>>>>>>> owner could always chmod anyway).
>>>>>>>>>>
>>>>>>>>>> So, knfsd is making this special exception--but gluster (which I believe
>>>>>>>>>> it's exporting in this case, via fuse?)--probably doesn't.... I'm not
>>>>>>>>>> sure what you can do about that.
>>>>>>>>>>
>>>>>>>>>> --b.
>>>>>>>>>
>>>>>>>> --
>>>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>>>>>>> the body of a message to [email protected]
>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>>>
>>>>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>> the body of a message to [email protected]
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>
>

2015-12-21 20:14:06

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

On Mon, Dec 21, 2015 at 11:28:36PM +0530, Soumya Koduri wrote:
>
>
> On 12/21/2015 10:17 PM, J. Bruce Fields wrote:
> >On Mon, Dec 21, 2015 at 02:18:20PM +0530, Soumya Koduri wrote:
> >>
> >>
> >>On 12/19/2015 01:38 AM, J. Bruce Fields wrote:
> >>>On Fri, Dec 18, 2015 at 10:47:42PM +0530, Soumya Koduri wrote:
> >>>>
> >>>>
> >>>>On 12/18/2015 08:50 PM, J. Bruce Fields wrote:
> >>>>>On Fri, Dec 18, 2015 at 02:13:40PM +0530, Soumya Koduri wrote:
> >>>>>>
> >>>>>>
> >>>>>>On 12/18/2015 06:07 AM, Malahal Naineni wrote:
> >>>>>>>IIRC, permission checks are done in open(). write/read syscalls should
> >>>>>>>NOT do much access checks (at least based on POSIX). This is why once an
> >>>>>>>open is done, you remove permissions for that process, but it should
> >>>>>>>still be able to read/write based on the open flags it did when it
> >>>>>>>opened the file.
> >>>>>>>
> >>>>>>>I don't know all the details of this defect, but gluster seems to be
> >>>>>>>doing what it is supposed to do.
> >>>>>>>
> >>>>>>Right. Thanks for the correction. I assumed the behavior should be
> >>>>>>same for both OPEN+WRITE vs CREATE+WRITE in the below scenario. But
> >>>>>>looks like (from 'man creat') the open() call that creates a
> >>>>>>read-only file may well return a read/write file descriptor, which
> >>>>>>is the reason the following WRITE can succeed.
> >>>>>
> >>>>>I forgot another complication, which is that knsfd actually does a
> >>>>>temporary open before each read or write--I assume that's getting
> >>>>>translated into fuse and gluster open operations?
> >>>>>
> >>>>yes. It is the OPEN done as part of NFS WRITE which fails with
> >>>>EACCESS error (with both NFSv3 and NFSv4 mounts).
> >>>
> >>>Makes sense for v3, but I wouldn't normally expect the extra temporary
> >>>open on v4 WRITEs. Could you share any details?
> >>>
> >>I re-tried the test on v4 mount using Fedora23 machine, acting as
> >>both NFS server and client (Linux#4.2.3-300.fc23.x86_64). Please
> >>find the pkt trace attached.
> >>
> >> 56 07:23:25.567134 ::1 -> ::1 NFS 288 V4 Call
> >>WRITE StateID: 0xf934 Offset: 0 Len: 7
> >> 57 07:23:25.567233 192.168.122.17 -> 192.168.122.202 GlusterFS 188
> >>V330 GETXATTR Call
> >> 58 07:23:25.567732 192.168.122.202 -> 192.168.122.17 GlusterFS 112
> >>V330 GETXATTR Reply (Call In 57)
> >> 59 07:23:25.567881 192.168.122.17 -> 192.168.122.202 GlusterFS 164
> >>V330 OPEN Call
> >
> >Remind me what kernel version your server is on?
>
> NFS server is on fedora23 VM - Linux version 4.2.3-300.fc23.x86_64

We did reshuffle the code that does the temporary open on WRITE around
there--but it looks right to me, and I can't reproduce an open on WRITE
on that kernel myself.

Maybe there's some further fuse or gluster debugging that would help
show where that open is coming from.

--b.

>
> Thanks,
> Soumya
>
> >
> >--b.
> >
> >
> >> 60 07:23:25.568354 192.168.122.202 -> 192.168.122.17 GlusterFS 116
> >>V330 OPEN Reply (Call In 59)
> >> 61 07:23:25.568570 ::1 -> ::1 NFS 144 V4 Reply
> >>(Call In 56) WRITE Status: NFS4ERR_ACCESS
> >>
> >>Thanks,
> >>Soumya
> >>
> >>>--b.
> >>>
> >>>>
> >>>> 63 16:59:09.278651000 ::1 -> ::1 NFS 232 V3 WRITE
> >>>>Call, FH: 0x49a35e54 Offset: 0 Len: 7 FILE_SYNC
> >>>> 64 16:59:09.278926000 192.168.122.1 -> 192.168.122.202 GlusterFS
> >>>>164 V330 OPEN Call
> >>>> 65 16:59:09.278937000 192.168.122.1 -> 192.168.122.202 GlusterFS
> >>>>164 [RPC retransmission of #64][TCP Retransmission] V330 OPEN Call
> >>>> 66 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
> >>>>116 V330 OPEN Reply (Call In 64)
> >>>> 67 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
> >>>>116 [RPC duplicate of #66][TCP Retransmission] V330 OPEN Reply (Call
> >>>>In 64)
> >>>> 68 16:59:09.279733000 ::1 -> ::1 NFS 212 V3 WRITE
> >>>>Reply (Call In 63) Error: NFS3ERR_ACCES
> >>>>
> >>>>
> >>>>Thanks,
> >>>>Soumya
> >>>>
> >>>>>In which case it might be worth experimenting with NFSv4 or with Jeff
> >>>>>Layton's filehandle-caching patches. Neither's a real fix, but that
> >>>>>could help confirm whether it's the temporary opens that are a problem.
> >>>>>
> >>>>>--b.
> >>>>>
> >>>>>>
> >>>>>>Thanks,
> >>>>>>Soumya
> >>>>>>
> >>>>>>
> >>>>>>>Regards, Malahal.
> >>>>>>>
> >>>>>>>Soumya Koduri [[email protected]] wrote:
> >>>>>>>>As mentioned by Bruce, GlusterFS doesn't have owner-override rule
> >>>>>>>>except for setattr.
> >>>>>>>>
> >>>>>>>>I did few experiments to check why this test case passes on plain
> >>>>>>>>glusterfs fuse mount & NFS-Ganesha but fails with kernel-NFS.
> >>>>>>>>
> >>>>>>>>NFS-Ganesha (for most of the FSALs) seem to be passing the actual
> >>>>>>>>request credentials to the back-end filesystem only for
> >>>>>>>>CREATE(-like) and UNLINK fops. For all the remaining fops, it does
> >>>>>>>>the access check at its end and then perform the operation with root
> >>>>>>>>credentials. That's the reason WRITE succeeded in your case as
> >>>>>>>>NFS-Ganesha (like kernel-NFS) skipped the access check if the
> >>>>>>>>request caller_uid proved to be the file's owner.
> >>>>>>>>
> >>>>>>>>In case of native GlusterFS FUSE mount, there is no OPEN fop
> >>>>>>>>involved. WRITE is performed on the fd returned by CREATE. And
> >>>>>>>>strangely GlusterFS seem to be doing certain access checks only
> >>>>>>>>during OPEN but not for WRITE (this seems like a bug and probably
> >>>>>>>>needs to be fixed in Gluster).
> >>>>>>>>
> >>>>>>>>Thanks,
> >>>>>>>>Soumya
> >>>>>>>>
> >>>>>>>>On 12/14/2015 10:27 PM, Omar Walid Llorente wrote:
> >>>>>>>>>
> >>>>>>>>>Thank you Bruce, others, for the responses. I send attached a complete
> >>>>>>>>>capture of the issue, including the glusterfs transactions.
> >>>>>>>>>
> >>>>>>>>>Hope this helps to clear where may it be...
> >>>>>>>>>
> >>>>>>>>>Omar
> >>>>>>>>>
> >>>>>>>>>El 10/12/15 a las 15:44, J. Bruce Fields escribió:
> >>>>>>>>>>On Thu, Dec 10, 2015 at 05:59:33PM +0530, Soumya Koduri wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>On 12/10/2015 04:02 PM, Omar Walid Llorente wrote:
> >>>>>>>>>>>>Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs
> >>>>>>>>>>>>packets (you can find them in attached file nfs-problem-nks.pcap.zip).
> >>>>>>>>>>>>Sorry for being so late.
> >>>>>>>>>>>>
> >>>>>>>>>>>>What I did was the following:
> >>>>>>>>>>>>
> >>>>>>>>>>>>1st) Create the RO file:
> >>>>>>>>>>>>cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
> >>>>>>>>>>>>chmod 444 444.txt;
> >>>>>>>>>>>>
> >>>>>>>>>>>>2nd) Init the capture:
> >>>>>>>>>>>>root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
> >>>>>>>>>>>>tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
> >>>>>>>>>>>>512 bytes
> >>>>>>>>>>>>
> >>>>>>>>>>>GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It
> >>>>>>>>>>>may be helpful to see what GlusterFS operations are being processed
> >>>>>>>>>>>as part of NFS WRITE call (which has failed in this case).
> >>>>>>>>>>>
> >>>>>>>>>>>Could you please try taking the packet trace on the machine where
> >>>>>>>>>>>NFS server is running (without filtering out based on the port
> >>>>>>>>>>>number).
> >>>>>>>>>>>
> >>>>>>>>>>>Also I tried out the same test on Fedora22 machine, but haven't run
> >>>>>>>>>>>into any issue. What are the fuse mount options you have used to
> >>>>>>>>>>>mount gluster volume?
> >>>>>>>>>>Oh, I think this is a simple problem (but maybe hard to fix). The
> >>>>>>>>>>capture shows NFSv3 traffic like:
> >>>>>>>>>>
> >>>>>>>>>> CREATE -> OK
> >>>>>>>>>> SETATTR (mode set to 0400) -> OK
> >>>>>>>>>> WRITE -> NFS3ERR_ACCES
> >>>>>>>>>>
> >>>>>>>>>>That write would succeed locally (because the mode doesn't matter to a
> >>>>>>>>>>local application that already holds the file open). It would fail over
> >>>>>>>>>>NFSv3, which doesn't know about the open--except that there's a hack for
> >>>>>>>>>>this case: NFSv3 servers allow IO operations to ignore the mode, if the
> >>>>>>>>>>operation comes from the owner of the file. NFSv3 clients are then
> >>>>>>>>>>careful to perform necessary access checks on open to ensure that this
> >>>>>>>>>>owner-override rule doesn't grant too many permissions.
> >>>>>>>>>>
> >>>>>>>>>>That allows NFSv3 applications to see behavior that's mostly like a
> >>>>>>>>>>local filesystem, without opening much of a security hole (since the
> >>>>>>>>>>owner could always chmod anyway).
> >>>>>>>>>>
> >>>>>>>>>>So, knfsd is making this special exception--but gluster (which I believe
> >>>>>>>>>>it's exporting in this case, via fuse?)--probably doesn't.... I'm not
> >>>>>>>>>>sure what you can do about that.
> >>>>>>>>>>
> >>>>>>>>>>--b.
> >>>>>>>>>
> >>>>>>>>--
> >>>>>>>>To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> >>>>>>>>the body of a message to [email protected]
> >>>>>>>>More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>>>>>>>
> >>>>>>>
> >>>--
> >>>To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> >>>the body of a message to [email protected]
> >>>More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>>
> >
> >

2016-11-08 20:16:12

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

On Wed, Oct 19, 2016 at 07:53:37PM +0200, Omar Walid Llorente wrote:
> Resending because it has not been accepted by
> [email protected] because of the HTML... Sorry if you get it
> twice.
>
> Thanks again.
>
> Omar
>
> --- Forwarded message ---
>
> Hi Bruce, others.

Sory for the delayed response. This is a fairly complicated setup that
may require people with nfs, gluster, and zfs experience to debug.

The server does need to be able to bypass certain permission checks on
open so that it can open a newly created read-only file for write.
Probably one of those lower layers is insisting on permission checks
where it shouldn't. That's all I can say from a quick look.

--b.

>
> I come back to this issue again in the hope that these evidences I
> bring can help to give some light to the problem.
>
> We have exactly the same environment that last year when I came to
> you, but with upgraded versions of gluster (3.8) and
> nfs-kernel-server (1.2.8) this time over an updated Ubuntu 16.04.
>
> (NOTE: Environment: glusterfs-cluster made of ZFS datasets exports
> via glusterfs a distributed volume + nfs-kernel server mounts via
> fuse that glusterfs volume and reexports it via nfs-v3 + nfs-common
> client mounts via nfs that nfs exported volume)
>
> The errors I get by the user side are exactly the same, but in the
> server side, I have different messages, which I forward to you.
>
> * /TESTS/ and errors by the user side:
>
> u056@l056:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod
> 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
> cp: cannot create regular file ‘kk.txt’: Permission denied
> ls: cannot access kk.txt: No such file or directory
> -r--r--r-- 1 u056 admincdc 7 oct 19 2016 444.txt
> u056@l056:~$ ls -lrt
> total 33
> ---------- 1 u056 admincdc 0 ene 4 1970 kk.txt
> drwx------ 2 u056 alumno 4096 oct 3 2014 Desktop
> drwxr-xr-x 2 u056 alumno 4096 sep 23 12:20 Público
> drwxr-xr-x 2 u056 alumno 4096 sep 23 12:20 Descargas
> drwxr-xr-x 2 u056 alumno 4096 sep 23 12:20 Documentos
> drwxr-xr-x 2 u056 alumno 4096 sep 23 12:20 Vídeos
> drwxr-xr-x 2 u056 alumno 4096 sep 23 12:20 Música
> drwxr-xr-x 2 u056 alumno 4096 sep 23 12:20 Plantillas
> drwxr-xr-x 2 u056 alumno 4096 sep 23 12:22 Imágenes
> -r--r--r-- 1 u056 admincdc 7 oct 19 17:34 444.txt
>
> u056@l056:~$ mount | grep cuentas09
> cuentas09:/home-3/u056 on /home/u056 type nfs (rw,noatime,intr,fsc,nolock,rsize=262140,wsize=262140,vers=4,addr=138.4.30.80,clientaddr=138.4.31.56)
> u056@l056:~$ df -h | grep cuentas09
> cuentas09:/home-3/u056 40T 55G 40T 1% /home/u056
> u056@l056:~$
>
> * /LOGS/ by the glusterfs client side (server side), versions of the
> software, info of the kernel and related kernel modules:
>
> root@cuentas09-lab:/var/log/glusterfs# tail home-3.log
> [2016-10-19 15:33:38.091300] I [io-stats.c:1574:io_stats_dump_fd]
> 0-home-lab-3: --- fd stats ---
> [2016-10-19 15:33:38.091331] I [io-stats.c:1579:io_stats_dump_fd]
> 0-home-lab-3: Filename : /u056/444.txt
> [2016-10-19 15:33:38.091344] I [io-stats.c:1594:io_stats_dump_fd]
> 0-home-lab-3: BytesWritten : 7 bytes
> [2016-10-19 15:33:38.091355] I [io-stats.c:1606:io_stats_dump_fd]
> 0-home-lab-3: Write 000004b+ : 1
> [2016-10-19 15:34:13.199285] I [io-stats.c:1574:io_stats_dump_fd]
> 0-home-lab-3: --- fd stats ---
> [2016-10-19 15:34:13.200291] I [io-stats.c:1579:io_stats_dump_fd]
> 0-home-lab-3: Filename : /u056/444.txt
> [2016-10-19 15:34:13.200313] I [io-stats.c:1594:io_stats_dump_fd]
> 0-home-lab-3: BytesWritten : 7 bytes
> [2016-10-19 15:34:13.200325] I [io-stats.c:1606:io_stats_dump_fd]
> 0-home-lab-3: Write 000004b+ : 1
> [2016-10-19 15:34:13.211696] E [MSGID: 114031]
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-0:
> remote operation failed. Path: /u056/kk.txt
> (d8fc54a3-f2eb-4538-889a-17afdcfbb255) [Permission denied]
> [2016-10-19 15:34:13.211758] W [fuse-bridge.c:989:fuse_fd_cbk]
> 0-glusterfs-fuse: 630: OPEN() /u056/kk.txt => -1 (Permission denied)
>
> root@cuentas09-lab:/var/log/glusterfs# dpkg -l | grep -e gluster -e nfs
> ii glusterfs-client 3.8.5-ubuntu1~xenial1 amd64
> clustered file-system (client package)
> ii glusterfs-common 3.8.5-ubuntu1~xenial1 amd64
> GlusterFS common libraries and translator modules
> ii libnfsidmap2:amd64 0.25-5 amd64
> NFS idmapping library
> ii nfs-common 1:1.2.8-9ubuntu12 amd64 NFS
> support files common to client and server
> ii nfs-kernel-server 1:1.2.8-9ubuntu12 amd64
> support for NFS kernel server
>
> root@cuentas09-lab:/var/log/glusterfs# uname -a
> Linux cuentas09-lab.lab.dit.upm.es 4.4.0-43-generic #63-Ubuntu SMP
> Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
>
> root@cuentas09-lab:/var/log/glusterfs# lsmod | grep nfs
> nfsd 319488 13
> auth_rpcgss 61440 1 nfsd
> nfs_acl 16384 1 nfsd
> lockd 94208 1 nfsd
> grace 16384 2 nfsd,lockd
> sunrpc 335872 21 nfsd,auth_rpcgss,lockd,nfs_acl
> root@cuentas09-lab:/var/log/glusterfs#
>
> * /LOGS/ by the nfs-kernel server side and basic configuration of
> the nfs-kernel-server:
>
> root@cuentas09-lab:~# less /var/log/kern.log
> root@cuentas09-lab:~# tail -30 !$
> tail -30 /var/log/kern.log
> Oct 19 17:30:24 cuentas09-lab kernel: [ 1573.631498]
> [<ffffffff810a0850>] ? kthread_create_on_node+0x1e0/0x1e0
> Oct 19 17:30:24 cuentas09-lab kernel: [ 1573.631504]
> [<ffffffff81831c4f>] ret_from_fork+0x3f/0x70
> Oct 19 17:30:24 cuentas09-lab kernel: [ 1573.631507]
> [<ffffffff810a0850>] ? kthread_create_on_node+0x1e0/0x1e0
> Oct 19 17:30:24 cuentas09-lab kernel: [ 1573.631510] ---[ end trace
> 34db7650fa22d1d0 ]---
> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.814898] ------------[
> cut here ]------------
> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.814948] WARNING: CPU: 0
> PID: 1373 at /build/linux-BwgxJb/linux-4.4.0/fs/nfsd/nfs4proc.c:464
> nfsd4_open+0x515/0x780 [nfsd]()
> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.814951]
> nfsd4_process_open2 failed to open newly-created file! status=13
> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.814954] Modules linked
> in: vmw_vsock_vmci_transport vsock ppdev vmw_balloon coretemp joydev
> input_leds serio_raw irda 8250_fintek parport_pc shpchp parport
> i2c_piix4 crc_ccitt vmw_vmci mac_hid ib_iser nfsd rdma_cm iw_cm
> ib_cm auth_rpcgss nfs_acl lockd grace ib_sa sunrpc ib_mad ib_core
> ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4
> btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq
> async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath
> linear vmwgfx ttm psmouse drm_kms_helper syscopyarea sysfillrect
> sysimgblt fb_sys_fops mptspi mptscsih mptbase drm e1000
> scsi_transport_spi pata_acpi floppy fjes
> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815018] CPU: 0 PID:
> 1373 Comm: nfsd Tainted: G W 4.4.0-43-generic
> #63-Ubuntu
> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815021] Hardware name:
> VMware, Inc. VMware Virtual Platform/440BX Desktop Reference
> Platform, BIOS 6.00 09/22/2009
> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815024]
> 0000000000000286 00000000c0e912c3 ffff8800b93c7c80 ffffffff813f1f93
> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815028]
> ffff8800b93c7cc8 ffffffffc047b668 ffff8800b93c7cb8 ffffffff81081212
> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815031]
> ffff8800b9e43240 ffff8800b9e44068 000000000d000000 ffff8800b85e6000
> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815035] Call Trace:
> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815047]
> [<ffffffff813f1f93>] dump_stack+0x63/0x90
> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815055]
> [<ffffffff81081212>] warn_slowpath_common+0x82/0xc0
> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815058]
> [<ffffffff810812ac>] warn_slowpath_fmt+0x5c/0x80
> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815073]
> [<ffffffffc04665db>] ? nfs4_free_ol_stateid+0x3b/0x40 [nfsd]
> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815085]
> [<ffffffffc0459d05>] nfsd4_open+0x515/0x780 [nfsd]
> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815098]
> [<ffffffffc045a2fa>] nfsd4_proc_compound+0x38a/0x660 [nfsd]
> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815108]
> [<ffffffffc0446e78>] nfsd_dispatch+0xb8/0x200 [nfsd]
> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815132]
> [<ffffffffc039eeac>] svc_process_common+0x40c/0x650 [sunrpc]
> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815151]
> [<ffffffffc03a0273>] svc_process+0x103/0x1c0 [sunrpc]
> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815160]
> [<ffffffffc04468cf>] nfsd+0xef/0x160 [nfsd]
> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815170]
> [<ffffffffc04467e0>] ? nfsd_destroy+0x60/0x60 [nfsd]
> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815174]
> [<ffffffff810a0928>] kthread+0xd8/0xf0
> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815178]
> [<ffffffff810a0850>] ? kthread_create_on_node+0x1e0/0x1e0
> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815184]
> [<ffffffff81831c4f>] ret_from_fork+0x3f/0x70
> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815187]
> [<ffffffff810a0850>] ? kthread_create_on_node+0x1e0/0x1e0
> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815190] ---[ end trace
> 34db7650fa22d1d1 ]---
> root@cuentas09-lab:~#
>
> root@cuentas09-lab:~# showmount -e
> Export list for cuentas09-lab.lab.dit.upm.es:
> /home-4 138.4.30.0/23
> /home-3 138.4.30.0/23
>
> root@cuentas09-lab:~# cat /etc/exports | grep -v ^#
> /home-3 138.4.30.0/23(rw,fsid=3,insecure,no_subtree_check,async,no_root_squash)
> /home-4 138.4.30.0/23(rw,fsid=4,insecure,no_subtree_check,async,no_root_squash)
> root@cuentas09-lab:~#
>
> root@cuentas09-lab:~# cat /etc/default/nfs-kernel-server | grep -v ^#
> RPCNFSDCOUNT="8 -V 3"
> RPCNFSDPRIORITY=0
> RPCMOUNTDOPTS="--manage-gids"
> NEED_SVCGSSD=""
> RPCSVCGSSDOPTS=""
> RPCNFSDOPTS="-d -V 3"
> root@cuentas09-lab:~#
>
> * LOGS by the glusterfs server side:
>
> # for i in 10 11 12 13; do echo recipiente$i; ssh recipiente$i tail
> /var/log/glusterfs/bricks/data-recipiente$i-gluster-home-lab-3.log;
> done
> recipiente10
> [2016-10-19 15:27:48.172858] I [io-stats.c:1606:io_stats_dump_fd]
> 0-/data/recipiente10-gluster-home-lab-3: Write 000008b+ : 1
> [2016-10-19 15:29:32.626450] I [io-stats.c:1574:io_stats_dump_fd]
> 0-/data/recipiente10-gluster-home-lab-3: --- fd stats ---
> [2016-10-19 15:29:32.626545] I [io-stats.c:1579:io_stats_dump_fd]
> 0-/data/recipiente10-gluster-home-lab-3: Filename :
> /u056/.local/share/tracker/data/tracker-store.journal
> [2016-10-19 15:29:32.626572] I [io-stats.c:1594:io_stats_dump_fd]
> 0-/data/recipiente10-gluster-home-lab-3: BytesWritten : 8 bytes
> [2016-10-19 15:29:32.626594] I [io-stats.c:1606:io_stats_dump_fd]
> 0-/data/recipiente10-gluster-home-lab-3: Write 000008b+ : 1
> [2016-10-19 15:30:23.997592] E [MSGID: 115070]
> [server-rpc-fops.c:1472:server_open_cbk] 0-home-lab-3-server: 372:
> OPEN /u056/kk.txt (cc97d150-c725-42b6-9aa5-50328f281e06) ==>
> (Permission denied) [Permission denied]
> [2016-10-19 15:32:28.423380] I [io-stats.c:1574:io_stats_dump_fd]
> 0-/data/recipiente10-gluster-home-lab-3: --- fd stats ---
> [2016-10-19 15:32:28.423478] I [io-stats.c:1579:io_stats_dump_fd]
> 0-/data/recipiente10-gluster-home-lab-3: Filename :
> /u056/.bashrc
> [2016-10-19 15:32:28.423504] I [io-stats.c:1584:io_stats_dump_fd]
> 0-/data/recipiente10-gluster-home-lab-3: Lifetime : 280secs,
> 232488usecs
> [2016-10-19 15:34:13.176805] E [MSGID: 115070]
> [server-rpc-fops.c:1472:server_open_cbk] 0-home-lab-3-server: 424:
> OPEN /u056/kk.txt (d8fc54a3-f2eb-4538-889a-17afdcfbb255) ==>
> (Permission denied) [Permission denied]
> recipiente11
> [2016-10-19 15:33:38.057524] I [io-stats.c:1579:io_stats_dump_fd]
> 0-/data/recipiente11-gluster-home-lab-3: Filename :
> /u056/444.txt
> [2016-10-19 15:33:38.057548] I [io-stats.c:1594:io_stats_dump_fd]
> 0-/data/recipiente11-gluster-home-lab-3: BytesWritten : 7 bytes
> [2016-10-19 15:33:38.057570] I [io-stats.c:1606:io_stats_dump_fd]
> 0-/data/recipiente11-gluster-home-lab-3: Write 000004b+ : 1
> [2016-10-19 15:34:13.164629] I [io-stats.c:1574:io_stats_dump_fd]
> 0-/data/recipiente11-gluster-home-lab-3: --- fd stats ---
> [2016-10-19 15:34:13.164722] I [io-stats.c:1579:io_stats_dump_fd]
> 0-/data/recipiente11-gluster-home-lab-3: Filename :
> /u056/444.txt
> [2016-10-19 15:34:13.164769] I [io-stats.c:1594:io_stats_dump_fd]
> 0-/data/recipiente11-gluster-home-lab-3: BytesWritten : 7 bytes
> [2016-10-19 15:34:13.164791] I [io-stats.c:1606:io_stats_dump_fd]
> 0-/data/recipiente11-gluster-home-lab-3: Write 000004b+ : 1
> [2016-10-19 15:36:29.066078] I [io-stats.c:1574:io_stats_dump_fd]
> 0-/data/recipiente11-gluster-home-lab-3: --- fd stats ---
> [2016-10-19 15:36:29.066180] I [io-stats.c:1579:io_stats_dump_fd]
> 0-/data/recipiente11-gluster-home-lab-3: Filename :
> /u056/444.txt
> [2016-10-19 15:36:29.066206] I [io-stats.c:1584:io_stats_dump_fd]
> 0-/data/recipiente11-gluster-home-lab-3: Lifetime : 136secs,
> 894418usecs
> recipiente12
> [2016-10-19 15:29:06.022950] I [io-stats.c:1594:io_stats_dump_fd]
> 0-/data/recipiente12-gluster-home-lab-3: BytesWritten : 986 bytes
> [2016-10-19 15:29:06.022972] I [io-stats.c:1606:io_stats_dump_fd]
> 0-/data/recipiente12-gluster-home-lab-3: Write 000512b+ : 1
> [2016-10-19 15:29:32.646111] I [io-stats.c:1574:io_stats_dump_fd]
> 0-/data/recipiente12-gluster-home-lab-3: --- fd stats ---
> [2016-10-19 15:29:32.646214] I [io-stats.c:1579:io_stats_dump_fd]
> 0-/data/recipiente12-gluster-home-lab-3: Filename :
> /u056/.profile
> [2016-10-19 15:32:28.423115] I [io-stats.c:1574:io_stats_dump_fd]
> 0-/data/recipiente12-gluster-home-lab-3: --- fd stats ---
> [2016-10-19 15:32:28.423219] I [io-stats.c:1579:io_stats_dump_fd]
> 0-/data/recipiente12-gluster-home-lab-3: Filename :
> /u056/.bash_logout
> [2016-10-19 15:32:28.423247] I [io-stats.c:1584:io_stats_dump_fd]
> 0-/data/recipiente12-gluster-home-lab-3: Lifetime : 202secs,
> 412682usecs
> [2016-10-19 15:32:28.423345] I [io-stats.c:1574:io_stats_dump_fd]
> 0-/data/recipiente12-gluster-home-lab-3: --- fd stats ---
> [2016-10-19 15:32:28.423378] I [io-stats.c:1579:io_stats_dump_fd]
> 0-/data/recipiente12-gluster-home-lab-3: Filename :
> /u056/.bash_history
> [2016-10-19 15:32:28.423400] I [io-stats.c:1584:io_stats_dump_fd]
> 0-/data/recipiente12-gluster-home-lab-3: Lifetime : 202secs,
> 399903usecs
> recipiente13
> [2016-10-19 15:04:26.825754] I [io-stats.c:1574:io_stats_dump_fd]
> 0-/data/recipiente13-gluster-home-lab-3: --- fd stats ---
> [2016-10-19 15:04:26.825788] I [io-stats.c:1579:io_stats_dump_fd]
> 0-/data/recipiente13-gluster-home-lab-3: Filename : /u056/pp
> [2016-10-19 15:04:26.825799] I [MSGID: 101055]
> [client_t.c:415:gf_client_unref] 0-home-lab-3-server: Shutting down
> connection cuentas09-lab.lab.dit.upm.es-1578-2016/10/19-05:00:48:464910-home-lab-3-client-3-0-0
> [2016-10-19 15:04:26.825813] I [io-stats.c:1584:io_stats_dump_fd]
> 0-/data/recipiente13-gluster-home-lab-3: Lifetime : 25804secs,
> 917102usecs
> [2016-10-19 15:04:26.825991] I [io-stats.c:1574:io_stats_dump_fd]
> 0-/data/recipiente13-gluster-home-lab-3: --- fd stats ---
> [2016-10-19 15:04:26.826026] I [io-stats.c:1579:io_stats_dump_fd]
> 0-/data/recipiente13-gluster-home-lab-3: Filename :
> /u056/.local/share/evolution/tasks/system/tasks.ics
> [2016-10-19 15:04:26.826050] I [io-stats.c:1584:io_stats_dump_fd]
> 0-/data/recipiente13-gluster-home-lab-3: Lifetime : 25774secs,
> 172800usecs
> [2016-10-19 15:04:48.333183] I [MSGID: 115029]
> [server-handshake.c:692:server_setvolume] 0-home-lab-3-server:
> accepted client from cuentas09-lab.lab.dit.upm.es-1610-2016/10/19-15:04:46:807972-home-lab-3-client-3-0-0
> (version: 3.8.5)
> [2016-10-19 15:27:27.763705] E [MSGID: 115070]
> [server-rpc-fops.c:1472:server_open_cbk] 0-home-lab-3-server: 149:
> OPEN /u056/.Xauthority-c (a4fa9bc4-a09e-4eb8-968b-b77cb6d49fd3) ==>
> (Permission denied) [Permission denied]
> [2016-10-19 15:29:12.202792] E [MSGID: 115070]
> [server-rpc-fops.c:1472:server_open_cbk] 0-home-lab-3-server: 267:
> OPEN /u056/.Xauthority-c (ce222a86-9f74-42d1-8e18-1f80f607d766) ==>
> (Permission denied) [Permission denied]
> root@admin2:~#
>
> As you can see, the glusterfs client part of the server
> fuse-mounting the volume shows a "/u056/kk.txt => -1 (Permission
> denied)" error that comes from a similar error in the glusterfs
> cluster, while at the nfs-kernel server log we can see a kernel
> trace directly related to the glusterfs fuse mount of the volume
> that says "nfsd4_process_open2 failed to open newly-created file!
> status=13".
>
> This error can be reproduced any number of times exactly the same,
> is not sporadic.
>
> Other test I've done at the client side with no logs at all at the
> server side, but the same error for the user, is the following:
>
> u056@l056:~$ rm -f 444.txt; echo "prueba" > 444.txt; chmod 444
> 444.txt; echo "kk" > 444.txt
> -bash: 444.txt: Permission denied
> u056@l056:~$
>
> We have been trying to reproduce this problem in a virtual
> environment using KVM, nfs-kernel-server, glusterfs, and ZFS over
> VFS, being unable to do it... This leads us to think that creating
> glusterfs volumes over ZFS over phisical disks has something to do
> with this problem, but we have no clue on how to track it down to
> the ZFS level, as there are no traces on the kernel.log or the
> syslog that we can use.
>
> Thank you for your time.
>
> Omar
>
> El 21/12/15 a las 21:14, J. Bruce Fields escribió:
> >On Mon, Dec 21, 2015 at 11:28:36PM +0530, Soumya Koduri wrote:
> >>
> >>On 12/21/2015 10:17 PM, J. Bruce Fields wrote:
> >>>On Mon, Dec 21, 2015 at 02:18:20PM +0530, Soumya Koduri wrote:
> >>>>
> >>>>On 12/19/2015 01:38 AM, J. Bruce Fields wrote:
> >>>>>On Fri, Dec 18, 2015 at 10:47:42PM +0530, Soumya Koduri wrote:
> >>>>>>
> >>>>>>On 12/18/2015 08:50 PM, J. Bruce Fields wrote:
> >>>>>>>On Fri, Dec 18, 2015 at 02:13:40PM +0530, Soumya Koduri wrote:
> >>>>>>>>
> >>>>>>>>On 12/18/2015 06:07 AM, Malahal Naineni wrote:
> >>>>>>>>>IIRC, permission checks are done in open(). write/read syscalls should
> >>>>>>>>>NOT do much access checks (at least based on POSIX). This is why once an
> >>>>>>>>>open is done, you remove permissions for that process, but it should
> >>>>>>>>>still be able to read/write based on the open flags it did when it
> >>>>>>>>>opened the file.
> >>>>>>>>>
> >>>>>>>>>I don't know all the details of this defect, but gluster seems to be
> >>>>>>>>>doing what it is supposed to do.
> >>>>>>>>>
> >>>>>>>>Right. Thanks for the correction. I assumed the behavior should be
> >>>>>>>>same for both OPEN+WRITE vs CREATE+WRITE in the below scenario. But
> >>>>>>>>looks like (from 'man creat') the open() call that creates a
> >>>>>>>>read-only file may well return a read/write file descriptor, which
> >>>>>>>>is the reason the following WRITE can succeed.
> >>>>>>>I forgot another complication, which is that knsfd actually does a
> >>>>>>>temporary open before each read or write--I assume that's getting
> >>>>>>>translated into fuse and gluster open operations?
> >>>>>>>
> >>>>>>yes. It is the OPEN done as part of NFS WRITE which fails with
> >>>>>>EACCESS error (with both NFSv3 and NFSv4 mounts).
> >>>>>Makes sense for v3, but I wouldn't normally expect the extra temporary
> >>>>>open on v4 WRITEs. Could you share any details?
> >>>>>
> >>>>I re-tried the test on v4 mount using Fedora23 machine, acting as
> >>>>both NFS server and client (Linux#4.2.3-300.fc23.x86_64). Please
> >>>>find the pkt trace attached.
> >>>>
> >>>> 56 07:23:25.567134 ::1 -> ::1 NFS 288 V4 Call
> >>>>WRITE StateID: 0xf934 Offset: 0 Len: 7
> >>>> 57 07:23:25.567233 192.168.122.17 -> 192.168.122.202 GlusterFS 188
> >>>>V330 GETXATTR Call
> >>>> 58 07:23:25.567732 192.168.122.202 -> 192.168.122.17 GlusterFS 112
> >>>>V330 GETXATTR Reply (Call In 57)
> >>>> 59 07:23:25.567881 192.168.122.17 -> 192.168.122.202 GlusterFS 164
> >>>>V330 OPEN Call
> >>>Remind me what kernel version your server is on?
> >>NFS server is on fedora23 VM - Linux version 4.2.3-300.fc23.x86_64
> >We did reshuffle the code that does the temporary open on WRITE around
> >there--but it looks right to me, and I can't reproduce an open on WRITE
> >on that kernel myself.
> >
> >Maybe there's some further fuse or gluster debugging that would help
> >show where that open is coming from.
> >
> >--b.
> >
> >>Thanks,
> >>Soumya
> >>
> >>>--b.
> >>>
> >>>
> >>>> 60 07:23:25.568354 192.168.122.202 -> 192.168.122.17 GlusterFS 116
> >>>>V330 OPEN Reply (Call In 59)
> >>>> 61 07:23:25.568570 ::1 -> ::1 NFS 144 V4 Reply
> >>>>(Call In 56) WRITE Status: NFS4ERR_ACCESS
> >>>>
> >>>>Thanks,
> >>>>Soumya
> >>>>
> >>>>>--b.
> >>>>>
> >>>>>> 63 16:59:09.278651000 ::1 -> ::1 NFS 232 V3 WRITE
> >>>>>>Call, FH: 0x49a35e54 Offset: 0 Len: 7 FILE_SYNC
> >>>>>> 64 16:59:09.278926000 192.168.122.1 -> 192.168.122.202 GlusterFS
> >>>>>>164 V330 OPEN Call
> >>>>>> 65 16:59:09.278937000 192.168.122.1 -> 192.168.122.202 GlusterFS
> >>>>>>164 [RPC retransmission of #64][TCP Retransmission] V330 OPEN Call
> >>>>>> 66 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
> >>>>>>116 V330 OPEN Reply (Call In 64)
> >>>>>> 67 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
> >>>>>>116 [RPC duplicate of #66][TCP Retransmission] V330 OPEN Reply (Call
> >>>>>>In 64)
> >>>>>> 68 16:59:09.279733000 ::1 -> ::1 NFS 212 V3 WRITE
> >>>>>>Reply (Call In 63) Error: NFS3ERR_ACCES
> >>>>>>
> >>>>>>
> >>>>>>Thanks,
> >>>>>>Soumya
> >>>>>>
> >>>>>>>In which case it might be worth experimenting with NFSv4 or with Jeff
> >>>>>>>Layton's filehandle-caching patches. Neither's a real fix, but that
> >>>>>>>could help confirm whether it's the temporary opens that are a problem.
> >>>>>>>
> >>>>>>>--b.
> >>>>>>>
> >>>>>>>>Thanks,
> >>>>>>>>Soumya
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>Regards, Malahal.
> >>>>>>>>>
> >>>>>>>>>Soumya Koduri [[email protected]] wrote:
> >>>>>>>>>>As mentioned by Bruce, GlusterFS doesn't have owner-override rule
> >>>>>>>>>>except for setattr.
> >>>>>>>>>>
> >>>>>>>>>>I did few experiments to check why this test case passes on plain
> >>>>>>>>>>glusterfs fuse mount & NFS-Ganesha but fails with kernel-NFS.
> >>>>>>>>>>
> >>>>>>>>>>NFS-Ganesha (for most of the FSALs) seem to be passing the actual
> >>>>>>>>>>request credentials to the back-end filesystem only for
> >>>>>>>>>>CREATE(-like) and UNLINK fops. For all the remaining fops, it does
> >>>>>>>>>>the access check at its end and then perform the operation with root
> >>>>>>>>>>credentials. That's the reason WRITE succeeded in your case as
> >>>>>>>>>>NFS-Ganesha (like kernel-NFS) skipped the access check if the
> >>>>>>>>>>request caller_uid proved to be the file's owner.
> >>>>>>>>>>
> >>>>>>>>>>In case of native GlusterFS FUSE mount, there is no OPEN fop
> >>>>>>>>>>involved. WRITE is performed on the fd returned by CREATE. And
> >>>>>>>>>>strangely GlusterFS seem to be doing certain access checks only
> >>>>>>>>>>during OPEN but not for WRITE (this seems like a bug and probably
> >>>>>>>>>>needs to be fixed in Gluster).
> >>>>>>>>>>
> >>>>>>>>>>Thanks,
> >>>>>>>>>>Soumya
> >>>>>>>>>>
> >>>>>>>>>>On 12/14/2015 10:27 PM, Omar Walid Llorente wrote:
> >>>>>>>>>>>Thank you Bruce, others, for the responses. I send attached a complete
> >>>>>>>>>>>capture of the issue, including the glusterfs transactions.
> >>>>>>>>>>>
> >>>>>>>>>>>Hope this helps to clear where may it be...
> >>>>>>>>>>>
> >>>>>>>>>>>Omar
> >>>>>>>>>>>
> >>>>>>>>>>>El 10/12/15 a las 15:44, J. Bruce Fields escribió:
> >>>>>>>>>>>>On Thu, Dec 10, 2015 at 05:59:33PM +0530, Soumya Koduri wrote:
> >>>>>>>>>>>>>On 12/10/2015 04:02 PM, Omar Walid Llorente wrote:
> >>>>>>>>>>>>>>Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs
> >>>>>>>>>>>>>>packets (you can find them in attached file nfs-problem-nks.pcap.zip).
> >>>>>>>>>>>>>>Sorry for being so late.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>What I did was the following:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>1st) Create the RO file:
> >>>>>>>>>>>>>>cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
> >>>>>>>>>>>>>>chmod 444 444.txt;
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>2nd) Init the capture:
> >>>>>>>>>>>>>>root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
> >>>>>>>>>>>>>>tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
> >>>>>>>>>>>>>>512 bytes
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It
> >>>>>>>>>>>>>may be helpful to see what GlusterFS operations are being processed
> >>>>>>>>>>>>>as part of NFS WRITE call (which has failed in this case).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>Could you please try taking the packet trace on the machine where
> >>>>>>>>>>>>>NFS server is running (without filtering out based on the port
> >>>>>>>>>>>>>number).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>Also I tried out the same test on Fedora22 machine, but haven't run
> >>>>>>>>>>>>>into any issue. What are the fuse mount options you have used to
> >>>>>>>>>>>>>mount gluster volume?
> >>>>>>>>>>>>Oh, I think this is a simple problem (but maybe hard to fix). The
> >>>>>>>>>>>>capture shows NFSv3 traffic like:
> >>>>>>>>>>>>
> >>>>>>>>>>>> CREATE -> OK
> >>>>>>>>>>>> SETATTR (mode set to 0400) -> OK
> >>>>>>>>>>>> WRITE -> NFS3ERR_ACCES
> >>>>>>>>>>>>
> >>>>>>>>>>>>That write would succeed locally (because the mode doesn't matter to a
> >>>>>>>>>>>>local application that already holds the file open). It would fail over
> >>>>>>>>>>>>NFSv3, which doesn't know about the open--except that there's a hack for
> >>>>>>>>>>>>this case: NFSv3 servers allow IO operations to ignore the mode, if the
> >>>>>>>>>>>>operation comes from the owner of the file. NFSv3 clients are then
> >>>>>>>>>>>>careful to perform necessary access checks on open to ensure that this
> >>>>>>>>>>>>owner-override rule doesn't grant too many permissions.
> >>>>>>>>>>>>
> >>>>>>>>>>>>That allows NFSv3 applications to see behavior that's mostly like a
> >>>>>>>>>>>>local filesystem, without opening much of a security hole (since the
> >>>>>>>>>>>>owner could always chmod anyway).
> >>>>>>>>>>>>
> >>>>>>>>>>>>So, knfsd is making this special exception--but gluster (which I believe
> >>>>>>>>>>>>it's exporting in this case, via fuse?)--probably doesn't.... I'm not
> >>>>>>>>>>>>sure what you can do about that.
> >>>>>>>>>>>>
> >>>>>>>>>>>>--b.
> >>>>>>>>>>--
> >>>>>>>>>>To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> >>>>>>>>>>the body of a message to [email protected]
> >>>>>>>>>>More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>>>>>>>>>
> >>>>>--
> >>>>>To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> >>>>>the body of a message to [email protected]
> >>>>>More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>>>>
> >>>
>
> --
> ----------------------------------------------------------------
> Centro de Cálculo Depto. Ingeniería Sistemas Telemáticos
> E-mail: [email protected] Universidad Politécnica de Madrid
> Fax:(+34) 913367333 E.T.S. Ing. Telecomunicación
> Tel:(+34) 915495700-Ext.3005 28040 Madrid (Spain)
> Tel:(+34) 915495762-Ext.3005
> Tel:(+34) 913367366-Ext.3005
> ----------------------------------------------------------------
>

2016-11-11 17:58:03

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

Hi Bruce, thanks for your answer, that is greatly appreciated.

This side of the problem feels a little bit lonely and maybe we can
manage to get to know what is happening if you have some time.

We have been doing tests and we have found a way to replicate the
problem in a virtual environment without ZFS, only with glusterfs and
nfs-kernel-server, where we can give you root password or you can
replicate it at home.

It is closely related to non-root users (with root credentials the
problem does not happen).

If you think this is good news and feel like checking, tell me and next
week we will prepare it and I'll forward you the data for both kind of
tests.

Thank you very much for your time.

Omar

El 08/11/16 a las 21:16, J. Bruce Fields escribió:
> On Wed, Oct 19, 2016 at 07:53:37PM +0200, Omar Walid Llorente wrote:
>> Resending because it has not been accepted by
>> [email protected] because of the HTML... Sorry if you get it
>> twice.
>>
>> Thanks again.
>>
>> Omar
>>
>> --- Forwarded message ---
>>
>> Hi Bruce, others.
> Sory for the delayed response. This is a fairly complicated setup that
> may require people with nfs, gluster, and zfs experience to debug.
>
> The server does need to be able to bypass certain permission checks on
> open so that it can open a newly created read-only file for write.
> Probably one of those lower layers is insisting on permission checks
> where it shouldn't. That's all I can say from a quick look.
>
> --b.
>
>> I come back to this issue again in the hope that these evidences I
>> bring can help to give some light to the problem.
>>
>> We have exactly the same environment that last year when I came to
>> you, but with upgraded versions of gluster (3.8) and
>> nfs-kernel-server (1.2.8) this time over an updated Ubuntu 16.04.
>>
>> (NOTE: Environment: glusterfs-cluster made of ZFS datasets exports
>> via glusterfs a distributed volume + nfs-kernel server mounts via
>> fuse that glusterfs volume and reexports it via nfs-v3 + nfs-common
>> client mounts via nfs that nfs exported volume)
>>
>> The errors I get by the user side are exactly the same, but in the
>> server side, I have different messages, which I forward to you.
>>
>> * /TESTS/ and errors by the user side:
>>
>> u056@l056:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod
>> 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
>> cp: cannot create regular file ‘kk.txt’: Permission denied
>> ls: cannot access kk.txt: No such file or directory
>> -r--r--r-- 1 u056 admincdc 7 oct 19 2016 444.txt
>> u056@l056:~$ ls -lrt
>> total 33
>> ---------- 1 u056 admincdc 0 ene 4 1970 kk.txt
>> drwx------ 2 u056 alumno 4096 oct 3 2014 Desktop
>> drwxr-xr-x 2 u056 alumno 4096 sep 23 12:20 Público
>> drwxr-xr-x 2 u056 alumno 4096 sep 23 12:20 Descargas
>> drwxr-xr-x 2 u056 alumno 4096 sep 23 12:20 Documentos
>> drwxr-xr-x 2 u056 alumno 4096 sep 23 12:20 Vídeos
>> drwxr-xr-x 2 u056 alumno 4096 sep 23 12:20 Música
>> drwxr-xr-x 2 u056 alumno 4096 sep 23 12:20 Plantillas
>> drwxr-xr-x 2 u056 alumno 4096 sep 23 12:22 Imágenes
>> -r--r--r-- 1 u056 admincdc 7 oct 19 17:34 444.txt
>>
>> u056@l056:~$ mount | grep cuentas09
>> cuentas09:/home-3/u056 on /home/u056 type nfs (rw,noatime,intr,fsc,nolock,rsize=262140,wsize=262140,vers=4,addr=138.4.30.80,clientaddr=138.4.31.56)
>> u056@l056:~$ df -h | grep cuentas09
>> cuentas09:/home-3/u056 40T 55G 40T 1% /home/u056
>> u056@l056:~$
>>
>> * /LOGS/ by the glusterfs client side (server side), versions of the
>> software, info of the kernel and related kernel modules:
>>
>> root@cuentas09-lab:/var/log/glusterfs# tail home-3.log
>> [2016-10-19 15:33:38.091300] I [io-stats.c:1574:io_stats_dump_fd]
>> 0-home-lab-3: --- fd stats ---
>> [2016-10-19 15:33:38.091331] I [io-stats.c:1579:io_stats_dump_fd]
>> 0-home-lab-3: Filename : /u056/444.txt
>> [2016-10-19 15:33:38.091344] I [io-stats.c:1594:io_stats_dump_fd]
>> 0-home-lab-3: BytesWritten : 7 bytes
>> [2016-10-19 15:33:38.091355] I [io-stats.c:1606:io_stats_dump_fd]
>> 0-home-lab-3: Write 000004b+ : 1
>> [2016-10-19 15:34:13.199285] I [io-stats.c:1574:io_stats_dump_fd]
>> 0-home-lab-3: --- fd stats ---
>> [2016-10-19 15:34:13.200291] I [io-stats.c:1579:io_stats_dump_fd]
>> 0-home-lab-3: Filename : /u056/444.txt
>> [2016-10-19 15:34:13.200313] I [io-stats.c:1594:io_stats_dump_fd]
>> 0-home-lab-3: BytesWritten : 7 bytes
>> [2016-10-19 15:34:13.200325] I [io-stats.c:1606:io_stats_dump_fd]
>> 0-home-lab-3: Write 000004b+ : 1
>> [2016-10-19 15:34:13.211696] E [MSGID: 114031]
>> [client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-0:
>> remote operation failed. Path: /u056/kk.txt
>> (d8fc54a3-f2eb-4538-889a-17afdcfbb255) [Permission denied]
>> [2016-10-19 15:34:13.211758] W [fuse-bridge.c:989:fuse_fd_cbk]
>> 0-glusterfs-fuse: 630: OPEN() /u056/kk.txt => -1 (Permission denied)
>>
>> root@cuentas09-lab:/var/log/glusterfs# dpkg -l | grep -e gluster -e nfs
>> ii glusterfs-client 3.8.5-ubuntu1~xenial1 amd64
>> clustered file-system (client package)
>> ii glusterfs-common 3.8.5-ubuntu1~xenial1 amd64
>> GlusterFS common libraries and translator modules
>> ii libnfsidmap2:amd64 0.25-5 amd64
>> NFS idmapping library
>> ii nfs-common 1:1.2.8-9ubuntu12 amd64 NFS
>> support files common to client and server
>> ii nfs-kernel-server 1:1.2.8-9ubuntu12 amd64
>> support for NFS kernel server
>>
>> root@cuentas09-lab:/var/log/glusterfs# uname -a
>> Linux cuentas09-lab.lab.dit.upm.es 4.4.0-43-generic #63-Ubuntu SMP
>> Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
>>
>> root@cuentas09-lab:/var/log/glusterfs# lsmod | grep nfs
>> nfsd 319488 13
>> auth_rpcgss 61440 1 nfsd
>> nfs_acl 16384 1 nfsd
>> lockd 94208 1 nfsd
>> grace 16384 2 nfsd,lockd
>> sunrpc 335872 21 nfsd,auth_rpcgss,lockd,nfs_acl
>> root@cuentas09-lab:/var/log/glusterfs#
>>
>> * /LOGS/ by the nfs-kernel server side and basic configuration of
>> the nfs-kernel-server:
>>
>> root@cuentas09-lab:~# less /var/log/kern.log
>> root@cuentas09-lab:~# tail -30 !$
>> tail -30 /var/log/kern.log
>> Oct 19 17:30:24 cuentas09-lab kernel: [ 1573.631498]
>> [<ffffffff810a0850>] ? kthread_create_on_node+0x1e0/0x1e0
>> Oct 19 17:30:24 cuentas09-lab kernel: [ 1573.631504]
>> [<ffffffff81831c4f>] ret_from_fork+0x3f/0x70
>> Oct 19 17:30:24 cuentas09-lab kernel: [ 1573.631507]
>> [<ffffffff810a0850>] ? kthread_create_on_node+0x1e0/0x1e0
>> Oct 19 17:30:24 cuentas09-lab kernel: [ 1573.631510] ---[ end trace
>> 34db7650fa22d1d0 ]---
>> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.814898] ------------[
>> cut here ]------------
>> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.814948] WARNING: CPU: 0
>> PID: 1373 at /build/linux-BwgxJb/linux-4.4.0/fs/nfsd/nfs4proc.c:464
>> nfsd4_open+0x515/0x780 [nfsd]()
>> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.814951]
>> nfsd4_process_open2 failed to open newly-created file! status=13
>> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.814954] Modules linked
>> in: vmw_vsock_vmci_transport vsock ppdev vmw_balloon coretemp joydev
>> input_leds serio_raw irda 8250_fintek parport_pc shpchp parport
>> i2c_piix4 crc_ccitt vmw_vmci mac_hid ib_iser nfsd rdma_cm iw_cm
>> ib_cm auth_rpcgss nfs_acl lockd grace ib_sa sunrpc ib_mad ib_core
>> ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4
>> btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq
>> async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath
>> linear vmwgfx ttm psmouse drm_kms_helper syscopyarea sysfillrect
>> sysimgblt fb_sys_fops mptspi mptscsih mptbase drm e1000
>> scsi_transport_spi pata_acpi floppy fjes
>> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815018] CPU: 0 PID:
>> 1373 Comm: nfsd Tainted: G W 4.4.0-43-generic
>> #63-Ubuntu
>> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815021] Hardware name:
>> VMware, Inc. VMware Virtual Platform/440BX Desktop Reference
>> Platform, BIOS 6.00 09/22/2009
>> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815024]
>> 0000000000000286 00000000c0e912c3 ffff8800b93c7c80 ffffffff813f1f93
>> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815028]
>> ffff8800b93c7cc8 ffffffffc047b668 ffff8800b93c7cb8 ffffffff81081212
>> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815031]
>> ffff8800b9e43240 ffff8800b9e44068 000000000d000000 ffff8800b85e6000
>> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815035] Call Trace:
>> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815047]
>> [<ffffffff813f1f93>] dump_stack+0x63/0x90
>> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815055]
>> [<ffffffff81081212>] warn_slowpath_common+0x82/0xc0
>> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815058]
>> [<ffffffff810812ac>] warn_slowpath_fmt+0x5c/0x80
>> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815073]
>> [<ffffffffc04665db>] ? nfs4_free_ol_stateid+0x3b/0x40 [nfsd]
>> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815085]
>> [<ffffffffc0459d05>] nfsd4_open+0x515/0x780 [nfsd]
>> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815098]
>> [<ffffffffc045a2fa>] nfsd4_proc_compound+0x38a/0x660 [nfsd]
>> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815108]
>> [<ffffffffc0446e78>] nfsd_dispatch+0xb8/0x200 [nfsd]
>> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815132]
>> [<ffffffffc039eeac>] svc_process_common+0x40c/0x650 [sunrpc]
>> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815151]
>> [<ffffffffc03a0273>] svc_process+0x103/0x1c0 [sunrpc]
>> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815160]
>> [<ffffffffc04468cf>] nfsd+0xef/0x160 [nfsd]
>> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815170]
>> [<ffffffffc04467e0>] ? nfsd_destroy+0x60/0x60 [nfsd]
>> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815174]
>> [<ffffffff810a0928>] kthread+0xd8/0xf0
>> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815178]
>> [<ffffffff810a0850>] ? kthread_create_on_node+0x1e0/0x1e0
>> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815184]
>> [<ffffffff81831c4f>] ret_from_fork+0x3f/0x70
>> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815187]
>> [<ffffffff810a0850>] ? kthread_create_on_node+0x1e0/0x1e0
>> Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815190] ---[ end trace
>> 34db7650fa22d1d1 ]---
>> root@cuentas09-lab:~#
>>
>> root@cuentas09-lab:~# showmount -e
>> Export list for cuentas09-lab.lab.dit.upm.es:
>> /home-4 138.4.30.0/23
>> /home-3 138.4.30.0/23
>>
>> root@cuentas09-lab:~# cat /etc/exports | grep -v ^#
>> /home-3 138.4.30.0/23(rw,fsid=3,insecure,no_subtree_check,async,no_root_squash)
>> /home-4 138.4.30.0/23(rw,fsid=4,insecure,no_subtree_check,async,no_root_squash)
>> root@cuentas09-lab:~#
>>
>> root@cuentas09-lab:~# cat /etc/default/nfs-kernel-server | grep -v ^#
>> RPCNFSDCOUNT="8 -V 3"
>> RPCNFSDPRIORITY=0
>> RPCMOUNTDOPTS="--manage-gids"
>> NEED_SVCGSSD=""
>> RPCSVCGSSDOPTS=""
>> RPCNFSDOPTS="-d -V 3"
>> root@cuentas09-lab:~#
>>
>> * LOGS by the glusterfs server side:
>>
>> # for i in 10 11 12 13; do echo recipiente$i; ssh recipiente$i tail
>> /var/log/glusterfs/bricks/data-recipiente$i-gluster-home-lab-3.log;
>> done
>> recipiente10
>> [2016-10-19 15:27:48.172858] I [io-stats.c:1606:io_stats_dump_fd]
>> 0-/data/recipiente10-gluster-home-lab-3: Write 000008b+ : 1
>> [2016-10-19 15:29:32.626450] I [io-stats.c:1574:io_stats_dump_fd]
>> 0-/data/recipiente10-gluster-home-lab-3: --- fd stats ---
>> [2016-10-19 15:29:32.626545] I [io-stats.c:1579:io_stats_dump_fd]
>> 0-/data/recipiente10-gluster-home-lab-3: Filename :
>> /u056/.local/share/tracker/data/tracker-store.journal
>> [2016-10-19 15:29:32.626572] I [io-stats.c:1594:io_stats_dump_fd]
>> 0-/data/recipiente10-gluster-home-lab-3: BytesWritten : 8 bytes
>> [2016-10-19 15:29:32.626594] I [io-stats.c:1606:io_stats_dump_fd]
>> 0-/data/recipiente10-gluster-home-lab-3: Write 000008b+ : 1
>> [2016-10-19 15:30:23.997592] E [MSGID: 115070]
>> [server-rpc-fops.c:1472:server_open_cbk] 0-home-lab-3-server: 372:
>> OPEN /u056/kk.txt (cc97d150-c725-42b6-9aa5-50328f281e06) ==>
>> (Permission denied) [Permission denied]
>> [2016-10-19 15:32:28.423380] I [io-stats.c:1574:io_stats_dump_fd]
>> 0-/data/recipiente10-gluster-home-lab-3: --- fd stats ---
>> [2016-10-19 15:32:28.423478] I [io-stats.c:1579:io_stats_dump_fd]
>> 0-/data/recipiente10-gluster-home-lab-3: Filename :
>> /u056/.bashrc
>> [2016-10-19 15:32:28.423504] I [io-stats.c:1584:io_stats_dump_fd]
>> 0-/data/recipiente10-gluster-home-lab-3: Lifetime : 280secs,
>> 232488usecs
>> [2016-10-19 15:34:13.176805] E [MSGID: 115070]
>> [server-rpc-fops.c:1472:server_open_cbk] 0-home-lab-3-server: 424:
>> OPEN /u056/kk.txt (d8fc54a3-f2eb-4538-889a-17afdcfbb255) ==>
>> (Permission denied) [Permission denied]
>> recipiente11
>> [2016-10-19 15:33:38.057524] I [io-stats.c:1579:io_stats_dump_fd]
>> 0-/data/recipiente11-gluster-home-lab-3: Filename :
>> /u056/444.txt
>> [2016-10-19 15:33:38.057548] I [io-stats.c:1594:io_stats_dump_fd]
>> 0-/data/recipiente11-gluster-home-lab-3: BytesWritten : 7 bytes
>> [2016-10-19 15:33:38.057570] I [io-stats.c:1606:io_stats_dump_fd]
>> 0-/data/recipiente11-gluster-home-lab-3: Write 000004b+ : 1
>> [2016-10-19 15:34:13.164629] I [io-stats.c:1574:io_stats_dump_fd]
>> 0-/data/recipiente11-gluster-home-lab-3: --- fd stats ---
>> [2016-10-19 15:34:13.164722] I [io-stats.c:1579:io_stats_dump_fd]
>> 0-/data/recipiente11-gluster-home-lab-3: Filename :
>> /u056/444.txt
>> [2016-10-19 15:34:13.164769] I [io-stats.c:1594:io_stats_dump_fd]
>> 0-/data/recipiente11-gluster-home-lab-3: BytesWritten : 7 bytes
>> [2016-10-19 15:34:13.164791] I [io-stats.c:1606:io_stats_dump_fd]
>> 0-/data/recipiente11-gluster-home-lab-3: Write 000004b+ : 1
>> [2016-10-19 15:36:29.066078] I [io-stats.c:1574:io_stats_dump_fd]
>> 0-/data/recipiente11-gluster-home-lab-3: --- fd stats ---
>> [2016-10-19 15:36:29.066180] I [io-stats.c:1579:io_stats_dump_fd]
>> 0-/data/recipiente11-gluster-home-lab-3: Filename :
>> /u056/444.txt
>> [2016-10-19 15:36:29.066206] I [io-stats.c:1584:io_stats_dump_fd]
>> 0-/data/recipiente11-gluster-home-lab-3: Lifetime : 136secs,
>> 894418usecs
>> recipiente12
>> [2016-10-19 15:29:06.022950] I [io-stats.c:1594:io_stats_dump_fd]
>> 0-/data/recipiente12-gluster-home-lab-3: BytesWritten : 986 bytes
>> [2016-10-19 15:29:06.022972] I [io-stats.c:1606:io_stats_dump_fd]
>> 0-/data/recipiente12-gluster-home-lab-3: Write 000512b+ : 1
>> [2016-10-19 15:29:32.646111] I [io-stats.c:1574:io_stats_dump_fd]
>> 0-/data/recipiente12-gluster-home-lab-3: --- fd stats ---
>> [2016-10-19 15:29:32.646214] I [io-stats.c:1579:io_stats_dump_fd]
>> 0-/data/recipiente12-gluster-home-lab-3: Filename :
>> /u056/.profile
>> [2016-10-19 15:32:28.423115] I [io-stats.c:1574:io_stats_dump_fd]
>> 0-/data/recipiente12-gluster-home-lab-3: --- fd stats ---
>> [2016-10-19 15:32:28.423219] I [io-stats.c:1579:io_stats_dump_fd]
>> 0-/data/recipiente12-gluster-home-lab-3: Filename :
>> /u056/.bash_logout
>> [2016-10-19 15:32:28.423247] I [io-stats.c:1584:io_stats_dump_fd]
>> 0-/data/recipiente12-gluster-home-lab-3: Lifetime : 202secs,
>> 412682usecs
>> [2016-10-19 15:32:28.423345] I [io-stats.c:1574:io_stats_dump_fd]
>> 0-/data/recipiente12-gluster-home-lab-3: --- fd stats ---
>> [2016-10-19 15:32:28.423378] I [io-stats.c:1579:io_stats_dump_fd]
>> 0-/data/recipiente12-gluster-home-lab-3: Filename :
>> /u056/.bash_history
>> [2016-10-19 15:32:28.423400] I [io-stats.c:1584:io_stats_dump_fd]
>> 0-/data/recipiente12-gluster-home-lab-3: Lifetime : 202secs,
>> 399903usecs
>> recipiente13
>> [2016-10-19 15:04:26.825754] I [io-stats.c:1574:io_stats_dump_fd]
>> 0-/data/recipiente13-gluster-home-lab-3: --- fd stats ---
>> [2016-10-19 15:04:26.825788] I [io-stats.c:1579:io_stats_dump_fd]
>> 0-/data/recipiente13-gluster-home-lab-3: Filename : /u056/pp
>> [2016-10-19 15:04:26.825799] I [MSGID: 101055]
>> [client_t.c:415:gf_client_unref] 0-home-lab-3-server: Shutting down
>> connection cuentas09-lab.lab.dit.upm.es-1578-2016/10/19-05:00:48:464910-home-lab-3-client-3-0-0
>> [2016-10-19 15:04:26.825813] I [io-stats.c:1584:io_stats_dump_fd]
>> 0-/data/recipiente13-gluster-home-lab-3: Lifetime : 25804secs,
>> 917102usecs
>> [2016-10-19 15:04:26.825991] I [io-stats.c:1574:io_stats_dump_fd]
>> 0-/data/recipiente13-gluster-home-lab-3: --- fd stats ---
>> [2016-10-19 15:04:26.826026] I [io-stats.c:1579:io_stats_dump_fd]
>> 0-/data/recipiente13-gluster-home-lab-3: Filename :
>> /u056/.local/share/evolution/tasks/system/tasks.ics
>> [2016-10-19 15:04:26.826050] I [io-stats.c:1584:io_stats_dump_fd]
>> 0-/data/recipiente13-gluster-home-lab-3: Lifetime : 25774secs,
>> 172800usecs
>> [2016-10-19 15:04:48.333183] I [MSGID: 115029]
>> [server-handshake.c:692:server_setvolume] 0-home-lab-3-server:
>> accepted client from cuentas09-lab.lab.dit.upm.es-1610-2016/10/19-15:04:46:807972-home-lab-3-client-3-0-0
>> (version: 3.8.5)
>> [2016-10-19 15:27:27.763705] E [MSGID: 115070]
>> [server-rpc-fops.c:1472:server_open_cbk] 0-home-lab-3-server: 149:
>> OPEN /u056/.Xauthority-c (a4fa9bc4-a09e-4eb8-968b-b77cb6d49fd3) ==>
>> (Permission denied) [Permission denied]
>> [2016-10-19 15:29:12.202792] E [MSGID: 115070]
>> [server-rpc-fops.c:1472:server_open_cbk] 0-home-lab-3-server: 267:
>> OPEN /u056/.Xauthority-c (ce222a86-9f74-42d1-8e18-1f80f607d766) ==>
>> (Permission denied) [Permission denied]
>> root@admin2:~#
>>
>> As you can see, the glusterfs client part of the server
>> fuse-mounting the volume shows a "/u056/kk.txt => -1 (Permission
>> denied)" error that comes from a similar error in the glusterfs
>> cluster, while at the nfs-kernel server log we can see a kernel
>> trace directly related to the glusterfs fuse mount of the volume
>> that says "nfsd4_process_open2 failed to open newly-created file!
>> status=13".
>>
>> This error can be reproduced any number of times exactly the same,
>> is not sporadic.
>>
>> Other test I've done at the client side with no logs at all at the
>> server side, but the same error for the user, is the following:
>>
>> u056@l056:~$ rm -f 444.txt; echo "prueba" > 444.txt; chmod 444
>> 444.txt; echo "kk" > 444.txt
>> -bash: 444.txt: Permission denied
>> u056@l056:~$
>>
>> We have been trying to reproduce this problem in a virtual
>> environment using KVM, nfs-kernel-server, glusterfs, and ZFS over
>> VFS, being unable to do it... This leads us to think that creating
>> glusterfs volumes over ZFS over phisical disks has something to do
>> with this problem, but we have no clue on how to track it down to
>> the ZFS level, as there are no traces on the kernel.log or the
>> syslog that we can use.
>>
>> Thank you for your time.
>>
>> Omar
>>
>> El 21/12/15 a las 21:14, J. Bruce Fields escribió:
>>> On Mon, Dec 21, 2015 at 11:28:36PM +0530, Soumya Koduri wrote:
>>>> On 12/21/2015 10:17 PM, J. Bruce Fields wrote:
>>>>> On Mon, Dec 21, 2015 at 02:18:20PM +0530, Soumya Koduri wrote:
>>>>>> On 12/19/2015 01:38 AM, J. Bruce Fields wrote:
>>>>>>> On Fri, Dec 18, 2015 at 10:47:42PM +0530, Soumya Koduri wrote:
>>>>>>>> On 12/18/2015 08:50 PM, J. Bruce Fields wrote:
>>>>>>>>> On Fri, Dec 18, 2015 at 02:13:40PM +0530, Soumya Koduri wrote:
>>>>>>>>>> On 12/18/2015 06:07 AM, Malahal Naineni wrote:
>>>>>>>>>>> IIRC, permission checks are done in open(). write/read syscalls should
>>>>>>>>>>> NOT do much access checks (at least based on POSIX). This is why once an
>>>>>>>>>>> open is done, you remove permissions for that process, but it should
>>>>>>>>>>> still be able to read/write based on the open flags it did when it
>>>>>>>>>>> opened the file.
>>>>>>>>>>>
>>>>>>>>>>> I don't know all the details of this defect, but gluster seems to be
>>>>>>>>>>> doing what it is supposed to do.
>>>>>>>>>>>
>>>>>>>>>> Right. Thanks for the correction. I assumed the behavior should be
>>>>>>>>>> same for both OPEN+WRITE vs CREATE+WRITE in the below scenario. But
>>>>>>>>>> looks like (from 'man creat') the open() call that creates a
>>>>>>>>>> read-only file may well return a read/write file descriptor, which
>>>>>>>>>> is the reason the following WRITE can succeed.
>>>>>>>>> I forgot another complication, which is that knsfd actually does a
>>>>>>>>> temporary open before each read or write--I assume that's getting
>>>>>>>>> translated into fuse and gluster open operations?
>>>>>>>>>
>>>>>>>> yes. It is the OPEN done as part of NFS WRITE which fails with
>>>>>>>> EACCESS error (with both NFSv3 and NFSv4 mounts).
>>>>>>> Makes sense for v3, but I wouldn't normally expect the extra temporary
>>>>>>> open on v4 WRITEs. Could you share any details?
>>>>>>>
>>>>>> I re-tried the test on v4 mount using Fedora23 machine, acting as
>>>>>> both NFS server and client (Linux#4.2.3-300.fc23.x86_64). Please
>>>>>> find the pkt trace attached.
>>>>>>
>>>>>> 56 07:23:25.567134 ::1 -> ::1 NFS 288 V4 Call
>>>>>> WRITE StateID: 0xf934 Offset: 0 Len: 7
>>>>>> 57 07:23:25.567233 192.168.122.17 -> 192.168.122.202 GlusterFS 188
>>>>>> V330 GETXATTR Call
>>>>>> 58 07:23:25.567732 192.168.122.202 -> 192.168.122.17 GlusterFS 112
>>>>>> V330 GETXATTR Reply (Call In 57)
>>>>>> 59 07:23:25.567881 192.168.122.17 -> 192.168.122.202 GlusterFS 164
>>>>>> V330 OPEN Call
>>>>> Remind me what kernel version your server is on?
>>>> NFS server is on fedora23 VM - Linux version 4.2.3-300.fc23.x86_64
>>> We did reshuffle the code that does the temporary open on WRITE around
>>> there--but it looks right to me, and I can't reproduce an open on WRITE
>>> on that kernel myself.
>>>
>>> Maybe there's some further fuse or gluster debugging that would help
>>> show where that open is coming from.
>>>
>>> --b.
>>>
>>>> Thanks,
>>>> Soumya
>>>>
>>>>> --b.
>>>>>
>>>>>
>>>>>> 60 07:23:25.568354 192.168.122.202 -> 192.168.122.17 GlusterFS 116
>>>>>> V330 OPEN Reply (Call In 59)
>>>>>> 61 07:23:25.568570 ::1 -> ::1 NFS 144 V4 Reply
>>>>>> (Call In 56) WRITE Status: NFS4ERR_ACCESS
>>>>>>
>>>>>> Thanks,
>>>>>> Soumya
>>>>>>
>>>>>>> --b.
>>>>>>>
>>>>>>>> 63 16:59:09.278651000 ::1 -> ::1 NFS 232 V3 WRITE
>>>>>>>> Call, FH: 0x49a35e54 Offset: 0 Len: 7 FILE_SYNC
>>>>>>>> 64 16:59:09.278926000 192.168.122.1 -> 192.168.122.202 GlusterFS
>>>>>>>> 164 V330 OPEN Call
>>>>>>>> 65 16:59:09.278937000 192.168.122.1 -> 192.168.122.202 GlusterFS
>>>>>>>> 164 [RPC retransmission of #64][TCP Retransmission] V330 OPEN Call
>>>>>>>> 66 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
>>>>>>>> 116 V330 OPEN Reply (Call In 64)
>>>>>>>> 67 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
>>>>>>>> 116 [RPC duplicate of #66][TCP Retransmission] V330 OPEN Reply (Call
>>>>>>>> In 64)
>>>>>>>> 68 16:59:09.279733000 ::1 -> ::1 NFS 212 V3 WRITE
>>>>>>>> Reply (Call In 63) Error: NFS3ERR_ACCES
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Soumya
>>>>>>>>
>>>>>>>>> In which case it might be worth experimenting with NFSv4 or with Jeff
>>>>>>>>> Layton's filehandle-caching patches. Neither's a real fix, but that
>>>>>>>>> could help confirm whether it's the temporary opens that are a problem.
>>>>>>>>>
>>>>>>>>> --b.
>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Soumya
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Regards, Malahal.
>>>>>>>>>>>
>>>>>>>>>>> Soumya Koduri [[email protected]] wrote:
>>>>>>>>>>>> As mentioned by Bruce, GlusterFS doesn't have owner-override rule
>>>>>>>>>>>> except for setattr.
>>>>>>>>>>>>
>>>>>>>>>>>> I did few experiments to check why this test case passes on plain
>>>>>>>>>>>> glusterfs fuse mount & NFS-Ganesha but fails with kernel-NFS.
>>>>>>>>>>>>
>>>>>>>>>>>> NFS-Ganesha (for most of the FSALs) seem to be passing the actual
>>>>>>>>>>>> request credentials to the back-end filesystem only for
>>>>>>>>>>>> CREATE(-like) and UNLINK fops. For all the remaining fops, it does
>>>>>>>>>>>> the access check at its end and then perform the operation with root
>>>>>>>>>>>> credentials. That's the reason WRITE succeeded in your case as
>>>>>>>>>>>> NFS-Ganesha (like kernel-NFS) skipped the access check if the
>>>>>>>>>>>> request caller_uid proved to be the file's owner.
>>>>>>>>>>>>
>>>>>>>>>>>> In case of native GlusterFS FUSE mount, there is no OPEN fop
>>>>>>>>>>>> involved. WRITE is performed on the fd returned by CREATE. And
>>>>>>>>>>>> strangely GlusterFS seem to be doing certain access checks only
>>>>>>>>>>>> during OPEN but not for WRITE (this seems like a bug and probably
>>>>>>>>>>>> needs to be fixed in Gluster).
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Soumya
>>>>>>>>>>>>
>>>>>>>>>>>> On 12/14/2015 10:27 PM, Omar Walid Llorente wrote:
>>>>>>>>>>>>> Thank you Bruce, others, for the responses. I send attached a complete
>>>>>>>>>>>>> capture of the issue, including the glusterfs transactions.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hope this helps to clear where may it be...
>>>>>>>>>>>>>
>>>>>>>>>>>>> Omar
>>>>>>>>>>>>>
>>>>>>>>>>>>> El 10/12/15 a las 15:44, J. Bruce Fields escribió:
>>>>>>>>>>>>>> On Thu, Dec 10, 2015 at 05:59:33PM +0530, Soumya Koduri wrote:
>>>>>>>>>>>>>>> On 12/10/2015 04:02 PM, Omar Walid Llorente wrote:
>>>>>>>>>>>>>>>> Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs
>>>>>>>>>>>>>>>> packets (you can find them in attached file nfs-problem-nks.pcap.zip).
>>>>>>>>>>>>>>>> Sorry for being so late.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> What I did was the following:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 1st) Create the RO file:
>>>>>>>>>>>>>>>> cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
>>>>>>>>>>>>>>>> chmod 444 444.txt;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2nd) Init the capture:
>>>>>>>>>>>>>>>> root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
>>>>>>>>>>>>>>>> tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
>>>>>>>>>>>>>>>> 512 bytes
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It
>>>>>>>>>>>>>>> may be helpful to see what GlusterFS operations are being processed
>>>>>>>>>>>>>>> as part of NFS WRITE call (which has failed in this case).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Could you please try taking the packet trace on the machine where
>>>>>>>>>>>>>>> NFS server is running (without filtering out based on the port
>>>>>>>>>>>>>>> number).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Also I tried out the same test on Fedora22 machine, but haven't run
>>>>>>>>>>>>>>> into any issue. What are the fuse mount options you have used to
>>>>>>>>>>>>>>> mount gluster volume?
>>>>>>>>>>>>>> Oh, I think this is a simple problem (but maybe hard to fix). The
>>>>>>>>>>>>>> capture shows NFSv3 traffic like:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> CREATE -> OK
>>>>>>>>>>>>>> SETATTR (mode set to 0400) -> OK
>>>>>>>>>>>>>> WRITE -> NFS3ERR_ACCES
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That write would succeed locally (because the mode doesn't matter to a
>>>>>>>>>>>>>> local application that already holds the file open). It would fail over
>>>>>>>>>>>>>> NFSv3, which doesn't know about the open--except that there's a hack for
>>>>>>>>>>>>>> this case: NFSv3 servers allow IO operations to ignore the mode, if the
>>>>>>>>>>>>>> operation comes from the owner of the file. NFSv3 clients are then
>>>>>>>>>>>>>> careful to perform necessary access checks on open to ensure that this
>>>>>>>>>>>>>> owner-override rule doesn't grant too many permissions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That allows NFSv3 applications to see behavior that's mostly like a
>>>>>>>>>>>>>> local filesystem, without opening much of a security hole (since the
>>>>>>>>>>>>>> owner could always chmod anyway).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So, knfsd is making this special exception--but gluster (which I believe
>>>>>>>>>>>>>> it's exporting in this case, via fuse?)--probably doesn't.... I'm not
>>>>>>>>>>>>>> sure what you can do about that.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --b.
>>>>>>>>>>>> --
>>>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>>>>>>>>>>> the body of a message to [email protected]
>>>>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>>>>>>>
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>>>>>> the body of a message to [email protected]
>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>>
>> --
>> ----------------------------------------------------------------
>> Centro de Cálculo Depto. Ingeniería Sistemas Telemáticos
>> E-mail: [email protected] Universidad Politécnica de Madrid
>> Fax:(+34) 913367333 E.T.S. Ing. Telecomunicación
>> Tel:(+34) 915495700-Ext.3005 28040 Madrid (Spain)
>> Tel:(+34) 915495762-Ext.3005
>> Tel:(+34) 913367366-Ext.3005
>> ----------------------------------------------------------------
>>

--
----------------------------------------------------------------
Centro de Cálculo Depto. Ingeniería Sistemas Telemáticos
E-mail: [email protected] Universidad Politécnica de Madrid
Fax:(+34) 913367333 E.T.S. Ing. Telecomunicación
Tel:(+34) 915495700-Ext.3005 28040 Madrid (Spain)
Tel:(+34) 915495762-Ext.3005
Tel:(+34) 913367366-Ext.3005
----------------------------------------------------------------

2016-11-11 19:03:35

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

On Fri, Nov 11, 2016 at 06:57:45PM +0100, Omar Walid Llorente wrote:
> Hi Bruce, thanks for your answer, that is greatly appreciated.
>
> This side of the problem feels a little bit lonely and maybe we can
> manage to get to know what is happening if you have some time.
>
> We have been doing tests and we have found a way to replicate the
> problem in a virtual environment without ZFS, only with glusterfs
> and nfs-kernel-server, where we can give you root password or you
> can replicate it at home.
>
> It is closely related to non-root users (with root credentials the
> problem does not happen).
>
> If you think this is good news and feel like checking, tell me and
> next week we will prepare it and I'll forward you the data for both
> kind of tests.

So if I'm reading right, the setup is

nfs client---NFSv3--->knfsd---FUSE-->gluster

I hate to just pass the buck, but my bet is on the issue being either on
the gluster side, or on limitations of the fuse protocol itself.

Cc'ing Miklos as I believe he did the original fuse nfsd export
implementation. Miklos, does fuse has a way for file owners to override
permissions? We need this sometimes because for example, an application
may hold a write open on a read-only file, but the server doesn't itself
already have an open (which can happen for lots of reasons).

--b.

>
> Thank you very much for your time.
>
> Omar
>
>
> El 08/11/16 a las 21:16, J. Bruce Fields escribió:
> >On Wed, Oct 19, 2016 at 07:53:37PM +0200, Omar Walid Llorente wrote:
> >>Resending because it has not been accepted by
> >>[email protected] because of the HTML... Sorry if you get it
> >>twice.
> >>
> >>Thanks again.
> >>
> >>Omar
> >>
> >>--- Forwarded message ---
> >>
> >>Hi Bruce, others.
> >Sory for the delayed response. This is a fairly complicated setup that
> >may require people with nfs, gluster, and zfs experience to debug.
> >
> >The server does need to be able to bypass certain permission checks on
> >open so that it can open a newly created read-only file for write.
> >Probably one of those lower layers is insisting on permission checks
> >where it shouldn't. That's all I can say from a quick look.
> >
> >--b.
> >
> >>I come back to this issue again in the hope that these evidences I
> >>bring can help to give some light to the problem.
> >>
> >>We have exactly the same environment that last year when I came to
> >>you, but with upgraded versions of gluster (3.8) and
> >>nfs-kernel-server (1.2.8) this time over an updated Ubuntu 16.04.
> >>
> >>(NOTE: Environment: glusterfs-cluster made of ZFS datasets exports
> >>via glusterfs a distributed volume + nfs-kernel server mounts via
> >>fuse that glusterfs volume and reexports it via nfs-v3 + nfs-common
> >>client mounts via nfs that nfs exported volume)
> >>
> >>The errors I get by the user side are exactly the same, but in the
> >>server side, I have different messages, which I forward to you.
> >>
> >>* /TESTS/ and errors by the user side:
> >>
> >>u056@l056:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod
> >>444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
> >>cp: cannot create regular file ‘kk.txt’: Permission denied
> >>ls: cannot access kk.txt: No such file or directory
> >>-r--r--r-- 1 u056 admincdc 7 oct 19 2016 444.txt
> >>u056@l056:~$ ls -lrt
> >>total 33
> >>---------- 1 u056 admincdc 0 ene 4 1970 kk.txt
> >>drwx------ 2 u056 alumno 4096 oct 3 2014 Desktop
> >>drwxr-xr-x 2 u056 alumno 4096 sep 23 12:20 Público
> >>drwxr-xr-x 2 u056 alumno 4096 sep 23 12:20 Descargas
> >>drwxr-xr-x 2 u056 alumno 4096 sep 23 12:20 Documentos
> >>drwxr-xr-x 2 u056 alumno 4096 sep 23 12:20 Vídeos
> >>drwxr-xr-x 2 u056 alumno 4096 sep 23 12:20 Música
> >>drwxr-xr-x 2 u056 alumno 4096 sep 23 12:20 Plantillas
> >>drwxr-xr-x 2 u056 alumno 4096 sep 23 12:22 Imágenes
> >>-r--r--r-- 1 u056 admincdc 7 oct 19 17:34 444.txt
> >>
> >>u056@l056:~$ mount | grep cuentas09
> >>cuentas09:/home-3/u056 on /home/u056 type nfs (rw,noatime,intr,fsc,nolock,rsize=262140,wsize=262140,vers=4,addr=138.4.30.80,clientaddr=138.4.31.56)
> >>u056@l056:~$ df -h | grep cuentas09
> >>cuentas09:/home-3/u056 40T 55G 40T 1% /home/u056
> >>u056@l056:~$
> >>
> >>* /LOGS/ by the glusterfs client side (server side), versions of the
> >>software, info of the kernel and related kernel modules:
> >>
> >>root@cuentas09-lab:/var/log/glusterfs# tail home-3.log
> >>[2016-10-19 15:33:38.091300] I [io-stats.c:1574:io_stats_dump_fd]
> >>0-home-lab-3: --- fd stats ---
> >>[2016-10-19 15:33:38.091331] I [io-stats.c:1579:io_stats_dump_fd]
> >>0-home-lab-3: Filename : /u056/444.txt
> >>[2016-10-19 15:33:38.091344] I [io-stats.c:1594:io_stats_dump_fd]
> >>0-home-lab-3: BytesWritten : 7 bytes
> >>[2016-10-19 15:33:38.091355] I [io-stats.c:1606:io_stats_dump_fd]
> >>0-home-lab-3: Write 000004b+ : 1
> >>[2016-10-19 15:34:13.199285] I [io-stats.c:1574:io_stats_dump_fd]
> >>0-home-lab-3: --- fd stats ---
> >>[2016-10-19 15:34:13.200291] I [io-stats.c:1579:io_stats_dump_fd]
> >>0-home-lab-3: Filename : /u056/444.txt
> >>[2016-10-19 15:34:13.200313] I [io-stats.c:1594:io_stats_dump_fd]
> >>0-home-lab-3: BytesWritten : 7 bytes
> >>[2016-10-19 15:34:13.200325] I [io-stats.c:1606:io_stats_dump_fd]
> >>0-home-lab-3: Write 000004b+ : 1
> >>[2016-10-19 15:34:13.211696] E [MSGID: 114031]
> >>[client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-0:
> >>remote operation failed. Path: /u056/kk.txt
> >>(d8fc54a3-f2eb-4538-889a-17afdcfbb255) [Permission denied]
> >>[2016-10-19 15:34:13.211758] W [fuse-bridge.c:989:fuse_fd_cbk]
> >>0-glusterfs-fuse: 630: OPEN() /u056/kk.txt => -1 (Permission denied)
> >>
> >>root@cuentas09-lab:/var/log/glusterfs# dpkg -l | grep -e gluster -e nfs
> >>ii glusterfs-client 3.8.5-ubuntu1~xenial1 amd64
> >>clustered file-system (client package)
> >>ii glusterfs-common 3.8.5-ubuntu1~xenial1 amd64
> >>GlusterFS common libraries and translator modules
> >>ii libnfsidmap2:amd64 0.25-5 amd64
> >>NFS idmapping library
> >>ii nfs-common 1:1.2.8-9ubuntu12 amd64 NFS
> >>support files common to client and server
> >>ii nfs-kernel-server 1:1.2.8-9ubuntu12 amd64
> >>support for NFS kernel server
> >>
> >>root@cuentas09-lab:/var/log/glusterfs# uname -a
> >>Linux cuentas09-lab.lab.dit.upm.es 4.4.0-43-generic #63-Ubuntu SMP
> >>Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
> >>
> >>root@cuentas09-lab:/var/log/glusterfs# lsmod | grep nfs
> >>nfsd 319488 13
> >>auth_rpcgss 61440 1 nfsd
> >>nfs_acl 16384 1 nfsd
> >>lockd 94208 1 nfsd
> >>grace 16384 2 nfsd,lockd
> >>sunrpc 335872 21 nfsd,auth_rpcgss,lockd,nfs_acl
> >>root@cuentas09-lab:/var/log/glusterfs#
> >>
> >>* /LOGS/ by the nfs-kernel server side and basic configuration of
> >>the nfs-kernel-server:
> >>
> >>root@cuentas09-lab:~# less /var/log/kern.log
> >>root@cuentas09-lab:~# tail -30 !$
> >>tail -30 /var/log/kern.log
> >>Oct 19 17:30:24 cuentas09-lab kernel: [ 1573.631498]
> >>[<ffffffff810a0850>] ? kthread_create_on_node+0x1e0/0x1e0
> >>Oct 19 17:30:24 cuentas09-lab kernel: [ 1573.631504]
> >>[<ffffffff81831c4f>] ret_from_fork+0x3f/0x70
> >>Oct 19 17:30:24 cuentas09-lab kernel: [ 1573.631507]
> >>[<ffffffff810a0850>] ? kthread_create_on_node+0x1e0/0x1e0
> >>Oct 19 17:30:24 cuentas09-lab kernel: [ 1573.631510] ---[ end trace
> >>34db7650fa22d1d0 ]---
> >>Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.814898] ------------[
> >>cut here ]------------
> >>Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.814948] WARNING: CPU: 0
> >>PID: 1373 at /build/linux-BwgxJb/linux-4.4.0/fs/nfsd/nfs4proc.c:464
> >>nfsd4_open+0x515/0x780 [nfsd]()
> >>Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.814951]
> >>nfsd4_process_open2 failed to open newly-created file! status=13
> >>Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.814954] Modules linked
> >>in: vmw_vsock_vmci_transport vsock ppdev vmw_balloon coretemp joydev
> >>input_leds serio_raw irda 8250_fintek parport_pc shpchp parport
> >>i2c_piix4 crc_ccitt vmw_vmci mac_hid ib_iser nfsd rdma_cm iw_cm
> >>ib_cm auth_rpcgss nfs_acl lockd grace ib_sa sunrpc ib_mad ib_core
> >>ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4
> >>btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq
> >>async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath
> >>linear vmwgfx ttm psmouse drm_kms_helper syscopyarea sysfillrect
> >>sysimgblt fb_sys_fops mptspi mptscsih mptbase drm e1000
> >>scsi_transport_spi pata_acpi floppy fjes
> >>Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815018] CPU: 0 PID:
> >>1373 Comm: nfsd Tainted: G W 4.4.0-43-generic
> >>#63-Ubuntu
> >>Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815021] Hardware name:
> >>VMware, Inc. VMware Virtual Platform/440BX Desktop Reference
> >>Platform, BIOS 6.00 09/22/2009
> >>Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815024]
> >>0000000000000286 00000000c0e912c3 ffff8800b93c7c80 ffffffff813f1f93
> >>Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815028]
> >>ffff8800b93c7cc8 ffffffffc047b668 ffff8800b93c7cb8 ffffffff81081212
> >>Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815031]
> >>ffff8800b9e43240 ffff8800b9e44068 000000000d000000 ffff8800b85e6000
> >>Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815035] Call Trace:
> >>Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815047]
> >>[<ffffffff813f1f93>] dump_stack+0x63/0x90
> >>Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815055]
> >>[<ffffffff81081212>] warn_slowpath_common+0x82/0xc0
> >>Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815058]
> >>[<ffffffff810812ac>] warn_slowpath_fmt+0x5c/0x80
> >>Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815073]
> >>[<ffffffffc04665db>] ? nfs4_free_ol_stateid+0x3b/0x40 [nfsd]
> >>Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815085]
> >>[<ffffffffc0459d05>] nfsd4_open+0x515/0x780 [nfsd]
> >>Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815098]
> >>[<ffffffffc045a2fa>] nfsd4_proc_compound+0x38a/0x660 [nfsd]
> >>Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815108]
> >>[<ffffffffc0446e78>] nfsd_dispatch+0xb8/0x200 [nfsd]
> >>Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815132]
> >>[<ffffffffc039eeac>] svc_process_common+0x40c/0x650 [sunrpc]
> >>Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815151]
> >>[<ffffffffc03a0273>] svc_process+0x103/0x1c0 [sunrpc]
> >>Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815160]
> >>[<ffffffffc04468cf>] nfsd+0xef/0x160 [nfsd]
> >>Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815170]
> >>[<ffffffffc04467e0>] ? nfsd_destroy+0x60/0x60 [nfsd]
> >>Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815174]
> >>[<ffffffff810a0928>] kthread+0xd8/0xf0
> >>Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815178]
> >>[<ffffffff810a0850>] ? kthread_create_on_node+0x1e0/0x1e0
> >>Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815184]
> >>[<ffffffff81831c4f>] ret_from_fork+0x3f/0x70
> >>Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815187]
> >>[<ffffffff810a0850>] ? kthread_create_on_node+0x1e0/0x1e0
> >>Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815190] ---[ end trace
> >>34db7650fa22d1d1 ]---
> >>root@cuentas09-lab:~#
> >>
> >>root@cuentas09-lab:~# showmount -e
> >>Export list for cuentas09-lab.lab.dit.upm.es:
> >>/home-4 138.4.30.0/23
> >>/home-3 138.4.30.0/23
> >>
> >>root@cuentas09-lab:~# cat /etc/exports | grep -v ^#
> >>/home-3 138.4.30.0/23(rw,fsid=3,insecure,no_subtree_check,async,no_root_squash)
> >>/home-4 138.4.30.0/23(rw,fsid=4,insecure,no_subtree_check,async,no_root_squash)
> >>root@cuentas09-lab:~#
> >>
> >>root@cuentas09-lab:~# cat /etc/default/nfs-kernel-server | grep -v ^#
> >>RPCNFSDCOUNT="8 -V 3"
> >>RPCNFSDPRIORITY=0
> >>RPCMOUNTDOPTS="--manage-gids"
> >>NEED_SVCGSSD=""
> >>RPCSVCGSSDOPTS=""
> >>RPCNFSDOPTS="-d -V 3"
> >>root@cuentas09-lab:~#
> >>
> >>* LOGS by the glusterfs server side:
> >>
> >># for i in 10 11 12 13; do echo recipiente$i; ssh recipiente$i tail
> >>/var/log/glusterfs/bricks/data-recipiente$i-gluster-home-lab-3.log;
> >>done
> >>recipiente10
> >>[2016-10-19 15:27:48.172858] I [io-stats.c:1606:io_stats_dump_fd]
> >>0-/data/recipiente10-gluster-home-lab-3: Write 000008b+ : 1
> >>[2016-10-19 15:29:32.626450] I [io-stats.c:1574:io_stats_dump_fd]
> >>0-/data/recipiente10-gluster-home-lab-3: --- fd stats ---
> >>[2016-10-19 15:29:32.626545] I [io-stats.c:1579:io_stats_dump_fd]
> >>0-/data/recipiente10-gluster-home-lab-3: Filename :
> >>/u056/.local/share/tracker/data/tracker-store.journal
> >>[2016-10-19 15:29:32.626572] I [io-stats.c:1594:io_stats_dump_fd]
> >>0-/data/recipiente10-gluster-home-lab-3: BytesWritten : 8 bytes
> >>[2016-10-19 15:29:32.626594] I [io-stats.c:1606:io_stats_dump_fd]
> >>0-/data/recipiente10-gluster-home-lab-3: Write 000008b+ : 1
> >>[2016-10-19 15:30:23.997592] E [MSGID: 115070]
> >>[server-rpc-fops.c:1472:server_open_cbk] 0-home-lab-3-server: 372:
> >>OPEN /u056/kk.txt (cc97d150-c725-42b6-9aa5-50328f281e06) ==>
> >>(Permission denied) [Permission denied]
> >>[2016-10-19 15:32:28.423380] I [io-stats.c:1574:io_stats_dump_fd]
> >>0-/data/recipiente10-gluster-home-lab-3: --- fd stats ---
> >>[2016-10-19 15:32:28.423478] I [io-stats.c:1579:io_stats_dump_fd]
> >>0-/data/recipiente10-gluster-home-lab-3: Filename :
> >>/u056/.bashrc
> >>[2016-10-19 15:32:28.423504] I [io-stats.c:1584:io_stats_dump_fd]
> >>0-/data/recipiente10-gluster-home-lab-3: Lifetime : 280secs,
> >>232488usecs
> >>[2016-10-19 15:34:13.176805] E [MSGID: 115070]
> >>[server-rpc-fops.c:1472:server_open_cbk] 0-home-lab-3-server: 424:
> >>OPEN /u056/kk.txt (d8fc54a3-f2eb-4538-889a-17afdcfbb255) ==>
> >>(Permission denied) [Permission denied]
> >>recipiente11
> >>[2016-10-19 15:33:38.057524] I [io-stats.c:1579:io_stats_dump_fd]
> >>0-/data/recipiente11-gluster-home-lab-3: Filename :
> >>/u056/444.txt
> >>[2016-10-19 15:33:38.057548] I [io-stats.c:1594:io_stats_dump_fd]
> >>0-/data/recipiente11-gluster-home-lab-3: BytesWritten : 7 bytes
> >>[2016-10-19 15:33:38.057570] I [io-stats.c:1606:io_stats_dump_fd]
> >>0-/data/recipiente11-gluster-home-lab-3: Write 000004b+ : 1
> >>[2016-10-19 15:34:13.164629] I [io-stats.c:1574:io_stats_dump_fd]
> >>0-/data/recipiente11-gluster-home-lab-3: --- fd stats ---
> >>[2016-10-19 15:34:13.164722] I [io-stats.c:1579:io_stats_dump_fd]
> >>0-/data/recipiente11-gluster-home-lab-3: Filename :
> >>/u056/444.txt
> >>[2016-10-19 15:34:13.164769] I [io-stats.c:1594:io_stats_dump_fd]
> >>0-/data/recipiente11-gluster-home-lab-3: BytesWritten : 7 bytes
> >>[2016-10-19 15:34:13.164791] I [io-stats.c:1606:io_stats_dump_fd]
> >>0-/data/recipiente11-gluster-home-lab-3: Write 000004b+ : 1
> >>[2016-10-19 15:36:29.066078] I [io-stats.c:1574:io_stats_dump_fd]
> >>0-/data/recipiente11-gluster-home-lab-3: --- fd stats ---
> >>[2016-10-19 15:36:29.066180] I [io-stats.c:1579:io_stats_dump_fd]
> >>0-/data/recipiente11-gluster-home-lab-3: Filename :
> >>/u056/444.txt
> >>[2016-10-19 15:36:29.066206] I [io-stats.c:1584:io_stats_dump_fd]
> >>0-/data/recipiente11-gluster-home-lab-3: Lifetime : 136secs,
> >>894418usecs
> >>recipiente12
> >>[2016-10-19 15:29:06.022950] I [io-stats.c:1594:io_stats_dump_fd]
> >>0-/data/recipiente12-gluster-home-lab-3: BytesWritten : 986 bytes
> >>[2016-10-19 15:29:06.022972] I [io-stats.c:1606:io_stats_dump_fd]
> >>0-/data/recipiente12-gluster-home-lab-3: Write 000512b+ : 1
> >>[2016-10-19 15:29:32.646111] I [io-stats.c:1574:io_stats_dump_fd]
> >>0-/data/recipiente12-gluster-home-lab-3: --- fd stats ---
> >>[2016-10-19 15:29:32.646214] I [io-stats.c:1579:io_stats_dump_fd]
> >>0-/data/recipiente12-gluster-home-lab-3: Filename :
> >>/u056/.profile
> >>[2016-10-19 15:32:28.423115] I [io-stats.c:1574:io_stats_dump_fd]
> >>0-/data/recipiente12-gluster-home-lab-3: --- fd stats ---
> >>[2016-10-19 15:32:28.423219] I [io-stats.c:1579:io_stats_dump_fd]
> >>0-/data/recipiente12-gluster-home-lab-3: Filename :
> >>/u056/.bash_logout
> >>[2016-10-19 15:32:28.423247] I [io-stats.c:1584:io_stats_dump_fd]
> >>0-/data/recipiente12-gluster-home-lab-3: Lifetime : 202secs,
> >>412682usecs
> >>[2016-10-19 15:32:28.423345] I [io-stats.c:1574:io_stats_dump_fd]
> >>0-/data/recipiente12-gluster-home-lab-3: --- fd stats ---
> >>[2016-10-19 15:32:28.423378] I [io-stats.c:1579:io_stats_dump_fd]
> >>0-/data/recipiente12-gluster-home-lab-3: Filename :
> >>/u056/.bash_history
> >>[2016-10-19 15:32:28.423400] I [io-stats.c:1584:io_stats_dump_fd]
> >>0-/data/recipiente12-gluster-home-lab-3: Lifetime : 202secs,
> >>399903usecs
> >>recipiente13
> >>[2016-10-19 15:04:26.825754] I [io-stats.c:1574:io_stats_dump_fd]
> >>0-/data/recipiente13-gluster-home-lab-3: --- fd stats ---
> >>[2016-10-19 15:04:26.825788] I [io-stats.c:1579:io_stats_dump_fd]
> >>0-/data/recipiente13-gluster-home-lab-3: Filename : /u056/pp
> >>[2016-10-19 15:04:26.825799] I [MSGID: 101055]
> >>[client_t.c:415:gf_client_unref] 0-home-lab-3-server: Shutting down
> >>connection cuentas09-lab.lab.dit.upm.es-1578-2016/10/19-05:00:48:464910-home-lab-3-client-3-0-0
> >>[2016-10-19 15:04:26.825813] I [io-stats.c:1584:io_stats_dump_fd]
> >>0-/data/recipiente13-gluster-home-lab-3: Lifetime : 25804secs,
> >>917102usecs
> >>[2016-10-19 15:04:26.825991] I [io-stats.c:1574:io_stats_dump_fd]
> >>0-/data/recipiente13-gluster-home-lab-3: --- fd stats ---
> >>[2016-10-19 15:04:26.826026] I [io-stats.c:1579:io_stats_dump_fd]
> >>0-/data/recipiente13-gluster-home-lab-3: Filename :
> >>/u056/.local/share/evolution/tasks/system/tasks.ics
> >>[2016-10-19 15:04:26.826050] I [io-stats.c:1584:io_stats_dump_fd]
> >>0-/data/recipiente13-gluster-home-lab-3: Lifetime : 25774secs,
> >>172800usecs
> >>[2016-10-19 15:04:48.333183] I [MSGID: 115029]
> >>[server-handshake.c:692:server_setvolume] 0-home-lab-3-server:
> >>accepted client from cuentas09-lab.lab.dit.upm.es-1610-2016/10/19-15:04:46:807972-home-lab-3-client-3-0-0
> >>(version: 3.8.5)
> >>[2016-10-19 15:27:27.763705] E [MSGID: 115070]
> >>[server-rpc-fops.c:1472:server_open_cbk] 0-home-lab-3-server: 149:
> >>OPEN /u056/.Xauthority-c (a4fa9bc4-a09e-4eb8-968b-b77cb6d49fd3) ==>
> >>(Permission denied) [Permission denied]
> >>[2016-10-19 15:29:12.202792] E [MSGID: 115070]
> >>[server-rpc-fops.c:1472:server_open_cbk] 0-home-lab-3-server: 267:
> >>OPEN /u056/.Xauthority-c (ce222a86-9f74-42d1-8e18-1f80f607d766) ==>
> >>(Permission denied) [Permission denied]
> >>root@admin2:~#
> >>
> >>As you can see, the glusterfs client part of the server
> >>fuse-mounting the volume shows a "/u056/kk.txt => -1 (Permission
> >>denied)" error that comes from a similar error in the glusterfs
> >>cluster, while at the nfs-kernel server log we can see a kernel
> >>trace directly related to the glusterfs fuse mount of the volume
> >>that says "nfsd4_process_open2 failed to open newly-created file!
> >>status=13".
> >>
> >>This error can be reproduced any number of times exactly the same,
> >>is not sporadic.
> >>
> >>Other test I've done at the client side with no logs at all at the
> >>server side, but the same error for the user, is the following:
> >>
> >>u056@l056:~$ rm -f 444.txt; echo "prueba" > 444.txt; chmod 444
> >>444.txt; echo "kk" > 444.txt
> >>-bash: 444.txt: Permission denied
> >>u056@l056:~$
> >>
> >>We have been trying to reproduce this problem in a virtual
> >>environment using KVM, nfs-kernel-server, glusterfs, and ZFS over
> >>VFS, being unable to do it... This leads us to think that creating
> >>glusterfs volumes over ZFS over phisical disks has something to do
> >>with this problem, but we have no clue on how to track it down to
> >>the ZFS level, as there are no traces on the kernel.log or the
> >>syslog that we can use.
> >>
> >>Thank you for your time.
> >>
> >>Omar
> >>
> >>El 21/12/15 a las 21:14, J. Bruce Fields escribió:
> >>>On Mon, Dec 21, 2015 at 11:28:36PM +0530, Soumya Koduri wrote:
> >>>>On 12/21/2015 10:17 PM, J. Bruce Fields wrote:
> >>>>>On Mon, Dec 21, 2015 at 02:18:20PM +0530, Soumya Koduri wrote:
> >>>>>>On 12/19/2015 01:38 AM, J. Bruce Fields wrote:
> >>>>>>>On Fri, Dec 18, 2015 at 10:47:42PM +0530, Soumya Koduri wrote:
> >>>>>>>>On 12/18/2015 08:50 PM, J. Bruce Fields wrote:
> >>>>>>>>>On Fri, Dec 18, 2015 at 02:13:40PM +0530, Soumya Koduri wrote:
> >>>>>>>>>>On 12/18/2015 06:07 AM, Malahal Naineni wrote:
> >>>>>>>>>>>IIRC, permission checks are done in open(). write/read syscalls should
> >>>>>>>>>>>NOT do much access checks (at least based on POSIX). This is why once an
> >>>>>>>>>>>open is done, you remove permissions for that process, but it should
> >>>>>>>>>>>still be able to read/write based on the open flags it did when it
> >>>>>>>>>>>opened the file.
> >>>>>>>>>>>
> >>>>>>>>>>>I don't know all the details of this defect, but gluster seems to be
> >>>>>>>>>>>doing what it is supposed to do.
> >>>>>>>>>>>
> >>>>>>>>>>Right. Thanks for the correction. I assumed the behavior should be
> >>>>>>>>>>same for both OPEN+WRITE vs CREATE+WRITE in the below scenario. But
> >>>>>>>>>>looks like (from 'man creat') the open() call that creates a
> >>>>>>>>>>read-only file may well return a read/write file descriptor, which
> >>>>>>>>>>is the reason the following WRITE can succeed.
> >>>>>>>>>I forgot another complication, which is that knsfd actually does a
> >>>>>>>>>temporary open before each read or write--I assume that's getting
> >>>>>>>>>translated into fuse and gluster open operations?
> >>>>>>>>>
> >>>>>>>>yes. It is the OPEN done as part of NFS WRITE which fails with
> >>>>>>>>EACCESS error (with both NFSv3 and NFSv4 mounts).
> >>>>>>>Makes sense for v3, but I wouldn't normally expect the extra temporary
> >>>>>>>open on v4 WRITEs. Could you share any details?
> >>>>>>>
> >>>>>>I re-tried the test on v4 mount using Fedora23 machine, acting as
> >>>>>>both NFS server and client (Linux#4.2.3-300.fc23.x86_64). Please
> >>>>>>find the pkt trace attached.
> >>>>>>
> >>>>>> 56 07:23:25.567134 ::1 -> ::1 NFS 288 V4 Call
> >>>>>>WRITE StateID: 0xf934 Offset: 0 Len: 7
> >>>>>> 57 07:23:25.567233 192.168.122.17 -> 192.168.122.202 GlusterFS 188
> >>>>>>V330 GETXATTR Call
> >>>>>> 58 07:23:25.567732 192.168.122.202 -> 192.168.122.17 GlusterFS 112
> >>>>>>V330 GETXATTR Reply (Call In 57)
> >>>>>> 59 07:23:25.567881 192.168.122.17 -> 192.168.122.202 GlusterFS 164
> >>>>>>V330 OPEN Call
> >>>>>Remind me what kernel version your server is on?
> >>>>NFS server is on fedora23 VM - Linux version 4.2.3-300.fc23.x86_64
> >>>We did reshuffle the code that does the temporary open on WRITE around
> >>>there--but it looks right to me, and I can't reproduce an open on WRITE
> >>>on that kernel myself.
> >>>
> >>>Maybe there's some further fuse or gluster debugging that would help
> >>>show where that open is coming from.
> >>>
> >>>--b.
> >>>
> >>>>Thanks,
> >>>>Soumya
> >>>>
> >>>>>--b.
> >>>>>
> >>>>>
> >>>>>> 60 07:23:25.568354 192.168.122.202 -> 192.168.122.17 GlusterFS 116
> >>>>>>V330 OPEN Reply (Call In 59)
> >>>>>> 61 07:23:25.568570 ::1 -> ::1 NFS 144 V4 Reply
> >>>>>>(Call In 56) WRITE Status: NFS4ERR_ACCESS
> >>>>>>
> >>>>>>Thanks,
> >>>>>>Soumya
> >>>>>>
> >>>>>>>--b.
> >>>>>>>
> >>>>>>>> 63 16:59:09.278651000 ::1 -> ::1 NFS 232 V3 WRITE
> >>>>>>>>Call, FH: 0x49a35e54 Offset: 0 Len: 7 FILE_SYNC
> >>>>>>>> 64 16:59:09.278926000 192.168.122.1 -> 192.168.122.202 GlusterFS
> >>>>>>>>164 V330 OPEN Call
> >>>>>>>> 65 16:59:09.278937000 192.168.122.1 -> 192.168.122.202 GlusterFS
> >>>>>>>>164 [RPC retransmission of #64][TCP Retransmission] V330 OPEN Call
> >>>>>>>> 66 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
> >>>>>>>>116 V330 OPEN Reply (Call In 64)
> >>>>>>>> 67 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
> >>>>>>>>116 [RPC duplicate of #66][TCP Retransmission] V330 OPEN Reply (Call
> >>>>>>>>In 64)
> >>>>>>>> 68 16:59:09.279733000 ::1 -> ::1 NFS 212 V3 WRITE
> >>>>>>>>Reply (Call In 63) Error: NFS3ERR_ACCES
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>Thanks,
> >>>>>>>>Soumya
> >>>>>>>>
> >>>>>>>>>In which case it might be worth experimenting with NFSv4 or with Jeff
> >>>>>>>>>Layton's filehandle-caching patches. Neither's a real fix, but that
> >>>>>>>>>could help confirm whether it's the temporary opens that are a problem.
> >>>>>>>>>
> >>>>>>>>>--b.
> >>>>>>>>>
> >>>>>>>>>>Thanks,
> >>>>>>>>>>Soumya
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>Regards, Malahal.
> >>>>>>>>>>>
> >>>>>>>>>>>Soumya Koduri [[email protected]] wrote:
> >>>>>>>>>>>>As mentioned by Bruce, GlusterFS doesn't have owner-override rule
> >>>>>>>>>>>>except for setattr.
> >>>>>>>>>>>>
> >>>>>>>>>>>>I did few experiments to check why this test case passes on plain
> >>>>>>>>>>>>glusterfs fuse mount & NFS-Ganesha but fails with kernel-NFS.
> >>>>>>>>>>>>
> >>>>>>>>>>>>NFS-Ganesha (for most of the FSALs) seem to be passing the actual
> >>>>>>>>>>>>request credentials to the back-end filesystem only for
> >>>>>>>>>>>>CREATE(-like) and UNLINK fops. For all the remaining fops, it does
> >>>>>>>>>>>>the access check at its end and then perform the operation with root
> >>>>>>>>>>>>credentials. That's the reason WRITE succeeded in your case as
> >>>>>>>>>>>>NFS-Ganesha (like kernel-NFS) skipped the access check if the
> >>>>>>>>>>>>request caller_uid proved to be the file's owner.
> >>>>>>>>>>>>
> >>>>>>>>>>>>In case of native GlusterFS FUSE mount, there is no OPEN fop
> >>>>>>>>>>>>involved. WRITE is performed on the fd returned by CREATE. And
> >>>>>>>>>>>>strangely GlusterFS seem to be doing certain access checks only
> >>>>>>>>>>>>during OPEN but not for WRITE (this seems like a bug and probably
> >>>>>>>>>>>>needs to be fixed in Gluster).
> >>>>>>>>>>>>
> >>>>>>>>>>>>Thanks,
> >>>>>>>>>>>>Soumya
> >>>>>>>>>>>>
> >>>>>>>>>>>>On 12/14/2015 10:27 PM, Omar Walid Llorente wrote:
> >>>>>>>>>>>>>Thank you Bruce, others, for the responses. I send attached a complete
> >>>>>>>>>>>>>capture of the issue, including the glusterfs transactions.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>Hope this helps to clear where may it be...
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>Omar
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>El 10/12/15 a las 15:44, J. Bruce Fields escribió:
> >>>>>>>>>>>>>>On Thu, Dec 10, 2015 at 05:59:33PM +0530, Soumya Koduri wrote:
> >>>>>>>>>>>>>>>On 12/10/2015 04:02 PM, Omar Walid Llorente wrote:
> >>>>>>>>>>>>>>>>Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs
> >>>>>>>>>>>>>>>>packets (you can find them in attached file nfs-problem-nks.pcap.zip).
> >>>>>>>>>>>>>>>>Sorry for being so late.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>What I did was the following:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>1st) Create the RO file:
> >>>>>>>>>>>>>>>>cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
> >>>>>>>>>>>>>>>>chmod 444 444.txt;
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>2nd) Init the capture:
> >>>>>>>>>>>>>>>>root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
> >>>>>>>>>>>>>>>>tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
> >>>>>>>>>>>>>>>>512 bytes
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It
> >>>>>>>>>>>>>>>may be helpful to see what GlusterFS operations are being processed
> >>>>>>>>>>>>>>>as part of NFS WRITE call (which has failed in this case).
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>Could you please try taking the packet trace on the machine where
> >>>>>>>>>>>>>>>NFS server is running (without filtering out based on the port
> >>>>>>>>>>>>>>>number).
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>Also I tried out the same test on Fedora22 machine, but haven't run
> >>>>>>>>>>>>>>>into any issue. What are the fuse mount options you have used to
> >>>>>>>>>>>>>>>mount gluster volume?
> >>>>>>>>>>>>>>Oh, I think this is a simple problem (but maybe hard to fix). The
> >>>>>>>>>>>>>>capture shows NFSv3 traffic like:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> CREATE -> OK
> >>>>>>>>>>>>>> SETATTR (mode set to 0400) -> OK
> >>>>>>>>>>>>>> WRITE -> NFS3ERR_ACCES
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>That write would succeed locally (because the mode doesn't matter to a
> >>>>>>>>>>>>>>local application that already holds the file open). It would fail over
> >>>>>>>>>>>>>>NFSv3, which doesn't know about the open--except that there's a hack for
> >>>>>>>>>>>>>>this case: NFSv3 servers allow IO operations to ignore the mode, if the
> >>>>>>>>>>>>>>operation comes from the owner of the file. NFSv3 clients are then
> >>>>>>>>>>>>>>careful to perform necessary access checks on open to ensure that this
> >>>>>>>>>>>>>>owner-override rule doesn't grant too many permissions.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>That allows NFSv3 applications to see behavior that's mostly like a
> >>>>>>>>>>>>>>local filesystem, without opening much of a security hole (since the
> >>>>>>>>>>>>>>owner could always chmod anyway).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>So, knfsd is making this special exception--but gluster (which I believe
> >>>>>>>>>>>>>>it's exporting in this case, via fuse?)--probably doesn't.... I'm not
> >>>>>>>>>>>>>>sure what you can do about that.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>--b.
> >>>>>>>>>>>>--
> >>>>>>>>>>>>To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> >>>>>>>>>>>>the body of a message to [email protected]
> >>>>>>>>>>>>More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>>>>>>>>>>>
> >>>>>>>--
> >>>>>>>To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> >>>>>>>the body of a message to [email protected]
> >>>>>>>More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>>>>>>
> >>--
> >>----------------------------------------------------------------
> >>Centro de Cálculo Depto. Ingeniería Sistemas Telemáticos
> >>E-mail: [email protected] Universidad Politécnica de Madrid
> >>Fax:(+34) 913367333 E.T.S. Ing. Telecomunicación
> >>Tel:(+34) 915495700-Ext.3005 28040 Madrid (Spain)
> >>Tel:(+34) 915495762-Ext.3005
> >>Tel:(+34) 913367366-Ext.3005
> >>----------------------------------------------------------------
> >>
>
> --
> ----------------------------------------------------------------
> Centro de Cálculo Depto. Ingeniería Sistemas Telemáticos
> E-mail: [email protected] Universidad Politécnica de Madrid
> Fax:(+34) 913367333 E.T.S. Ing. Telecomunicación
> Tel:(+34) 915495700-Ext.3005 28040 Madrid (Spain)
> Tel:(+34) 915495762-Ext.3005
> Tel:(+34) 913367366-Ext.3005
> ----------------------------------------------------------------

2016-11-11 22:04:26

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

Sorry, resending with Miklos's address fixed....

On Fri, Nov 11, 2016 at 06:57:45PM +0100, Omar Walid Llorente wrote:
> Hi Bruce, thanks for your answer, that is greatly appreciated.
>
> This side of the problem feels a little bit lonely and maybe we can
> manage to get to know what is happening if you have some time.
>
> We have been doing tests and we have found a way to replicate the
> problem in a virtual environment without ZFS, only with glusterfs
> and nfs-kernel-server, where we can give you root password or you
> can replicate it at home.
>
> It is closely related to non-root users (with root credentials the
> problem does not happen).
>
> If you think this is good news and feel like checking, tell me and
> next week we will prepare it and I'll forward you the data for both
> kind of tests.

So if I'm reading right, the setup is

nfs client---NFSv3--->knfsd---FUSE-->gluster

I hate to just pass the buck, but my bet is on the issue being either on
the gluster side, or on limitations of the fuse protocol itself.

Cc'ing Miklos as I believe he did the original fuse nfsd export
implementation. Miklos, does fuse has a way for file owners to override
permissions? We need this sometimes because for example, an application
may hold a write open on a read-only file, but the server doesn't itself
already have an open (which can happen for lots of reasons).

--b.

2016-11-15 10:13:29

by Miklos Szeredi

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

On Fri, Nov 11, 2016 at 11:04 PM, J. Bruce Fields <[email protected]> wrote:
> So if I'm reading right, the setup is
>
> nfs client---NFSv3--->knfsd---FUSE-->gluster
>
> I hate to just pass the buck, but my bet is on the issue being either on
> the gluster side, or on limitations of the fuse protocol itself.
>
> Cc'ing Miklos as I believe he did the original fuse nfsd export
> implementation. Miklos, does fuse has a way for file owners to override
> permissions? We need this sometimes because for example, an application
> may hold a write open on a read-only file, but the server doesn't itself
> already have an open (which can happen for lots of reasons).

Is glusterfs using "default_permissions" fuse option?

If so, then the above shouldn't be an issue, since permission checking
is done by generic_permission() which works the same for all
filesystems.

If not, then there's the same problem as re-exporting NFSv4 as NFSv3
would have: the permissions are not checked in ->permission(), but in
e.g. ->atomic_open() so nfsd's override mechanism wouldn't work.

Thanks,
Miklos

2016-11-16 18:20:18

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

Hi Miklos, thanks for your time and your work.

AFAIK, we are using the "default_permissions" fuse option, along with
"allow_other" fuse option, among others. See the /proc/mounts of the
glusterfs-client and nfs-server behind:

root@cuentas03-lab:/etc# cat /proc/mounts
rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,relatime,size=2012892k,nr_inodes=503223,mode=755 0 0
devpts /dev/pts devpts
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=404752k,mode=755 0 0
/dev/sda2 / ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0
none /sys/fs/fuse/connections fusectl rw,relatime 0 0
none /sys/kernel/debug debugfs rw,relatime 0 0
none /sys/kernel/security securityfs rw,relatime 0 0
none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0
none /run/user tmpfs
rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755 0 0
none /sys/fs/pstore pstore rw,relatime 0 0
rpc_pipefs /run/rpc_pipefs rpc_pipefs rw,relatime 0 0
systemd /sys/fs/cgroup/systemd cgroup
rw,nosuid,nodev,noexec,relatime,name=systemd 0 0
recipiente6hd:/home-lab-3.tcp /home-3-old fuse.glusterfs
rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072
0 0
root@cuentas03-lab:/etc#

In any case, the kernel we are using (3.13.0-96-generic #143-Ubuntu SMP
Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux) has fuse
included (not as a module) so if it would be necessary to change the
fuse mount options some help would be needed.

Would you like to have access as root to a fully functional virtual
environment where the problem can be reproduced?

Thanks again, and if above is not possible for you, please tell us how
we can manage to debug the issue. See at end a log of the last tests.

Omar

# At the nfs-client:

cdc@l056:~$ cat /proc/mounts | grep "/home/cdc"
cuentas03:/home-3-old/cdc /home/cdc nfs
rw,noatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=138.4.30.18,mountvers=3,mountport=49531,mountproto=tcp,fsc,local_lock=all,addr=138.4.30.18
0 0
cdc@l056:~$
cdc@l056:~$
cdc@l056:~$
cdc@l056:~$
cdc@l056:~$
cdc@l056:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444
444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
cp: failed to close ‘kk.txt’: Permission denied
-r--r--r-- 1 cdc admincdc 7 nov 16 2016 444.txt
-r--r--r-- 1 cdc admincdc 0 nov 16 2016 kk.txt
cdc@l056:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444
444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
cp: failed to close ‘kk.txt’: Permission denied
-r--r--r-- 1 cdc admincdc 7 nov 16 2016 444.txt
-r--r--r-- 1 cdc admincdc 0 nov 16 2016 kk.txt
cdc@l056:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444
444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
cp: failed to close ‘kk.txt’: Permission denied
-r--r--r-- 1 cdc admincdc 7 nov 16 2016 444.txt
-r--r--r-- 1 cdc admincdc 0 nov 16 2016 kk.txt
cdc@l056:~$
cdc@l056:~$
cdc@l056:~$
cdc@l056:~$
cdc@l056:~$ sudo su
root@l056:/home/cdc# rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
-r--r--r-- 1 root root 7 nov 16 2016 444.txt
-r--r--r-- 1 root root 7 nov 16 2016 kk.txt
root@l056:/home/cdc# rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
-r--r--r-- 1 root root 7 nov 16 2016 444.txt
-r--r--r-- 1 root root 7 nov 16 2016 kk.txt
root@l056:/home/cdc# rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
-r--r--r-- 1 root root 7 nov 16 2016 444.txt
-r--r--r-- 1 root root 7 nov 16 2016 kk.txt
root@l056:/home/cdc# exit
cdc@l056:~$

El 15/11/16 a las 11:13, Miklos Szeredi escribió:
> On Fri, Nov 11, 2016 at 11:04 PM, J. Bruce Fields <[email protected]> wrote:
>> So if I'm reading right, the setup is
>>
>> nfs client---NFSv3--->knfsd---FUSE-->gluster
>>
>> I hate to just pass the buck, but my bet is on the issue being either on
>> the gluster side, or on limitations of the fuse protocol itself.
>>
>> Cc'ing Miklos as I believe he did the original fuse nfsd export
>> implementation. Miklos, does fuse has a way for file owners to override
>> permissions? We need this sometimes because for example, an application
>> may hold a write open on a read-only file, but the server doesn't itself
>> already have an open (which can happen for lots of reasons).
> Is glusterfs using "default_permissions" fuse option?
>
> If so, then the above shouldn't be an issue, since permission checking
> is done by generic_permission() which works the same for all
> filesystems.
>
> If not, then there's the same problem as re-exporting NFSv4 as NFSv3
> would have: the permissions are not checked in ->permission(), but in
> e.g. ->atomic_open() so nfsd's override mechanism wouldn't work.
>
> Thanks,
> Miklos

--
----------------------------------------------------------------
Centro de Cálculo Depto. Ingeniería Sistemas Telemáticos
E-mail: [email protected] Universidad Politécnica de Madrid
Fax:(+34) 913367333 E.T.S. Ing. Telecomunicación
Tel:(+34) 915495700-Ext.3005 28040 Madrid (Spain)
Tel:(+34) 915495762-Ext.3005
Tel:(+34) 913367366-Ext.3005
----------------------------------------------------------------

2016-11-18 14:16:26

by Miklos Szeredi

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

On Wed, Nov 16, 2016 at 7:19 PM, Omar Walid Llorente <[email protected]> wrote:

> In any case, the kernel we are using (3.13.0-96-generic #143-Ubuntu SMP Mon
> Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux) has fuse included
> (not as a module) so if it would be necessary to change the fuse mount
> options some help would be needed.

Can you reproduce this on latest (e.g 4.8) kernel?

Also fuse debug logs (with whaterver kernel) could be useful; not sure
how to enable it for glusterfs, usual convention for fuse filesystems
is "-d".

Thanks,
Miklos

2016-11-18 16:03:24

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

Yes, of course, we can reproduce the problem on latest (e.g 4.8) kernel.

Tested on a server with fedora 24 (Linux cuentas11.lab.dit.upm.es
4.8.4-200.fc24.x86_64 #1 SMP Tue Oct 25 13:06:04 UTC 2016 x86_64 x86_64
x86_64 GNU/Linux), glusterfs-fuse-3.8.5-1.fc24.x86_64,
fuse-2.9.7-1.fc24.x86_64 and nfs-utils-1.3.4-1.rc2.fc24.x86_64 that
mounts as a glusterfs-client and exports as a nfs-server a ditributed
glusterfs volume (that comes from a 3.7.4 glusterfs server over Ubuntu
14.04.

[root@cuentas11 ~]# cat /proc/mounts | grep home
recipiente8hd:/home-lab-3.tcp /home-3 fuse.glusterfs
rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072
0 0
[root@cuentas11 ~]#

As you may remember the schema is the one that follows (glusterfs volume
name: home-lab-3, glusterfs client mount point: /home-3, nfs client
mount point: /home):

nfs client (v1.2.8) ---> NFSv3 ---> knfsd (v1.3.4 over a Linux v4.8.4
kernel) ---> FUSE (v2.9.7) --> glusterfs client (v3.8.5) --> GLUSTER
over TCP --> glusterfs server (v3.7.4)

The errors at the nfs-client side are the same:

cdc@l056:~$ mount | grep "/home/cdc"
cuentas11:/home-3/cdc on /home/cdc type nfs
(rw,noatime,nfsvers=3,vers=3,tcp,intr,fsc,nolock,rsize=262140,wsize=262140,addr=138.4.30.86)
cdc@l056:~$
cdc@l056:~$
cdc@l056:~$
cdc@l056:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444
444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
cp: failed to close ‘kk.txt’: Permission denied
-r--r--r-- 1 cdc admincdc 7 nov 18 2016 444.txt
-r--r--r-- 1 cdc admincdc 0 nov 18 2016 kk.txt
cdc@l056:~$
cdc@l056:~$
cdc@l056:~$
cdc@l056:~$ sudo su
[sudo] password for cdc:
root@l056:/home/cdc# rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
-r--r--r-- 1 root root 7 nov 18 2016 444.txt
-r--r--r-- 1 root root 7 nov 18 2016 kk.txt
root@l056:/home/cdc#
root@l056:/home/cdc#
root@l056:/home/cdc# exit
cdc@l056:~$

See at the end the logs with and without the debug option activated at
the glusterfs client.

Thanks again, and sorry for the long message.

Omar

* Logs on the glusterfs client side with NO DEBUG:

[2016-11-18 15:45:43.681215] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-3: remote
operation failed. Path: /cdc/kk.txt
(cc98e831-3410-4202-8254-2d4b48e8ea68) [Permission denied]
[2016-11-18 15:45:43.681297] W [fuse-bridge.c:989:fuse_fd_cbk]
0-glusterfs-fuse: 399: OPEN() /cdc/kk.txt => -1 (Permission denied)
[2016-11-18 15:45:55.457922] I [io-stats.c:1574:io_stats_dump_fd]
0-home-lab-3: --- fd stats ---
[2016-11-18 15:45:55.457978] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2016-11-18 15:45:55.457997] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 15:45:55.458011] I [io-stats.c:1606:io_stats_dump_fd]
0-home-lab-3: Write 000004b+ : 1
[2016-11-18 15:45:55.471933] I [io-stats.c:1574:io_stats_dump_fd]
0-home-lab-3: --- fd stats ---

* Logs on the glusterfs client side with DEBUG activated:

[2016-11-18 16:06:17.040092] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
as fast fop
[2016-11-18 16:06:17.045986] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
scheduled as fast fop
[2016-11-18 16:06:17.046132] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
home-lab-3-client-3
[2016-11-18 16:06:17.046833] D [MSGID: 0]
[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
returned for /cdc/kk.txt with op_ret 0 [Invalid argument]
[2016-11-18 16:06:17.047671] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK
scheduled as normal fop
[2016-11-18 16:06:17.050539] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph -1418558531
[2016-11-18 16:06:17.050867] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:06:17.051948] D [logging.c:1953:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
[2016-11-18 16:06:17.051871] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
scheduled as fast fop
[2016-11-18 16:06:17.051946] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
home-lab-3-client-0
[2016-11-18 16:06:17.062108] D [MSGID: 0]
[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
returned for /cdc/444.txt with op_ret 0 [Invalid argument]
[2016-11-18 16:06:17.063692] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph -1418558531
[2016-11-18 16:06:17.064015] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:06:17.065068] D [logging.c:1953:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 4 times
between [2016-11-18 16:06:17.040092] and [2016-11-18 16:06:17.064175]
[2016-11-18 16:06:17.065066] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
scheduled as fast fop
[2016-11-18 16:06:17.065232] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
home-lab-3-client-0
[2016-11-18 16:06:17.065755] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.065897] D [MSGID: 0]
[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
returned for /cdc/444.txt with op_ret -1 [No such file or directory]
[2016-11-18 16:06:17.066016] D [MSGID: 0]
[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
missing on subvol home-lab-3-client-0
[2016-11-18 16:06:17.066104] D [MSGID: 0]
[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
lookup call to 4 subvols
[2016-11-18 16:06:17.066594] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.066669] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:06:17.066771] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.066898] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:06:17.066906] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.067019] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:06:17.066915] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.067066] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:06:17.067107] D [MSGID: 0]
[dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null
[2016-11-18 16:06:17.067151] D [MSGID: 0]
[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
was no cached file and unlink on hashed is not skipped /cdc/444.txt
[2016-11-18 16:06:17.067179] D [MSGID: 0]
[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1 error: No such
file or directory [No such file or directory]
[2016-11-18 16:06:17.067206] D [MSGID: 0]
[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.067250] D [MSGID: 0]
[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.067275] D [MSGID: 0]
[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.067308] D [MSGID: 0]
[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.067387] D [MSGID: 0]
[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:06:17.068157] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
scheduled as fast fop
[2016-11-18 16:06:17.068281] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
home-lab-3-client-0
[2016-11-18 16:06:17.068738] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.068781] D [MSGID: 0]
[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
returned for /cdc/444.txt with op_ret -1 [No such file or directory]
[2016-11-18 16:06:17.068805] D [MSGID: 0]
[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
missing on subvol home-lab-3-client-0
[2016-11-18 16:06:17.068827] D [MSGID: 0]
[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
lookup call to 4 subvols
[2016-11-18 16:06:17.069286] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.069345] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:06:17.069452] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.069507] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:06:17.069556] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.069585] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:06:17.069939] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.069983] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:06:17.070005] D [MSGID: 0]
[dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null
[2016-11-18 16:06:17.070038] D [MSGID: 0]
[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
was no cached file and unlink on hashed is not skipped /cdc/444.txt
[2016-11-18 16:06:17.070098] D [MSGID: 0]
[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1 error: No such
file or directory [No such file or directory]
[2016-11-18 16:06:17.070128] D [MSGID: 0]
[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.070154] D [MSGID: 0]
[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.070178] D [MSGID: 0]
[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.070208] D [MSGID: 0]
[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.070251] D [MSGID: 0]
[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:06:17.070537] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled
as normal fop
[2016-11-18 16:06:17.070826] D [MSGID: 0]
[dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:06:17.071100] D [MSGID: 0]
[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
10456189239296 and avail_inodes is: 99.00
[2016-11-18 16:06:17.071142] D [MSGID: 0]
[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
10499693871104 and avail_inodes is: 99.00
[2016-11-18 16:06:17.071183] D [MSGID: 0]
[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
10500911267840 and avail_inodes is: 99.00
[2016-11-18 16:06:17.071343] D [MSGID: 0]
[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
8600432410624 and avail_inodes is: 99.00
[2016-11-18 16:06:17.073943] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph -1418558531
[2016-11-18 16:06:17.074273] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
as fast fop
[2016-11-18 16:06:17.075035] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled
as fast fop
[2016-11-18 16:06:17.075913] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled
as slow fop
[2016-11-18 16:06:17.076205] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled
as fast fop
[2016-11-18 16:06:17.076636] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:06:17.077282] D [fuse-bridge.c:52:fuse_invalidate]
0-fuse: Invalidate inode id 140363155120276.
[2016-11-18 16:06:17.077981] I [io-stats.c:1574:io_stats_dump_fd]
0-home-lab-3: --- fd stats ---
[2016-11-18 16:06:17.078032] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2016-11-18 16:06:17.078049] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 16:06:17.078064] I [io-stats.c:1606:io_stats_dump_fd]
0-home-lab-3: Write 000004b+ : 1
[2016-11-18 16:06:17.079156] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
scheduled as normal fop
[2016-11-18 16:06:17.079794] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:06:17.082196] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
scheduled as fast fop
[2016-11-18 16:06:17.082772] D [logging.c:1953:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
[2016-11-18 16:06:17.082318] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
home-lab-3-client-3
[2016-11-18 16:06:17.082770] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.083168] D [MSGID: 0]
[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
[2016-11-18 16:06:17.083282] D [MSGID: 0]
[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
missing on subvol home-lab-3-client-3
[2016-11-18 16:06:17.083320] D [MSGID: 0]
[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
lookup call to 4 subvols
[2016-11-18 16:06:17.083991] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.084043] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
[2016-11-18 16:06:17.084048] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.084092] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
[2016-11-18 16:06:17.084114] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.084167] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
[2016-11-18 16:06:17.084091] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.084377] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
[2016-11-18 16:06:17.084406] D [MSGID: 0]
[dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null
[2016-11-18 16:06:17.084511] D [MSGID: 0]
[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
was no cached file and unlink on hashed is not skipped /cdc/kk.txt
[2016-11-18 16:06:17.084534] D [MSGID: 0]
[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
file or directory [No such file or directory]
[2016-11-18 16:06:17.084683] D [MSGID: 0]
[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.084783] D [MSGID: 0]
[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.084857] D [MSGID: 0]
[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.084889] D [MSGID: 0]
[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.085001] D [MSGID: 0]
[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:06:17.086098] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
as fast fop
[2016-11-18 16:06:17.087256] D [logging.c:1953:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
[2016-11-18 16:06:17.087173] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
scheduled as fast fop
[2016-11-18 16:06:17.087254] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
home-lab-3-client-3
[2016-11-18 16:06:17.087749] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.087795] D [MSGID: 0]
[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
[2016-11-18 16:06:17.088014] D [MSGID: 0]
[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
missing on subvol home-lab-3-client-3
[2016-11-18 16:06:17.088036] D [MSGID: 0]
[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
lookup call to 4 subvols
[2016-11-18 16:06:17.088598] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.088639] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
[2016-11-18 16:06:17.088804] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.088855] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
[2016-11-18 16:06:17.088819] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.089069] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
[2016-11-18 16:06:17.089380] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:06:17.089421] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
[2016-11-18 16:06:17.089443] D [MSGID: 0]
[dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null
[2016-11-18 16:06:17.089463] D [MSGID: 0]
[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
was no cached file and unlink on hashed is not skipped /cdc/kk.txt
[2016-11-18 16:06:17.089575] D [MSGID: 0]
[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
file or directory [No such file or directory]
[2016-11-18 16:06:17.089601] D [MSGID: 0]
[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.089712] D [MSGID: 0]
[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.089738] D [MSGID: 0]
[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.090006] D [MSGID: 0]
[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.090156] D [MSGID: 0]
[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:06:17.090648] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled
as normal fop
[2016-11-18 16:06:17.090732] D [MSGID: 0]
[dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:06:17.091499] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph -1418558531
[2016-11-18 16:06:17.091843] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
scheduled as normal fop
[2016-11-18 16:06:17.093831] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:06:17.094711] D [logging.c:1953:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
[2016-11-18 16:06:17.093216] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
scheduled as normal fop
[2016-11-18 16:06:17.094709] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled
as fast fop
[2016-11-18 16:06:17.095280] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-3: remote
operation failed. Path: /cdc/kk.txt
(09bfb6fa-82fb-463b-abd9-bb7ef88e10c0) [Permission denied]
[2016-11-18 16:06:17.095341] D [MSGID: 0]
[client-rpc-fops.c:448:client3_3_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: Permission denied
[Permission denied]
[2016-11-18 16:06:17.095366] D [MSGID: 0]
[dht-inode-read.c:45:dht_open_cbk] 0-home-lab-3-dht: subvolume
home-lab-3-client-3 returned -1 [Permission denied]
[2016-11-18 16:06:17.095480] D [MSGID: 0]
[dht-inode-read.c:59:dht_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-dht returned -1 error: Permission denied
[Permission denied]
[2016-11-18 16:06:17.095671] D [MSGID: 0] [read-ahead.c:101:ra_open_cbk]
0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-read-ahead
returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.095759] D [MSGID: 0] [io-cache.c:605:ioc_open_cbk]
0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-io-cache
returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.095963] D [MSGID: 0]
[defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: Permission
denied [Permission denied]
[2016-11-18 16:06:17.095988] D [MSGID: 0]
[defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: Permission
denied [Permission denied]
[2016-11-18 16:06:17.096275] D [MSGID: 0]
[io-stats.c:1936:io_stats_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: Permission denied
[Permission denied]
[2016-11-18 16:06:17.096362] W [fuse-bridge.c:989:fuse_fd_cbk]
0-glusterfs-fuse: 338: OPEN() /cdc/kk.txt => -1 (Permission denied)
[2016-11-18 16:06:17.097104] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
scheduled as normal fop
[2016-11-18 16:06:17.097674] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:06:17.099912] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:06:24.785225] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
0-home-lab-3-client-0: returning as transport is already disconnected OR
there are no frames (0 || 0)
[2016-11-18 16:06:24.785338] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
0-home-lab-3-client-1: returning as transport is already disconnected OR
there are no frames (0 || 0)
[2016-11-18 16:06:24.785359] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
0-home-lab-3-client-2: returning as transport is already disconnected OR
there are no frames (0 || 0)
[2016-11-18 16:06:24.785377] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
0-home-lab-3-client-3: returning as transport is already disconnected OR
there are no frames (0 || 0)

** TEST DONE as user cdc --> ERROR

[2016-11-18 16:06:46.737686] D
[rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
))))) 0-: 138.4.2.246:49172: ping timer event already removed
[2016-11-18 16:06:49.545698] D
[rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
))))) 0-: 138.4.2.177:49157: ping timer event already removed
[2016-11-18 16:06:49.546474] D
[rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
))))) 0-: 138.4.2.189:49157: ping timer event already removed
[2016-11-18 16:06:49.547093] D
[rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
/lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
))))) 0-: 138.4.2.248:49157: ping timer event already removed

** user CDC becomed root at this point

[2016-11-18 16:07:11.907892] D [logging.c:1953:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 7 times
between [2016-11-18 16:06:17.086098] and [2016-11-18 16:07:11.906944]
[2016-11-18 16:07:11.907884] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
scheduled as fast fop
[2016-11-18 16:07:11.908875] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph -1418558531
[2016-11-18 16:07:11.909179] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:07:11.909256] D [logging.c:1953:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
0-home-lab-3-io-threads: UNLINK scheduled as normal fop" repeated 2
times between [2016-11-18 16:06:17.047671] and [2016-11-18 16:07:11.908114]
[2016-11-18 16:07:11.909254] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
as fast fop
[2016-11-18 16:07:11.911109] D [logging.c:1953:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
[2016-11-18 16:07:11.911018] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
scheduled as fast fop
[2016-11-18 16:07:11.911107] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK
scheduled as normal fop
[2016-11-18 16:07:11.911871] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph -1418558531
[2016-11-18 16:07:11.912045] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:07:11.913067] D [logging.c:1953:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 2 times
between [2016-11-18 16:07:11.909254] and [2016-11-18 16:07:11.912165]
[2016-11-18 16:07:11.913065] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
scheduled as fast fop
[2016-11-18 16:07:11.913193] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
home-lab-3-client-0
[2016-11-18 16:07:11.913710] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.913757] D [MSGID: 0]
[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
returned for /cdc/444.txt with op_ret -1 [No such file or directory]
[2016-11-18 16:07:11.913788] D [MSGID: 0]
[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
missing on subvol home-lab-3-client-0
[2016-11-18 16:07:11.913810] D [MSGID: 0]
[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
lookup call to 4 subvols
[2016-11-18 16:07:11.914336] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.914379] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:07:11.914424] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.914448] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:07:11.914501] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.914617] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:07:11.914649] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.914698] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:07:11.914724] D [MSGID: 0]
[dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null
[2016-11-18 16:07:11.914743] D [MSGID: 0]
[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
was no cached file and unlink on hashed is not skipped /cdc/444.txt
[2016-11-18 16:07:11.914764] D [MSGID: 0]
[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
file or directory [No such file or directory]
[2016-11-18 16:07:11.914791] D [MSGID: 0]
[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.914825] D [MSGID: 0]
[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.914855] D [MSGID: 0]
[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.914880] D [MSGID: 0]
[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.914947] D [MSGID: 0]
[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:07:11.915569] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
scheduled as fast fop
[2016-11-18 16:07:11.915644] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
home-lab-3-client-0
[2016-11-18 16:07:11.916090] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.916134] D [MSGID: 0]
[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
returned for /cdc/444.txt with op_ret -1 [No such file or directory]
[2016-11-18 16:07:11.916158] D [MSGID: 0]
[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
missing on subvol home-lab-3-client-0
[2016-11-18 16:07:11.916185] D [MSGID: 0]
[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
lookup call to 4 subvols
[2016-11-18 16:07:11.916593] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.916780] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:07:11.916878] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.916994] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:07:11.917076] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.916906] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.917117] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:07:11.917153] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:07:11.917205] D [MSGID: 0]
[dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null
[2016-11-18 16:07:11.917368] D [MSGID: 0]
[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
was no cached file and unlink on hashed is not skipped /cdc/444.txt
[2016-11-18 16:07:11.917437] D [MSGID: 0]
[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
file or directory [No such file or directory]
[2016-11-18 16:07:11.917467] D [MSGID: 0]
[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.917496] D [MSGID: 0]
[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.917521] D [MSGID: 0]
[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.917551] D [MSGID: 0]
[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.917726] D [MSGID: 0]
[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:07:11.917958] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled
as normal fop
[2016-11-18 16:07:11.918230] D [MSGID: 0]
[dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:07:11.918488] D [MSGID: 0]
[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
10456166432768 and avail_inodes is: 99.00
[2016-11-18 16:07:11.918570] D [MSGID: 0]
[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
8600417075200 and avail_inodes is: 99.00
[2016-11-18 16:07:11.918603] D [MSGID: 0]
[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
10500905500672 and avail_inodes is: 99.00
[2016-11-18 16:07:11.918675] D [MSGID: 0]
[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
10499686531072 and avail_inodes is: 99.00
[2016-11-18 16:07:11.919095] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph -1418558531
[2016-11-18 16:07:11.919377] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
as fast fop
[2016-11-18 16:07:11.920040] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled
as fast fop
[2016-11-18 16:07:11.920889] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled
as slow fop
[2016-11-18 16:07:11.921051] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled
as fast fop
[2016-11-18 16:07:11.921583] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:07:11.922068] D [fuse-bridge.c:52:fuse_invalidate]
0-fuse: Invalidate inode id 140363155120276.
[2016-11-18 16:07:11.922936] I [io-stats.c:1574:io_stats_dump_fd]
0-home-lab-3: --- fd stats ---
[2016-11-18 16:07:11.922988] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2016-11-18 16:07:11.923004] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 16:07:11.923026] I [io-stats.c:1606:io_stats_dump_fd]
0-home-lab-3: Write 000004b+ : 1
[2016-11-18 16:07:11.923779] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
as fast fop
[2016-11-18 16:07:11.924504] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:07:11.926692] D [logging.c:1953:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
0-home-lab-3-io-threads: SETATTR scheduled as normal fop" repeated 2
times between [2016-11-18 16:06:17.097104] and [2016-11-18 16:07:11.923946]
[2016-11-18 16:07:11.926690] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
scheduled as fast fop
[2016-11-18 16:07:11.926858] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
home-lab-3-client-3
[2016-11-18 16:07:11.927327] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.927376] D [MSGID: 0]
[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
[2016-11-18 16:07:11.927401] D [MSGID: 0]
[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
missing on subvol home-lab-3-client-3
[2016-11-18 16:07:11.927422] D [MSGID: 0]
[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
lookup call to 4 subvols
[2016-11-18 16:07:11.927940] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.927987] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
[2016-11-18 16:07:11.928043] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.928055] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.928058] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.928170] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
[2016-11-18 16:07:11.928174] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
[2016-11-18 16:07:11.928277] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
[2016-11-18 16:07:11.928360] D [MSGID: 0]
[dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null
[2016-11-18 16:07:11.928381] D [MSGID: 0]
[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
was no cached file and unlink on hashed is not skipped /cdc/kk.txt
[2016-11-18 16:07:11.928403] D [MSGID: 0]
[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
file or directory [No such file or directory]
[2016-11-18 16:07:11.928431] D [MSGID: 0]
[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.928456] D [MSGID: 0]
[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.928480] D [MSGID: 0]
[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.928511] D [MSGID: 0]
[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.928753] D [MSGID: 0]
[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:07:11.930485] D [logging.c:1953:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
[2016-11-18 16:07:11.929862] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
as fast fop
[2016-11-18 16:07:11.930483] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
scheduled as fast fop
[2016-11-18 16:07:11.930602] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
home-lab-3-client-3
[2016-11-18 16:07:11.931071] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.931124] D [MSGID: 0]
[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
[2016-11-18 16:07:11.931148] D [MSGID: 0]
[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
missing on subvol home-lab-3-client-3
[2016-11-18 16:07:11.931348] D [MSGID: 0]
[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
lookup call to 4 subvols
[2016-11-18 16:07:11.931880] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.931927] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
[2016-11-18 16:07:11.932002] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.931994] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.931991] D [MSGID: 0]
[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory]
[2016-11-18 16:07:11.932078] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
[2016-11-18 16:07:11.932116] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
[2016-11-18 16:07:11.932315] D [MSGID: 0]
[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
[2016-11-18 16:07:11.932363] D [MSGID: 0]
[dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null
[2016-11-18 16:07:11.932383] D [MSGID: 0]
[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
was no cached file and unlink on hashed is not skipped /cdc/kk.txt
[2016-11-18 16:07:11.932407] D [MSGID: 0]
[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
file or directory [No such file or directory]
[2016-11-18 16:07:11.932446] D [MSGID: 0]
[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.932471] D [MSGID: 0]
[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.932494] D [MSGID: 0]
[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.932518] D [MSGID: 0]
[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.932725] D [logging.c:1953:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
[2016-11-18 16:07:11.932545] D [MSGID: 0]
[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:07:11.932722] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled
as normal fop
[2016-11-18 16:07:11.933008] D [MSGID: 0]
[dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:07:11.934086] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph -1418558531
[2016-11-18 16:07:11.934323] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
scheduled as normal fop
[2016-11-18 16:07:11.934922] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
as fast fop
[2016-11-18 16:07:11.935497] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
scheduled as normal fop
[2016-11-18 16:07:11.936102] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:07:11.936903] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled
as fast fop
[2016-11-18 16:07:11.937540] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled
as slow fop
[2016-11-18 16:07:11.937682] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled
as fast fop
[2016-11-18 16:07:11.938267] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:07:11.938757] D [fuse-bridge.c:52:fuse_invalidate]
0-fuse: Invalidate inode id 140363155120652.
[2016-11-18 16:07:11.938994] I [io-stats.c:1574:io_stats_dump_fd]
0-home-lab-3: --- fd stats ---
[2016-11-18 16:07:11.939072] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/kk.txt
[2016-11-18 16:07:11.939096] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 16:07:11.939112] I [io-stats.c:1606:io_stats_dump_fd]
0-home-lab-3: Write 000004b+ : 1
[2016-11-18 16:07:11.939448] D [MSGID: 0]
[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
scheduled as normal fop
[2016-11-18 16:07:11.940028] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0
[2016-11-18 16:07:11.942130] D [fuse-bridge.c:5264:notify] 0-fuse: got
event 19 on graph 0

** TEST DONE as root --> NO PROBLEM

[2016-11-18 16:07:28.794150] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
0-home-lab-3-client-2: returning as transport is already disconnected OR
there are no frames (0 || 0)
[2016-11-18 16:07:31.794656] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
0-home-lab-3-client-0: returning as transport is already disconnected OR
there are no frames (0 || 0)
[2016-11-18 16:07:31.794925] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
0-home-lab-3-client-1: returning as transport is already disconnected OR
there are no frames (0 || 0)
[2016-11-18 16:07:31.794984] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
0-home-lab-3-client-3: returning as transport is already disconnected OR
there are no frames (0 || 0)

** USER ROOT EXITED at this point

El 18/11/16 a las 15:16, Miklos Szeredi escribió:
> On Wed, Nov 16, 2016 at 7:19 PM, Omar Walid Llorente <[email protected]> wrote:
>
>> In any case, the kernel we are using (3.13.0-96-generic #143-Ubuntu SMP Mon
>> Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux) has fuse included
>> (not as a module) so if it would be necessary to change the fuse mount
>> options some help would be needed.
> Can you reproduce this on latest (e.g 4.8) kernel?
>
> Also fuse debug logs (with whaterver kernel) could be useful; not sure
> how to enable it for glusterfs, usual convention for fuse filesystems
> is "-d".
>
> Thanks,
> Miklos

--
----------------------------------------------------------------
Centro de Cálculo Depto. Ingeniería Sistemas Telemáticos
E-mail: [email protected] Universidad Politécnica de Madrid
Fax:(+34) 913367333 E.T.S. Ing. Telecomunicación
Tel:(+34) 915495700-Ext.3005 28040 Madrid (Spain)
Tel:(+34) 915495762-Ext.3005
Tel:(+34) 913367366-Ext.3005
----------------------------------------------------------------

2016-11-21 12:56:22

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

Hi,

Sorry for the delay. I could reproduce the issue on my gluster setup as
well. The error is generated at posix-acl xlator layer of glusterfs server.

As Bruce mentioned in the earlier mail, as part of WRITE call,
NFS-server issues OPEN() with O_WRONLY flag set. Since the attributes of
the file "kk.txt" when created was 0400, the corresponding acl maps to
below ACL -

[root@dhcp35-197 vol1]# getfacl kk.txt
# file: kk.txt
# owner: guest
# group: mock
user::r--
group::---
other::---

And hence the permission check for WRITE access fails. Also not just
glusterfs, I see similar behavior with ext4 filesystem as well when
exported via kernel-NFS server. So I am not sure if fixing glusterfs for
this use-case is valid.

Thanks,
Soumya

On 11/18/2016 09:33 PM, Omar Walid Llorente wrote:
> Yes, of course, we can reproduce the problem on latest (e.g 4.8) kernel.
>
> Tested on a server with fedora 24 (Linux cuentas11.lab.dit.upm.es
> 4.8.4-200.fc24.x86_64 #1 SMP Tue Oct 25 13:06:04 UTC 2016 x86_64 x86_64
> x86_64 GNU/Linux), glusterfs-fuse-3.8.5-1.fc24.x86_64,
> fuse-2.9.7-1.fc24.x86_64 and nfs-utils-1.3.4-1.rc2.fc24.x86_64 that
> mounts as a glusterfs-client and exports as a nfs-server a ditributed
> glusterfs volume (that comes from a 3.7.4 glusterfs server over Ubuntu
> 14.04.
>
> [root@cuentas11 ~]# cat /proc/mounts | grep home
> recipiente8hd:/home-lab-3.tcp /home-3 fuse.glusterfs
> rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072
> 0 0
> [root@cuentas11 ~]#
>
> As you may remember the schema is the one that follows (glusterfs volume
> name: home-lab-3, glusterfs client mount point: /home-3, nfs client
> mount point: /home):
>
> nfs client (v1.2.8) ---> NFSv3 ---> knfsd (v1.3.4 over a Linux v4.8.4
> kernel) ---> FUSE (v2.9.7) --> glusterfs client (v3.8.5) --> GLUSTER
> over TCP --> glusterfs server (v3.7.4)
>
> The errors at the nfs-client side are the same:
>
> cdc@l056:~$ mount | grep "/home/cdc"
> cuentas11:/home-3/cdc on /home/cdc type nfs
> (rw,noatime,nfsvers=3,vers=3,tcp,intr,fsc,nolock,rsize=262140,wsize=262140,addr=138.4.30.86)
>
> cdc@l056:~$
> cdc@l056:~$
> cdc@l056:~$
> cdc@l056:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444
> 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
> cp: failed to close ‘kk.txt’: Permission denied
> -r--r--r-- 1 cdc admincdc 7 nov 18 2016 444.txt
> -r--r--r-- 1 cdc admincdc 0 nov 18 2016 kk.txt
> cdc@l056:~$
> cdc@l056:~$
> cdc@l056:~$
> cdc@l056:~$ sudo su
> [sudo] password for cdc:
> root@l056:/home/cdc# rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
> chmod 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
> -r--r--r-- 1 root root 7 nov 18 2016 444.txt
> -r--r--r-- 1 root root 7 nov 18 2016 kk.txt
> root@l056:/home/cdc#
> root@l056:/home/cdc#
> root@l056:/home/cdc# exit
> cdc@l056:~$
>
> See at the end the logs with and without the debug option activated at
> the glusterfs client.
>
> Thanks again, and sorry for the long message.
>
> Omar
>
>
>
> * Logs on the glusterfs client side with NO DEBUG:
>
> [2016-11-18 15:45:43.681215] E [MSGID: 114031]
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-3: remote
> operation failed. Path: /cdc/kk.txt
> (cc98e831-3410-4202-8254-2d4b48e8ea68) [Permission denied]
> [2016-11-18 15:45:43.681297] W [fuse-bridge.c:989:fuse_fd_cbk]
> 0-glusterfs-fuse: 399: OPEN() /cdc/kk.txt => -1 (Permission denied)
> [2016-11-18 15:45:55.457922] I [io-stats.c:1574:io_stats_dump_fd]
> 0-home-lab-3: --- fd stats ---
> [2016-11-18 15:45:55.457978] I [io-stats.c:1579:io_stats_dump_fd]
> 0-home-lab-3: Filename : /cdc/444.txt
> [2016-11-18 15:45:55.457997] I [io-stats.c:1594:io_stats_dump_fd]
> 0-home-lab-3: BytesWritten : 7 bytes
> [2016-11-18 15:45:55.458011] I [io-stats.c:1606:io_stats_dump_fd]
> 0-home-lab-3: Write 000004b+ : 1
> [2016-11-18 15:45:55.471933] I [io-stats.c:1574:io_stats_dump_fd]
> 0-home-lab-3: --- fd stats ---
>
>
> * Logs on the glusterfs client side with DEBUG activated:
>
> [2016-11-18 16:06:17.040092] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
> as fast fop
> [2016-11-18 16:06:17.045986] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> scheduled as fast fop
> [2016-11-18 16:06:17.046132] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
> 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
> home-lab-3-client-3
> [2016-11-18 16:06:17.046833] D [MSGID: 0]
> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> returned for /cdc/kk.txt with op_ret 0 [Invalid argument]
> [2016-11-18 16:06:17.047671] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK
> scheduled as normal fop
> [2016-11-18 16:06:17.050539] D [fuse-bridge.c:5264:notify] 0-fuse: got
> event 19 on graph -1418558531
> [2016-11-18 16:06:17.050867] D [fuse-bridge.c:5264:notify] 0-fuse: got
> event 19 on graph 0
> [2016-11-18 16:06:17.051948] D [logging.c:1953:_gf_msg_internal]
> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> About to flush least recently used log message to disk
> [2016-11-18 16:06:17.051871] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> scheduled as fast fop
> [2016-11-18 16:06:17.051946] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
> 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
> home-lab-3-client-0
> [2016-11-18 16:06:17.062108] D [MSGID: 0]
> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> returned for /cdc/444.txt with op_ret 0 [Invalid argument]
> [2016-11-18 16:06:17.063692] D [fuse-bridge.c:5264:notify] 0-fuse: got
> event 19 on graph -1418558531
> [2016-11-18 16:06:17.064015] D [fuse-bridge.c:5264:notify] 0-fuse: got
> event 19 on graph 0
> [2016-11-18 16:06:17.065068] D [logging.c:1953:_gf_msg_internal]
> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> About to flush least recently used log message to disk
> The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
> 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 4 times
> between [2016-11-18 16:06:17.040092] and [2016-11-18 16:06:17.064175]
> [2016-11-18 16:06:17.065066] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> scheduled as fast fop
> [2016-11-18 16:06:17.065232] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
> 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
> home-lab-3-client-0
> [2016-11-18 16:06:17.065755] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:06:17.065897] D [MSGID: 0]
> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> returned for /cdc/444.txt with op_ret -1 [No such file or directory]
> [2016-11-18 16:06:17.066016] D [MSGID: 0]
> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
> missing on subvol home-lab-3-client-0
> [2016-11-18 16:06:17.066104] D [MSGID: 0]
> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
> lookup call to 4 subvols
> [2016-11-18 16:06:17.066594] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:06:17.066669] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> home-lab-3-client-0
> [2016-11-18 16:06:17.066771] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:06:17.066898] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> home-lab-3-client-1
> [2016-11-18 16:06:17.066906] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:06:17.067019] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> home-lab-3-client-2
> [2016-11-18 16:06:17.066915] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:06:17.067066] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> home-lab-3-client-3
> [2016-11-18 16:06:17.067107] D [MSGID: 0]
> [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
> hashed_subvol home-lab-3-client-0 cached_subvol null
> [2016-11-18 16:06:17.067151] D [MSGID: 0]
> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
> was no cached file and unlink on hashed is not skipped /cdc/444.txt
> [2016-11-18 16:06:17.067179] D [MSGID: 0]
> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
> stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1 error: No such
> file or directory [No such file or directory]
> [2016-11-18 16:06:17.067206] D [MSGID: 0]
> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:06:17.067250] D [MSGID: 0]
> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:06:17.067275] D [MSGID: 0]
> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:06:17.067308] D [MSGID: 0]
> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:06:17.067387] D [MSGID: 0]
> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file or directory
> [No such file or directory]
> [2016-11-18 16:06:17.068157] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> scheduled as fast fop
> [2016-11-18 16:06:17.068281] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
> 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
> home-lab-3-client-0
> [2016-11-18 16:06:17.068738] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:06:17.068781] D [MSGID: 0]
> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> returned for /cdc/444.txt with op_ret -1 [No such file or directory]
> [2016-11-18 16:06:17.068805] D [MSGID: 0]
> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
> missing on subvol home-lab-3-client-0
> [2016-11-18 16:06:17.068827] D [MSGID: 0]
> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
> lookup call to 4 subvols
> [2016-11-18 16:06:17.069286] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:06:17.069345] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> home-lab-3-client-0
> [2016-11-18 16:06:17.069452] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:06:17.069507] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> home-lab-3-client-2
> [2016-11-18 16:06:17.069556] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:06:17.069585] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> home-lab-3-client-3
> [2016-11-18 16:06:17.069939] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:06:17.069983] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> home-lab-3-client-1
> [2016-11-18 16:06:17.070005] D [MSGID: 0]
> [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
> hashed_subvol home-lab-3-client-0 cached_subvol null
> [2016-11-18 16:06:17.070038] D [MSGID: 0]
> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
> was no cached file and unlink on hashed is not skipped /cdc/444.txt
> [2016-11-18 16:06:17.070098] D [MSGID: 0]
> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
> stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1 error: No such
> file or directory [No such file or directory]
> [2016-11-18 16:06:17.070128] D [MSGID: 0]
> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:06:17.070154] D [MSGID: 0]
> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:06:17.070178] D [MSGID: 0]
> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:06:17.070208] D [MSGID: 0]
> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:06:17.070251] D [MSGID: 0]
> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file or directory
> [No such file or directory]
> [2016-11-18 16:06:17.070537] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled
> as normal fop
> [2016-11-18 16:06:17.070826] D [MSGID: 0]
> [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
> creating /cdc/444.txt on home-lab-3-client-0
> [2016-11-18 16:06:17.071100] D [MSGID: 0]
> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
> 'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
> 10456189239296 and avail_inodes is: 99.00
> [2016-11-18 16:06:17.071142] D [MSGID: 0]
> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
> 'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
> 10499693871104 and avail_inodes is: 99.00
> [2016-11-18 16:06:17.071183] D [MSGID: 0]
> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
> 'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
> 10500911267840 and avail_inodes is: 99.00
> [2016-11-18 16:06:17.071343] D [MSGID: 0]
> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
> 'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
> 8600432410624 and avail_inodes is: 99.00
> [2016-11-18 16:06:17.073943] D [fuse-bridge.c:5264:notify] 0-fuse: got
> event 19 on graph -1418558531
> [2016-11-18 16:06:17.074273] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
> as fast fop
> [2016-11-18 16:06:17.075035] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled
> as fast fop
> [2016-11-18 16:06:17.075913] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled
> as slow fop
> [2016-11-18 16:06:17.076205] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled
> as fast fop
> [2016-11-18 16:06:17.076636] D [fuse-bridge.c:5264:notify] 0-fuse: got
> event 19 on graph 0
> [2016-11-18 16:06:17.077282] D [fuse-bridge.c:52:fuse_invalidate]
> 0-fuse: Invalidate inode id 140363155120276.
> [2016-11-18 16:06:17.077981] I [io-stats.c:1574:io_stats_dump_fd]
> 0-home-lab-3: --- fd stats ---
> [2016-11-18 16:06:17.078032] I [io-stats.c:1579:io_stats_dump_fd]
> 0-home-lab-3: Filename : /cdc/444.txt
> [2016-11-18 16:06:17.078049] I [io-stats.c:1594:io_stats_dump_fd]
> 0-home-lab-3: BytesWritten : 7 bytes
> [2016-11-18 16:06:17.078064] I [io-stats.c:1606:io_stats_dump_fd]
> 0-home-lab-3: Write 000004b+ : 1
> [2016-11-18 16:06:17.079156] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
> scheduled as normal fop
> [2016-11-18 16:06:17.079794] D [fuse-bridge.c:5264:notify] 0-fuse: got
> event 19 on graph 0
> [2016-11-18 16:06:17.082196] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> scheduled as fast fop
> [2016-11-18 16:06:17.082772] D [logging.c:1953:_gf_msg_internal]
> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> About to flush least recently used log message to disk
> [2016-11-18 16:06:17.082318] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
> 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
> home-lab-3-client-3
> [2016-11-18 16:06:17.082770] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:06:17.083168] D [MSGID: 0]
> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
> [2016-11-18 16:06:17.083282] D [MSGID: 0]
> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
> missing on subvol home-lab-3-client-3
> [2016-11-18 16:06:17.083320] D [MSGID: 0]
> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
> lookup call to 4 subvols
> [2016-11-18 16:06:17.083991] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:06:17.084043] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
> [2016-11-18 16:06:17.084048] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:06:17.084092] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
> [2016-11-18 16:06:17.084114] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:06:17.084167] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
> [2016-11-18 16:06:17.084091] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:06:17.084377] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
> [2016-11-18 16:06:17.084406] D [MSGID: 0]
> [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
> hashed_subvol home-lab-3-client-3 cached_subvol null
> [2016-11-18 16:06:17.084511] D [MSGID: 0]
> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
> was no cached file and unlink on hashed is not skipped /cdc/kk.txt
> [2016-11-18 16:06:17.084534] D [MSGID: 0]
> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
> file or directory [No such file or directory]
> [2016-11-18 16:06:17.084683] D [MSGID: 0]
> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:06:17.084783] D [MSGID: 0]
> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:06:17.084857] D [MSGID: 0]
> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:06:17.084889] D [MSGID: 0]
> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:06:17.085001] D [MSGID: 0]
> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
> [No such file or directory]
> [2016-11-18 16:06:17.086098] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
> as fast fop
> [2016-11-18 16:06:17.087256] D [logging.c:1953:_gf_msg_internal]
> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> About to flush least recently used log message to disk
> [2016-11-18 16:06:17.087173] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> scheduled as fast fop
> [2016-11-18 16:06:17.087254] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
> 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
> home-lab-3-client-3
> [2016-11-18 16:06:17.087749] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:06:17.087795] D [MSGID: 0]
> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
> [2016-11-18 16:06:17.088014] D [MSGID: 0]
> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
> missing on subvol home-lab-3-client-3
> [2016-11-18 16:06:17.088036] D [MSGID: 0]
> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
> lookup call to 4 subvols
> [2016-11-18 16:06:17.088598] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:06:17.088639] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
> [2016-11-18 16:06:17.088804] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:06:17.088855] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
> [2016-11-18 16:06:17.088819] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:06:17.089069] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
> [2016-11-18 16:06:17.089380] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:06:17.089421] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
> [2016-11-18 16:06:17.089443] D [MSGID: 0]
> [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
> hashed_subvol home-lab-3-client-3 cached_subvol null
> [2016-11-18 16:06:17.089463] D [MSGID: 0]
> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
> was no cached file and unlink on hashed is not skipped /cdc/kk.txt
> [2016-11-18 16:06:17.089575] D [MSGID: 0]
> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
> file or directory [No such file or directory]
> [2016-11-18 16:06:17.089601] D [MSGID: 0]
> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:06:17.089712] D [MSGID: 0]
> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:06:17.089738] D [MSGID: 0]
> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:06:17.090006] D [MSGID: 0]
> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:06:17.090156] D [MSGID: 0]
> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
> [No such file or directory]
> [2016-11-18 16:06:17.090648] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled
> as normal fop
> [2016-11-18 16:06:17.090732] D [MSGID: 0]
> [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
> creating /cdc/kk.txt on home-lab-3-client-3
> [2016-11-18 16:06:17.091499] D [fuse-bridge.c:5264:notify] 0-fuse: got
> event 19 on graph -1418558531
> [2016-11-18 16:06:17.091843] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
> scheduled as normal fop
> [2016-11-18 16:06:17.093831] D [fuse-bridge.c:5264:notify] 0-fuse: got
> event 19 on graph 0
> [2016-11-18 16:06:17.094711] D [logging.c:1953:_gf_msg_internal]
> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> About to flush least recently used log message to disk
> [2016-11-18 16:06:17.093216] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
> scheduled as normal fop
> [2016-11-18 16:06:17.094709] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled
> as fast fop
> [2016-11-18 16:06:17.095280] E [MSGID: 114031]
> [client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-3: remote
> operation failed. Path: /cdc/kk.txt
> (09bfb6fa-82fb-463b-abd9-bb7ef88e10c0) [Permission denied]
> [2016-11-18 16:06:17.095341] D [MSGID: 0]
> [client-rpc-fops.c:448:client3_3_open_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: Permission denied
> [Permission denied]
> [2016-11-18 16:06:17.095366] D [MSGID: 0]
> [dht-inode-read.c:45:dht_open_cbk] 0-home-lab-3-dht: subvolume
> home-lab-3-client-3 returned -1 [Permission denied]
> [2016-11-18 16:06:17.095480] D [MSGID: 0]
> [dht-inode-read.c:59:dht_open_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: Permission denied
> [Permission denied]
> [2016-11-18 16:06:17.095671] D [MSGID: 0] [read-ahead.c:101:ra_open_cbk]
> 0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-read-ahead
> returned -1 error: Permission denied [Permission denied]
> [2016-11-18 16:06:17.095759] D [MSGID: 0] [io-cache.c:605:ioc_open_cbk]
> 0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-io-cache
> returned -1 error: Permission denied [Permission denied]
> [2016-11-18 16:06:17.095963] D [MSGID: 0]
> [defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: Permission
> denied [Permission denied]
> [2016-11-18 16:06:17.095988] D [MSGID: 0]
> [defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: Permission
> denied [Permission denied]
> [2016-11-18 16:06:17.096275] D [MSGID: 0]
> [io-stats.c:1936:io_stats_open_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3 returned -1 error: Permission denied
> [Permission denied]
> [2016-11-18 16:06:17.096362] W [fuse-bridge.c:989:fuse_fd_cbk]
> 0-glusterfs-fuse: 338: OPEN() /cdc/kk.txt => -1 (Permission denied)
> [2016-11-18 16:06:17.097104] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
> scheduled as normal fop
> [2016-11-18 16:06:17.097674] D [fuse-bridge.c:5264:notify] 0-fuse: got
> event 19 on graph 0
> [2016-11-18 16:06:17.099912] D [fuse-bridge.c:5264:notify] 0-fuse: got
> event 19 on graph 0
> [2016-11-18 16:06:24.785225] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
> 0-home-lab-3-client-0: returning as transport is already disconnected OR
> there are no frames (0 || 0)
> [2016-11-18 16:06:24.785338] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
> 0-home-lab-3-client-1: returning as transport is already disconnected OR
> there are no frames (0 || 0)
> [2016-11-18 16:06:24.785359] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
> 0-home-lab-3-client-2: returning as transport is already disconnected OR
> there are no frames (0 || 0)
> [2016-11-18 16:06:24.785377] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
> 0-home-lab-3-client-3: returning as transport is already disconnected OR
> there are no frames (0 || 0)
>
>
>
>
>
> ** TEST DONE as user cdc --> ERROR
>
>
>
>
>
>
> [2016-11-18 16:06:46.737686] D
> [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
> (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
> ))))) 0-: 138.4.2.246:49172: ping timer event already removed
> [2016-11-18 16:06:49.545698] D
> [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
> (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
> ))))) 0-: 138.4.2.177:49157: ping timer event already removed
> [2016-11-18 16:06:49.546474] D
> [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
> (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
> ))))) 0-: 138.4.2.189:49157: ping timer event already removed
> [2016-11-18 16:06:49.547093] D
> [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
> (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
> ))))) 0-: 138.4.2.248:49157: ping timer event already removed
>
>
>
>
>
> ** user CDC becomed root at this point
>
>
>
>
> [2016-11-18 16:07:11.907892] D [logging.c:1953:_gf_msg_internal]
> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> About to flush least recently used log message to disk
> The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
> 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 7 times
> between [2016-11-18 16:06:17.086098] and [2016-11-18 16:07:11.906944]
> [2016-11-18 16:07:11.907884] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> scheduled as fast fop
> [2016-11-18 16:07:11.908875] D [fuse-bridge.c:5264:notify] 0-fuse: got
> event 19 on graph -1418558531
> [2016-11-18 16:07:11.909179] D [fuse-bridge.c:5264:notify] 0-fuse: got
> event 19 on graph 0
> [2016-11-18 16:07:11.909256] D [logging.c:1953:_gf_msg_internal]
> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> About to flush least recently used log message to disk
> The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
> 0-home-lab-3-io-threads: UNLINK scheduled as normal fop" repeated 2
> times between [2016-11-18 16:06:17.047671] and [2016-11-18 16:07:11.908114]
> [2016-11-18 16:07:11.909254] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
> as fast fop
> [2016-11-18 16:07:11.911109] D [logging.c:1953:_gf_msg_internal]
> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> About to flush least recently used log message to disk
> [2016-11-18 16:07:11.911018] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> scheduled as fast fop
> [2016-11-18 16:07:11.911107] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK
> scheduled as normal fop
> [2016-11-18 16:07:11.911871] D [fuse-bridge.c:5264:notify] 0-fuse: got
> event 19 on graph -1418558531
> [2016-11-18 16:07:11.912045] D [fuse-bridge.c:5264:notify] 0-fuse: got
> event 19 on graph 0
> [2016-11-18 16:07:11.913067] D [logging.c:1953:_gf_msg_internal]
> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> About to flush least recently used log message to disk
> The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
> 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 2 times
> between [2016-11-18 16:07:11.909254] and [2016-11-18 16:07:11.912165]
> [2016-11-18 16:07:11.913065] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> scheduled as fast fop
> [2016-11-18 16:07:11.913193] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
> 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
> home-lab-3-client-0
> [2016-11-18 16:07:11.913710] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:07:11.913757] D [MSGID: 0]
> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> returned for /cdc/444.txt with op_ret -1 [No such file or directory]
> [2016-11-18 16:07:11.913788] D [MSGID: 0]
> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
> missing on subvol home-lab-3-client-0
> [2016-11-18 16:07:11.913810] D [MSGID: 0]
> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
> lookup call to 4 subvols
> [2016-11-18 16:07:11.914336] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:07:11.914379] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> home-lab-3-client-0
> [2016-11-18 16:07:11.914424] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:07:11.914448] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> home-lab-3-client-1
> [2016-11-18 16:07:11.914501] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:07:11.914617] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> home-lab-3-client-3
> [2016-11-18 16:07:11.914649] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:07:11.914698] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> home-lab-3-client-2
> [2016-11-18 16:07:11.914724] D [MSGID: 0]
> [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
> hashed_subvol home-lab-3-client-0 cached_subvol null
> [2016-11-18 16:07:11.914743] D [MSGID: 0]
> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
> was no cached file and unlink on hashed is not skipped /cdc/444.txt
> [2016-11-18 16:07:11.914764] D [MSGID: 0]
> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
> file or directory [No such file or directory]
> [2016-11-18 16:07:11.914791] D [MSGID: 0]
> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:07:11.914825] D [MSGID: 0]
> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:07:11.914855] D [MSGID: 0]
> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:07:11.914880] D [MSGID: 0]
> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:07:11.914947] D [MSGID: 0]
> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
> [No such file or directory]
> [2016-11-18 16:07:11.915569] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> scheduled as fast fop
> [2016-11-18 16:07:11.915644] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
> 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
> home-lab-3-client-0
> [2016-11-18 16:07:11.916090] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:07:11.916134] D [MSGID: 0]
> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> returned for /cdc/444.txt with op_ret -1 [No such file or directory]
> [2016-11-18 16:07:11.916158] D [MSGID: 0]
> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
> missing on subvol home-lab-3-client-0
> [2016-11-18 16:07:11.916185] D [MSGID: 0]
> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
> lookup call to 4 subvols
> [2016-11-18 16:07:11.916593] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:07:11.916780] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> home-lab-3-client-0
> [2016-11-18 16:07:11.916878] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:07:11.916994] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> home-lab-3-client-3
> [2016-11-18 16:07:11.917076] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:07:11.916906] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:07:11.917117] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> home-lab-3-client-2
> [2016-11-18 16:07:11.917153] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> home-lab-3-client-1
> [2016-11-18 16:07:11.917205] D [MSGID: 0]
> [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
> hashed_subvol home-lab-3-client-0 cached_subvol null
> [2016-11-18 16:07:11.917368] D [MSGID: 0]
> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
> was no cached file and unlink on hashed is not skipped /cdc/444.txt
> [2016-11-18 16:07:11.917437] D [MSGID: 0]
> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
> file or directory [No such file or directory]
> [2016-11-18 16:07:11.917467] D [MSGID: 0]
> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:07:11.917496] D [MSGID: 0]
> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:07:11.917521] D [MSGID: 0]
> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:07:11.917551] D [MSGID: 0]
> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:07:11.917726] D [MSGID: 0]
> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
> [No such file or directory]
> [2016-11-18 16:07:11.917958] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled
> as normal fop
> [2016-11-18 16:07:11.918230] D [MSGID: 0]
> [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
> creating /cdc/444.txt on home-lab-3-client-0
> [2016-11-18 16:07:11.918488] D [MSGID: 0]
> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
> 'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
> 10456166432768 and avail_inodes is: 99.00
> [2016-11-18 16:07:11.918570] D [MSGID: 0]
> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
> 'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
> 8600417075200 and avail_inodes is: 99.00
> [2016-11-18 16:07:11.918603] D [MSGID: 0]
> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
> 'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
> 10500905500672 and avail_inodes is: 99.00
> [2016-11-18 16:07:11.918675] D [MSGID: 0]
> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
> 'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
> 10499686531072 and avail_inodes is: 99.00
> [2016-11-18 16:07:11.919095] D [fuse-bridge.c:5264:notify] 0-fuse: got
> event 19 on graph -1418558531
> [2016-11-18 16:07:11.919377] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
> as fast fop
> [2016-11-18 16:07:11.920040] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled
> as fast fop
> [2016-11-18 16:07:11.920889] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled
> as slow fop
> [2016-11-18 16:07:11.921051] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled
> as fast fop
> [2016-11-18 16:07:11.921583] D [fuse-bridge.c:5264:notify] 0-fuse: got
> event 19 on graph 0
> [2016-11-18 16:07:11.922068] D [fuse-bridge.c:52:fuse_invalidate]
> 0-fuse: Invalidate inode id 140363155120276.
> [2016-11-18 16:07:11.922936] I [io-stats.c:1574:io_stats_dump_fd]
> 0-home-lab-3: --- fd stats ---
> [2016-11-18 16:07:11.922988] I [io-stats.c:1579:io_stats_dump_fd]
> 0-home-lab-3: Filename : /cdc/444.txt
> [2016-11-18 16:07:11.923004] I [io-stats.c:1594:io_stats_dump_fd]
> 0-home-lab-3: BytesWritten : 7 bytes
> [2016-11-18 16:07:11.923026] I [io-stats.c:1606:io_stats_dump_fd]
> 0-home-lab-3: Write 000004b+ : 1
> [2016-11-18 16:07:11.923779] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
> as fast fop
> [2016-11-18 16:07:11.924504] D [fuse-bridge.c:5264:notify] 0-fuse: got
> event 19 on graph 0
> [2016-11-18 16:07:11.926692] D [logging.c:1953:_gf_msg_internal]
> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> About to flush least recently used log message to disk
> The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
> 0-home-lab-3-io-threads: SETATTR scheduled as normal fop" repeated 2
> times between [2016-11-18 16:06:17.097104] and [2016-11-18 16:07:11.923946]
> [2016-11-18 16:07:11.926690] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> scheduled as fast fop
> [2016-11-18 16:07:11.926858] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
> 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
> home-lab-3-client-3
> [2016-11-18 16:07:11.927327] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:07:11.927376] D [MSGID: 0]
> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
> [2016-11-18 16:07:11.927401] D [MSGID: 0]
> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
> missing on subvol home-lab-3-client-3
> [2016-11-18 16:07:11.927422] D [MSGID: 0]
> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
> lookup call to 4 subvols
> [2016-11-18 16:07:11.927940] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:07:11.927987] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
> [2016-11-18 16:07:11.928043] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:07:11.928055] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:07:11.928058] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:07:11.928170] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
> [2016-11-18 16:07:11.928174] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
> [2016-11-18 16:07:11.928277] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
> [2016-11-18 16:07:11.928360] D [MSGID: 0]
> [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
> hashed_subvol home-lab-3-client-3 cached_subvol null
> [2016-11-18 16:07:11.928381] D [MSGID: 0]
> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
> was no cached file and unlink on hashed is not skipped /cdc/kk.txt
> [2016-11-18 16:07:11.928403] D [MSGID: 0]
> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
> file or directory [No such file or directory]
> [2016-11-18 16:07:11.928431] D [MSGID: 0]
> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:07:11.928456] D [MSGID: 0]
> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:07:11.928480] D [MSGID: 0]
> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:07:11.928511] D [MSGID: 0]
> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:07:11.928753] D [MSGID: 0]
> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
> [No such file or directory]
> [2016-11-18 16:07:11.930485] D [logging.c:1953:_gf_msg_internal]
> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> About to flush least recently used log message to disk
> [2016-11-18 16:07:11.929862] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
> as fast fop
> [2016-11-18 16:07:11.930483] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> scheduled as fast fop
> [2016-11-18 16:07:11.930602] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
> 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
> home-lab-3-client-3
> [2016-11-18 16:07:11.931071] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:07:11.931124] D [MSGID: 0]
> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
> [2016-11-18 16:07:11.931148] D [MSGID: 0]
> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
> missing on subvol home-lab-3-client-3
> [2016-11-18 16:07:11.931348] D [MSGID: 0]
> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
> lookup call to 4 subvols
> [2016-11-18 16:07:11.931880] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:07:11.931927] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
> [2016-11-18 16:07:11.932002] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:07:11.931994] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:07:11.931991] D [MSGID: 0]
> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
> such file or directory [No such file or directory]
> [2016-11-18 16:07:11.932078] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
> [2016-11-18 16:07:11.932116] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
> [2016-11-18 16:07:11.932315] D [MSGID: 0]
> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
> [2016-11-18 16:07:11.932363] D [MSGID: 0]
> [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
> hashed_subvol home-lab-3-client-3 cached_subvol null
> [2016-11-18 16:07:11.932383] D [MSGID: 0]
> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
> was no cached file and unlink on hashed is not skipped /cdc/kk.txt
> [2016-11-18 16:07:11.932407] D [MSGID: 0]
> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
> stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
> file or directory [No such file or directory]
> [2016-11-18 16:07:11.932446] D [MSGID: 0]
> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:07:11.932471] D [MSGID: 0]
> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:07:11.932494] D [MSGID: 0]
> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:07:11.932518] D [MSGID: 0]
> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
> directory [No such file or directory]
> [2016-11-18 16:07:11.932725] D [logging.c:1953:_gf_msg_internal]
> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> About to flush least recently used log message to disk
> [2016-11-18 16:07:11.932545] D [MSGID: 0]
> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
> 0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
> [No such file or directory]
> [2016-11-18 16:07:11.932722] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled
> as normal fop
> [2016-11-18 16:07:11.933008] D [MSGID: 0]
> [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
> creating /cdc/kk.txt on home-lab-3-client-3
> [2016-11-18 16:07:11.934086] D [fuse-bridge.c:5264:notify] 0-fuse: got
> event 19 on graph -1418558531
> [2016-11-18 16:07:11.934323] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
> scheduled as normal fop
> [2016-11-18 16:07:11.934922] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
> as fast fop
> [2016-11-18 16:07:11.935497] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
> scheduled as normal fop
> [2016-11-18 16:07:11.936102] D [fuse-bridge.c:5264:notify] 0-fuse: got
> event 19 on graph 0
> [2016-11-18 16:07:11.936903] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled
> as fast fop
> [2016-11-18 16:07:11.937540] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled
> as slow fop
> [2016-11-18 16:07:11.937682] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled
> as fast fop
> [2016-11-18 16:07:11.938267] D [fuse-bridge.c:5264:notify] 0-fuse: got
> event 19 on graph 0
> [2016-11-18 16:07:11.938757] D [fuse-bridge.c:52:fuse_invalidate]
> 0-fuse: Invalidate inode id 140363155120652.
> [2016-11-18 16:07:11.938994] I [io-stats.c:1574:io_stats_dump_fd]
> 0-home-lab-3: --- fd stats ---
> [2016-11-18 16:07:11.939072] I [io-stats.c:1579:io_stats_dump_fd]
> 0-home-lab-3: Filename : /cdc/kk.txt
> [2016-11-18 16:07:11.939096] I [io-stats.c:1594:io_stats_dump_fd]
> 0-home-lab-3: BytesWritten : 7 bytes
> [2016-11-18 16:07:11.939112] I [io-stats.c:1606:io_stats_dump_fd]
> 0-home-lab-3: Write 000004b+ : 1
> [2016-11-18 16:07:11.939448] D [MSGID: 0]
> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
> scheduled as normal fop
> [2016-11-18 16:07:11.940028] D [fuse-bridge.c:5264:notify] 0-fuse: got
> event 19 on graph 0
> [2016-11-18 16:07:11.942130] D [fuse-bridge.c:5264:notify] 0-fuse: got
> event 19 on graph 0
>
>
>
>
>
> ** TEST DONE as root --> NO PROBLEM
>
>
>
>
>
> [2016-11-18 16:07:28.794150] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
> 0-home-lab-3-client-2: returning as transport is already disconnected OR
> there are no frames (0 || 0)
> [2016-11-18 16:07:31.794656] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
> 0-home-lab-3-client-0: returning as transport is already disconnected OR
> there are no frames (0 || 0)
> [2016-11-18 16:07:31.794925] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
> 0-home-lab-3-client-1: returning as transport is already disconnected OR
> there are no frames (0 || 0)
> [2016-11-18 16:07:31.794984] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
> 0-home-lab-3-client-3: returning as transport is already disconnected OR
> there are no frames (0 || 0)
>
>
> ** USER ROOT EXITED at this point
>
> El 18/11/16 a las 15:16, Miklos Szeredi escribió:
>> On Wed, Nov 16, 2016 at 7:19 PM, Omar Walid Llorente <[email protected]>
>> wrote:
>>
>>> In any case, the kernel we are using (3.13.0-96-generic #143-Ubuntu
>>> SMP Mon
>>> Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux) has fuse
>>> included
>>> (not as a module) so if it would be necessary to change the fuse mount
>>> options some help would be needed.
>> Can you reproduce this on latest (e.g 4.8) kernel?
>>
>> Also fuse debug logs (with whaterver kernel) could be useful; not sure
>> how to enable it for glusterfs, usual convention for fuse filesystems
>> is "-d".
>>
>> Thanks,
>> Miklos
>

2016-11-21 14:57:46

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

On Mon, Nov 21, 2016 at 06:26:16PM +0530, Soumya Koduri wrote:
> Hi,
>
> Sorry for the delay. I could reproduce the issue on my gluster setup
> as well. The error is generated at posix-acl xlator layer of
> glusterfs server.
>
> As Bruce mentioned in the earlier mail, as part of WRITE call,
> NFS-server issues OPEN() with O_WRONLY flag set. Since the
> attributes of the file "kk.txt" when created was 0400, the
> corresponding acl maps to below ACL -
>
> [root@dhcp35-197 vol1]# getfacl kk.txt
> # file: kk.txt
> # owner: guest
> # group: mock
> user::r--
> group::---
> other::---
>
> And hence the permission check for WRITE access fails. Also not just
> glusterfs, I see similar behavior with ext4 filesystem as well when
> exported via kernel-NFS server.

That definitely shouldn't be happening, how are you reproducing?

--b.

> So I am not sure if fixing glusterfs
> for this use-case is valid.
>
> Thanks,
> Soumya
>
>
> On 11/18/2016 09:33 PM, Omar Walid Llorente wrote:
> >Yes, of course, we can reproduce the problem on latest (e.g 4.8) kernel.
> >
> >Tested on a server with fedora 24 (Linux cuentas11.lab.dit.upm.es
> >4.8.4-200.fc24.x86_64 #1 SMP Tue Oct 25 13:06:04 UTC 2016 x86_64 x86_64
> >x86_64 GNU/Linux), glusterfs-fuse-3.8.5-1.fc24.x86_64,
> >fuse-2.9.7-1.fc24.x86_64 and nfs-utils-1.3.4-1.rc2.fc24.x86_64 that
> >mounts as a glusterfs-client and exports as a nfs-server a ditributed
> >glusterfs volume (that comes from a 3.7.4 glusterfs server over Ubuntu
> >14.04.
> >
> >[root@cuentas11 ~]# cat /proc/mounts | grep home
> >recipiente8hd:/home-lab-3.tcp /home-3 fuse.glusterfs
> >rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072
> >0 0
> >[root@cuentas11 ~]#
> >
> >As you may remember the schema is the one that follows (glusterfs volume
> >name: home-lab-3, glusterfs client mount point: /home-3, nfs client
> >mount point: /home):
> >
> >nfs client (v1.2.8) ---> NFSv3 ---> knfsd (v1.3.4 over a Linux v4.8.4
> >kernel) ---> FUSE (v2.9.7) --> glusterfs client (v3.8.5) --> GLUSTER
> >over TCP --> glusterfs server (v3.7.4)
> >
> >The errors at the nfs-client side are the same:
> >
> >cdc@l056:~$ mount | grep "/home/cdc"
> >cuentas11:/home-3/cdc on /home/cdc type nfs
> >(rw,noatime,nfsvers=3,vers=3,tcp,intr,fsc,nolock,rsize=262140,wsize=262140,addr=138.4.30.86)
> >
> >cdc@l056:~$
> >cdc@l056:~$
> >cdc@l056:~$
> >cdc@l056:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444
> >444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
> >cp: failed to close ‘kk.txt’: Permission denied
> >-r--r--r-- 1 cdc admincdc 7 nov 18 2016 444.txt
> >-r--r--r-- 1 cdc admincdc 0 nov 18 2016 kk.txt
> >cdc@l056:~$
> >cdc@l056:~$
> >cdc@l056:~$
> >cdc@l056:~$ sudo su
> >[sudo] password for cdc:
> >root@l056:/home/cdc# rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
> >chmod 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
> >-r--r--r-- 1 root root 7 nov 18 2016 444.txt
> >-r--r--r-- 1 root root 7 nov 18 2016 kk.txt
> >root@l056:/home/cdc#
> >root@l056:/home/cdc#
> >root@l056:/home/cdc# exit
> >cdc@l056:~$
> >
> >See at the end the logs with and without the debug option activated at
> >the glusterfs client.
> >
> >Thanks again, and sorry for the long message.
> >
> >Omar
> >
> >
> >
> >* Logs on the glusterfs client side with NO DEBUG:
> >
> >[2016-11-18 15:45:43.681215] E [MSGID: 114031]
> >[client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-3: remote
> >operation failed. Path: /cdc/kk.txt
> >(cc98e831-3410-4202-8254-2d4b48e8ea68) [Permission denied]
> >[2016-11-18 15:45:43.681297] W [fuse-bridge.c:989:fuse_fd_cbk]
> >0-glusterfs-fuse: 399: OPEN() /cdc/kk.txt => -1 (Permission denied)
> >[2016-11-18 15:45:55.457922] I [io-stats.c:1574:io_stats_dump_fd]
> >0-home-lab-3: --- fd stats ---
> >[2016-11-18 15:45:55.457978] I [io-stats.c:1579:io_stats_dump_fd]
> >0-home-lab-3: Filename : /cdc/444.txt
> >[2016-11-18 15:45:55.457997] I [io-stats.c:1594:io_stats_dump_fd]
> >0-home-lab-3: BytesWritten : 7 bytes
> >[2016-11-18 15:45:55.458011] I [io-stats.c:1606:io_stats_dump_fd]
> >0-home-lab-3: Write 000004b+ : 1
> >[2016-11-18 15:45:55.471933] I [io-stats.c:1574:io_stats_dump_fd]
> >0-home-lab-3: --- fd stats ---
> >
> >
> >* Logs on the glusterfs client side with DEBUG activated:
> >
> >[2016-11-18 16:06:17.040092] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
> >as fast fop
> >[2016-11-18 16:06:17.045986] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> >scheduled as fast fop
> >[2016-11-18 16:06:17.046132] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
> >0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
> >home-lab-3-client-3
> >[2016-11-18 16:06:17.046833] D [MSGID: 0]
> >[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> >returned for /cdc/kk.txt with op_ret 0 [Invalid argument]
> >[2016-11-18 16:06:17.047671] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK
> >scheduled as normal fop
> >[2016-11-18 16:06:17.050539] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >event 19 on graph -1418558531
> >[2016-11-18 16:06:17.050867] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >event 19 on graph 0
> >[2016-11-18 16:06:17.051948] D [logging.c:1953:_gf_msg_internal]
> >0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> >About to flush least recently used log message to disk
> >[2016-11-18 16:06:17.051871] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> >scheduled as fast fop
> >[2016-11-18 16:06:17.051946] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
> >0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
> >home-lab-3-client-0
> >[2016-11-18 16:06:17.062108] D [MSGID: 0]
> >[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> >returned for /cdc/444.txt with op_ret 0 [Invalid argument]
> >[2016-11-18 16:06:17.063692] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >event 19 on graph -1418558531
> >[2016-11-18 16:06:17.064015] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >event 19 on graph 0
> >[2016-11-18 16:06:17.065068] D [logging.c:1953:_gf_msg_internal]
> >0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> >About to flush least recently used log message to disk
> >The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
> >0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 4 times
> >between [2016-11-18 16:06:17.040092] and [2016-11-18 16:06:17.064175]
> >[2016-11-18 16:06:17.065066] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> >scheduled as fast fop
> >[2016-11-18 16:06:17.065232] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
> >0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
> >home-lab-3-client-0
> >[2016-11-18 16:06:17.065755] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:06:17.065897] D [MSGID: 0]
> >[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> >returned for /cdc/444.txt with op_ret -1 [No such file or directory]
> >[2016-11-18 16:06:17.066016] D [MSGID: 0]
> >[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
> >missing on subvol home-lab-3-client-0
> >[2016-11-18 16:06:17.066104] D [MSGID: 0]
> >[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
> >lookup call to 4 subvols
> >[2016-11-18 16:06:17.066594] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:06:17.066669] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >home-lab-3-client-0
> >[2016-11-18 16:06:17.066771] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:06:17.066898] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >home-lab-3-client-1
> >[2016-11-18 16:06:17.066906] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:06:17.067019] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >home-lab-3-client-2
> >[2016-11-18 16:06:17.066915] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:06:17.067066] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >home-lab-3-client-3
> >[2016-11-18 16:06:17.067107] D [MSGID: 0]
> >[dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
> >hashed_subvol home-lab-3-client-0 cached_subvol null
> >[2016-11-18 16:06:17.067151] D [MSGID: 0]
> >[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
> >was no cached file and unlink on hashed is not skipped /cdc/444.txt
> >[2016-11-18 16:06:17.067179] D [MSGID: 0]
> >[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
> >stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1 error: No such
> >file or directory [No such file or directory]
> >[2016-11-18 16:06:17.067206] D [MSGID: 0]
> >[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:06:17.067250] D [MSGID: 0]
> >[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:06:17.067275] D [MSGID: 0]
> >[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:06:17.067308] D [MSGID: 0]
> >[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:06:17.067387] D [MSGID: 0]
> >[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file or directory
> >[No such file or directory]
> >[2016-11-18 16:06:17.068157] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> >scheduled as fast fop
> >[2016-11-18 16:06:17.068281] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
> >0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
> >home-lab-3-client-0
> >[2016-11-18 16:06:17.068738] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:06:17.068781] D [MSGID: 0]
> >[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> >returned for /cdc/444.txt with op_ret -1 [No such file or directory]
> >[2016-11-18 16:06:17.068805] D [MSGID: 0]
> >[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
> >missing on subvol home-lab-3-client-0
> >[2016-11-18 16:06:17.068827] D [MSGID: 0]
> >[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
> >lookup call to 4 subvols
> >[2016-11-18 16:06:17.069286] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:06:17.069345] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >home-lab-3-client-0
> >[2016-11-18 16:06:17.069452] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:06:17.069507] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >home-lab-3-client-2
> >[2016-11-18 16:06:17.069556] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:06:17.069585] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >home-lab-3-client-3
> >[2016-11-18 16:06:17.069939] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:06:17.069983] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >home-lab-3-client-1
> >[2016-11-18 16:06:17.070005] D [MSGID: 0]
> >[dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
> >hashed_subvol home-lab-3-client-0 cached_subvol null
> >[2016-11-18 16:06:17.070038] D [MSGID: 0]
> >[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
> >was no cached file and unlink on hashed is not skipped /cdc/444.txt
> >[2016-11-18 16:06:17.070098] D [MSGID: 0]
> >[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
> >stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1 error: No such
> >file or directory [No such file or directory]
> >[2016-11-18 16:06:17.070128] D [MSGID: 0]
> >[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:06:17.070154] D [MSGID: 0]
> >[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:06:17.070178] D [MSGID: 0]
> >[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:06:17.070208] D [MSGID: 0]
> >[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:06:17.070251] D [MSGID: 0]
> >[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file or directory
> >[No such file or directory]
> >[2016-11-18 16:06:17.070537] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled
> >as normal fop
> >[2016-11-18 16:06:17.070826] D [MSGID: 0]
> >[dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
> >creating /cdc/444.txt on home-lab-3-client-0
> >[2016-11-18 16:06:17.071100] D [MSGID: 0]
> >[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
> >'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
> >10456189239296 and avail_inodes is: 99.00
> >[2016-11-18 16:06:17.071142] D [MSGID: 0]
> >[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
> >'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
> >10499693871104 and avail_inodes is: 99.00
> >[2016-11-18 16:06:17.071183] D [MSGID: 0]
> >[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
> >'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
> >10500911267840 and avail_inodes is: 99.00
> >[2016-11-18 16:06:17.071343] D [MSGID: 0]
> >[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
> >'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
> >8600432410624 and avail_inodes is: 99.00
> >[2016-11-18 16:06:17.073943] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >event 19 on graph -1418558531
> >[2016-11-18 16:06:17.074273] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
> >as fast fop
> >[2016-11-18 16:06:17.075035] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled
> >as fast fop
> >[2016-11-18 16:06:17.075913] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled
> >as slow fop
> >[2016-11-18 16:06:17.076205] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled
> >as fast fop
> >[2016-11-18 16:06:17.076636] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >event 19 on graph 0
> >[2016-11-18 16:06:17.077282] D [fuse-bridge.c:52:fuse_invalidate]
> >0-fuse: Invalidate inode id 140363155120276.
> >[2016-11-18 16:06:17.077981] I [io-stats.c:1574:io_stats_dump_fd]
> >0-home-lab-3: --- fd stats ---
> >[2016-11-18 16:06:17.078032] I [io-stats.c:1579:io_stats_dump_fd]
> >0-home-lab-3: Filename : /cdc/444.txt
> >[2016-11-18 16:06:17.078049] I [io-stats.c:1594:io_stats_dump_fd]
> >0-home-lab-3: BytesWritten : 7 bytes
> >[2016-11-18 16:06:17.078064] I [io-stats.c:1606:io_stats_dump_fd]
> >0-home-lab-3: Write 000004b+ : 1
> >[2016-11-18 16:06:17.079156] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
> >scheduled as normal fop
> >[2016-11-18 16:06:17.079794] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >event 19 on graph 0
> >[2016-11-18 16:06:17.082196] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> >scheduled as fast fop
> >[2016-11-18 16:06:17.082772] D [logging.c:1953:_gf_msg_internal]
> >0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> >About to flush least recently used log message to disk
> >[2016-11-18 16:06:17.082318] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
> >0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
> >home-lab-3-client-3
> >[2016-11-18 16:06:17.082770] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:06:17.083168] D [MSGID: 0]
> >[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> >returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
> >[2016-11-18 16:06:17.083282] D [MSGID: 0]
> >[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
> >missing on subvol home-lab-3-client-3
> >[2016-11-18 16:06:17.083320] D [MSGID: 0]
> >[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
> >lookup call to 4 subvols
> >[2016-11-18 16:06:17.083991] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:06:17.084043] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
> >[2016-11-18 16:06:17.084048] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:06:17.084092] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
> >[2016-11-18 16:06:17.084114] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:06:17.084167] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
> >[2016-11-18 16:06:17.084091] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:06:17.084377] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
> >[2016-11-18 16:06:17.084406] D [MSGID: 0]
> >[dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
> >hashed_subvol home-lab-3-client-3 cached_subvol null
> >[2016-11-18 16:06:17.084511] D [MSGID: 0]
> >[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
> >was no cached file and unlink on hashed is not skipped /cdc/kk.txt
> >[2016-11-18 16:06:17.084534] D [MSGID: 0]
> >[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
> >file or directory [No such file or directory]
> >[2016-11-18 16:06:17.084683] D [MSGID: 0]
> >[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:06:17.084783] D [MSGID: 0]
> >[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:06:17.084857] D [MSGID: 0]
> >[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:06:17.084889] D [MSGID: 0]
> >[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:06:17.085001] D [MSGID: 0]
> >[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
> >[No such file or directory]
> >[2016-11-18 16:06:17.086098] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
> >as fast fop
> >[2016-11-18 16:06:17.087256] D [logging.c:1953:_gf_msg_internal]
> >0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> >About to flush least recently used log message to disk
> >[2016-11-18 16:06:17.087173] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> >scheduled as fast fop
> >[2016-11-18 16:06:17.087254] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
> >0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
> >home-lab-3-client-3
> >[2016-11-18 16:06:17.087749] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:06:17.087795] D [MSGID: 0]
> >[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> >returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
> >[2016-11-18 16:06:17.088014] D [MSGID: 0]
> >[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
> >missing on subvol home-lab-3-client-3
> >[2016-11-18 16:06:17.088036] D [MSGID: 0]
> >[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
> >lookup call to 4 subvols
> >[2016-11-18 16:06:17.088598] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:06:17.088639] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
> >[2016-11-18 16:06:17.088804] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:06:17.088855] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
> >[2016-11-18 16:06:17.088819] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:06:17.089069] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
> >[2016-11-18 16:06:17.089380] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:06:17.089421] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
> >[2016-11-18 16:06:17.089443] D [MSGID: 0]
> >[dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
> >hashed_subvol home-lab-3-client-3 cached_subvol null
> >[2016-11-18 16:06:17.089463] D [MSGID: 0]
> >[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
> >was no cached file and unlink on hashed is not skipped /cdc/kk.txt
> >[2016-11-18 16:06:17.089575] D [MSGID: 0]
> >[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
> >file or directory [No such file or directory]
> >[2016-11-18 16:06:17.089601] D [MSGID: 0]
> >[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:06:17.089712] D [MSGID: 0]
> >[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:06:17.089738] D [MSGID: 0]
> >[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:06:17.090006] D [MSGID: 0]
> >[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:06:17.090156] D [MSGID: 0]
> >[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
> >[No such file or directory]
> >[2016-11-18 16:06:17.090648] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled
> >as normal fop
> >[2016-11-18 16:06:17.090732] D [MSGID: 0]
> >[dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
> >creating /cdc/kk.txt on home-lab-3-client-3
> >[2016-11-18 16:06:17.091499] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >event 19 on graph -1418558531
> >[2016-11-18 16:06:17.091843] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
> >scheduled as normal fop
> >[2016-11-18 16:06:17.093831] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >event 19 on graph 0
> >[2016-11-18 16:06:17.094711] D [logging.c:1953:_gf_msg_internal]
> >0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> >About to flush least recently used log message to disk
> >[2016-11-18 16:06:17.093216] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
> >scheduled as normal fop
> >[2016-11-18 16:06:17.094709] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled
> >as fast fop
> >[2016-11-18 16:06:17.095280] E [MSGID: 114031]
> >[client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-3: remote
> >operation failed. Path: /cdc/kk.txt
> >(09bfb6fa-82fb-463b-abd9-bb7ef88e10c0) [Permission denied]
> >[2016-11-18 16:06:17.095341] D [MSGID: 0]
> >[client-rpc-fops.c:448:client3_3_open_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: Permission denied
> >[Permission denied]
> >[2016-11-18 16:06:17.095366] D [MSGID: 0]
> >[dht-inode-read.c:45:dht_open_cbk] 0-home-lab-3-dht: subvolume
> >home-lab-3-client-3 returned -1 [Permission denied]
> >[2016-11-18 16:06:17.095480] D [MSGID: 0]
> >[dht-inode-read.c:59:dht_open_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-dht returned -1 error: Permission denied
> >[Permission denied]
> >[2016-11-18 16:06:17.095671] D [MSGID: 0] [read-ahead.c:101:ra_open_cbk]
> >0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-read-ahead
> >returned -1 error: Permission denied [Permission denied]
> >[2016-11-18 16:06:17.095759] D [MSGID: 0] [io-cache.c:605:ioc_open_cbk]
> >0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-io-cache
> >returned -1 error: Permission denied [Permission denied]
> >[2016-11-18 16:06:17.095963] D [MSGID: 0]
> >[defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: Permission
> >denied [Permission denied]
> >[2016-11-18 16:06:17.095988] D [MSGID: 0]
> >[defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: Permission
> >denied [Permission denied]
> >[2016-11-18 16:06:17.096275] D [MSGID: 0]
> >[io-stats.c:1936:io_stats_open_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3 returned -1 error: Permission denied
> >[Permission denied]
> >[2016-11-18 16:06:17.096362] W [fuse-bridge.c:989:fuse_fd_cbk]
> >0-glusterfs-fuse: 338: OPEN() /cdc/kk.txt => -1 (Permission denied)
> >[2016-11-18 16:06:17.097104] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
> >scheduled as normal fop
> >[2016-11-18 16:06:17.097674] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >event 19 on graph 0
> >[2016-11-18 16:06:17.099912] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >event 19 on graph 0
> >[2016-11-18 16:06:24.785225] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
> >0-home-lab-3-client-0: returning as transport is already disconnected OR
> >there are no frames (0 || 0)
> >[2016-11-18 16:06:24.785338] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
> >0-home-lab-3-client-1: returning as transport is already disconnected OR
> >there are no frames (0 || 0)
> >[2016-11-18 16:06:24.785359] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
> >0-home-lab-3-client-2: returning as transport is already disconnected OR
> >there are no frames (0 || 0)
> >[2016-11-18 16:06:24.785377] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
> >0-home-lab-3-client-3: returning as transport is already disconnected OR
> >there are no frames (0 || 0)
> >
> >
> >
> >
> >
> >** TEST DONE as user cdc --> ERROR
> >
> >
> >
> >
> >
> >
> >[2016-11-18 16:06:46.737686] D
> >[rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
> >/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
> >/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
> >(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
> >/lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
> >/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
> >))))) 0-: 138.4.2.246:49172: ping timer event already removed
> >[2016-11-18 16:06:49.545698] D
> >[rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
> >/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
> >/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
> >(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
> >/lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
> >/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
> >))))) 0-: 138.4.2.177:49157: ping timer event already removed
> >[2016-11-18 16:06:49.546474] D
> >[rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
> >/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
> >/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
> >(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
> >/lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
> >/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
> >))))) 0-: 138.4.2.189:49157: ping timer event already removed
> >[2016-11-18 16:06:49.547093] D
> >[rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
> >/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
> >/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
> >(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
> >/lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
> >/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
> >))))) 0-: 138.4.2.248:49157: ping timer event already removed
> >
> >
> >
> >
> >
> >** user CDC becomed root at this point
> >
> >
> >
> >
> >[2016-11-18 16:07:11.907892] D [logging.c:1953:_gf_msg_internal]
> >0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> >About to flush least recently used log message to disk
> >The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
> >0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 7 times
> >between [2016-11-18 16:06:17.086098] and [2016-11-18 16:07:11.906944]
> >[2016-11-18 16:07:11.907884] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> >scheduled as fast fop
> >[2016-11-18 16:07:11.908875] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >event 19 on graph -1418558531
> >[2016-11-18 16:07:11.909179] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >event 19 on graph 0
> >[2016-11-18 16:07:11.909256] D [logging.c:1953:_gf_msg_internal]
> >0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> >About to flush least recently used log message to disk
> >The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
> >0-home-lab-3-io-threads: UNLINK scheduled as normal fop" repeated 2
> >times between [2016-11-18 16:06:17.047671] and [2016-11-18 16:07:11.908114]
> >[2016-11-18 16:07:11.909254] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
> >as fast fop
> >[2016-11-18 16:07:11.911109] D [logging.c:1953:_gf_msg_internal]
> >0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> >About to flush least recently used log message to disk
> >[2016-11-18 16:07:11.911018] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> >scheduled as fast fop
> >[2016-11-18 16:07:11.911107] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK
> >scheduled as normal fop
> >[2016-11-18 16:07:11.911871] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >event 19 on graph -1418558531
> >[2016-11-18 16:07:11.912045] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >event 19 on graph 0
> >[2016-11-18 16:07:11.913067] D [logging.c:1953:_gf_msg_internal]
> >0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> >About to flush least recently used log message to disk
> >The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
> >0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 2 times
> >between [2016-11-18 16:07:11.909254] and [2016-11-18 16:07:11.912165]
> >[2016-11-18 16:07:11.913065] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> >scheduled as fast fop
> >[2016-11-18 16:07:11.913193] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
> >0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
> >home-lab-3-client-0
> >[2016-11-18 16:07:11.913710] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:07:11.913757] D [MSGID: 0]
> >[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> >returned for /cdc/444.txt with op_ret -1 [No such file or directory]
> >[2016-11-18 16:07:11.913788] D [MSGID: 0]
> >[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
> >missing on subvol home-lab-3-client-0
> >[2016-11-18 16:07:11.913810] D [MSGID: 0]
> >[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
> >lookup call to 4 subvols
> >[2016-11-18 16:07:11.914336] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:07:11.914379] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >home-lab-3-client-0
> >[2016-11-18 16:07:11.914424] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:07:11.914448] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >home-lab-3-client-1
> >[2016-11-18 16:07:11.914501] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:07:11.914617] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >home-lab-3-client-3
> >[2016-11-18 16:07:11.914649] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:07:11.914698] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >home-lab-3-client-2
> >[2016-11-18 16:07:11.914724] D [MSGID: 0]
> >[dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
> >hashed_subvol home-lab-3-client-0 cached_subvol null
> >[2016-11-18 16:07:11.914743] D [MSGID: 0]
> >[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
> >was no cached file and unlink on hashed is not skipped /cdc/444.txt
> >[2016-11-18 16:07:11.914764] D [MSGID: 0]
> >[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
> >file or directory [No such file or directory]
> >[2016-11-18 16:07:11.914791] D [MSGID: 0]
> >[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:07:11.914825] D [MSGID: 0]
> >[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:07:11.914855] D [MSGID: 0]
> >[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:07:11.914880] D [MSGID: 0]
> >[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:07:11.914947] D [MSGID: 0]
> >[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
> >[No such file or directory]
> >[2016-11-18 16:07:11.915569] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> >scheduled as fast fop
> >[2016-11-18 16:07:11.915644] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
> >0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
> >home-lab-3-client-0
> >[2016-11-18 16:07:11.916090] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:07:11.916134] D [MSGID: 0]
> >[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> >returned for /cdc/444.txt with op_ret -1 [No such file or directory]
> >[2016-11-18 16:07:11.916158] D [MSGID: 0]
> >[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
> >missing on subvol home-lab-3-client-0
> >[2016-11-18 16:07:11.916185] D [MSGID: 0]
> >[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
> >lookup call to 4 subvols
> >[2016-11-18 16:07:11.916593] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:07:11.916780] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >home-lab-3-client-0
> >[2016-11-18 16:07:11.916878] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:07:11.916994] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >home-lab-3-client-3
> >[2016-11-18 16:07:11.917076] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:07:11.916906] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:07:11.917117] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >home-lab-3-client-2
> >[2016-11-18 16:07:11.917153] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >home-lab-3-client-1
> >[2016-11-18 16:07:11.917205] D [MSGID: 0]
> >[dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
> >hashed_subvol home-lab-3-client-0 cached_subvol null
> >[2016-11-18 16:07:11.917368] D [MSGID: 0]
> >[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
> >was no cached file and unlink on hashed is not skipped /cdc/444.txt
> >[2016-11-18 16:07:11.917437] D [MSGID: 0]
> >[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
> >file or directory [No such file or directory]
> >[2016-11-18 16:07:11.917467] D [MSGID: 0]
> >[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:07:11.917496] D [MSGID: 0]
> >[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:07:11.917521] D [MSGID: 0]
> >[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:07:11.917551] D [MSGID: 0]
> >[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:07:11.917726] D [MSGID: 0]
> >[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
> >[No such file or directory]
> >[2016-11-18 16:07:11.917958] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled
> >as normal fop
> >[2016-11-18 16:07:11.918230] D [MSGID: 0]
> >[dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
> >creating /cdc/444.txt on home-lab-3-client-0
> >[2016-11-18 16:07:11.918488] D [MSGID: 0]
> >[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
> >'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
> >10456166432768 and avail_inodes is: 99.00
> >[2016-11-18 16:07:11.918570] D [MSGID: 0]
> >[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
> >'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
> >8600417075200 and avail_inodes is: 99.00
> >[2016-11-18 16:07:11.918603] D [MSGID: 0]
> >[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
> >'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
> >10500905500672 and avail_inodes is: 99.00
> >[2016-11-18 16:07:11.918675] D [MSGID: 0]
> >[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
> >'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
> >10499686531072 and avail_inodes is: 99.00
> >[2016-11-18 16:07:11.919095] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >event 19 on graph -1418558531
> >[2016-11-18 16:07:11.919377] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
> >as fast fop
> >[2016-11-18 16:07:11.920040] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled
> >as fast fop
> >[2016-11-18 16:07:11.920889] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled
> >as slow fop
> >[2016-11-18 16:07:11.921051] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled
> >as fast fop
> >[2016-11-18 16:07:11.921583] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >event 19 on graph 0
> >[2016-11-18 16:07:11.922068] D [fuse-bridge.c:52:fuse_invalidate]
> >0-fuse: Invalidate inode id 140363155120276.
> >[2016-11-18 16:07:11.922936] I [io-stats.c:1574:io_stats_dump_fd]
> >0-home-lab-3: --- fd stats ---
> >[2016-11-18 16:07:11.922988] I [io-stats.c:1579:io_stats_dump_fd]
> >0-home-lab-3: Filename : /cdc/444.txt
> >[2016-11-18 16:07:11.923004] I [io-stats.c:1594:io_stats_dump_fd]
> >0-home-lab-3: BytesWritten : 7 bytes
> >[2016-11-18 16:07:11.923026] I [io-stats.c:1606:io_stats_dump_fd]
> >0-home-lab-3: Write 000004b+ : 1
> >[2016-11-18 16:07:11.923779] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
> >as fast fop
> >[2016-11-18 16:07:11.924504] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >event 19 on graph 0
> >[2016-11-18 16:07:11.926692] D [logging.c:1953:_gf_msg_internal]
> >0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> >About to flush least recently used log message to disk
> >The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
> >0-home-lab-3-io-threads: SETATTR scheduled as normal fop" repeated 2
> >times between [2016-11-18 16:06:17.097104] and [2016-11-18 16:07:11.923946]
> >[2016-11-18 16:07:11.926690] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> >scheduled as fast fop
> >[2016-11-18 16:07:11.926858] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
> >0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
> >home-lab-3-client-3
> >[2016-11-18 16:07:11.927327] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:07:11.927376] D [MSGID: 0]
> >[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> >returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
> >[2016-11-18 16:07:11.927401] D [MSGID: 0]
> >[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
> >missing on subvol home-lab-3-client-3
> >[2016-11-18 16:07:11.927422] D [MSGID: 0]
> >[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
> >lookup call to 4 subvols
> >[2016-11-18 16:07:11.927940] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:07:11.927987] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
> >[2016-11-18 16:07:11.928043] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:07:11.928055] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:07:11.928058] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:07:11.928170] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
> >[2016-11-18 16:07:11.928174] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
> >[2016-11-18 16:07:11.928277] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
> >[2016-11-18 16:07:11.928360] D [MSGID: 0]
> >[dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
> >hashed_subvol home-lab-3-client-3 cached_subvol null
> >[2016-11-18 16:07:11.928381] D [MSGID: 0]
> >[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
> >was no cached file and unlink on hashed is not skipped /cdc/kk.txt
> >[2016-11-18 16:07:11.928403] D [MSGID: 0]
> >[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
> >file or directory [No such file or directory]
> >[2016-11-18 16:07:11.928431] D [MSGID: 0]
> >[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:07:11.928456] D [MSGID: 0]
> >[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:07:11.928480] D [MSGID: 0]
> >[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:07:11.928511] D [MSGID: 0]
> >[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:07:11.928753] D [MSGID: 0]
> >[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
> >[No such file or directory]
> >[2016-11-18 16:07:11.930485] D [logging.c:1953:_gf_msg_internal]
> >0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> >About to flush least recently used log message to disk
> >[2016-11-18 16:07:11.929862] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
> >as fast fop
> >[2016-11-18 16:07:11.930483] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> >scheduled as fast fop
> >[2016-11-18 16:07:11.930602] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
> >0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
> >home-lab-3-client-3
> >[2016-11-18 16:07:11.931071] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:07:11.931124] D [MSGID: 0]
> >[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> >returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
> >[2016-11-18 16:07:11.931148] D [MSGID: 0]
> >[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
> >missing on subvol home-lab-3-client-3
> >[2016-11-18 16:07:11.931348] D [MSGID: 0]
> >[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
> >lookup call to 4 subvols
> >[2016-11-18 16:07:11.931880] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:07:11.931927] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
> >[2016-11-18 16:07:11.932002] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:07:11.931994] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:07:11.931991] D [MSGID: 0]
> >[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
> >such file or directory [No such file or directory]
> >[2016-11-18 16:07:11.932078] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
> >[2016-11-18 16:07:11.932116] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
> >[2016-11-18 16:07:11.932315] D [MSGID: 0]
> >[dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
> >with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
> >[2016-11-18 16:07:11.932363] D [MSGID: 0]
> >[dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
> >hashed_subvol home-lab-3-client-3 cached_subvol null
> >[2016-11-18 16:07:11.932383] D [MSGID: 0]
> >[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
> >was no cached file and unlink on hashed is not skipped /cdc/kk.txt
> >[2016-11-18 16:07:11.932407] D [MSGID: 0]
> >[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
> >stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
> >file or directory [No such file or directory]
> >[2016-11-18 16:07:11.932446] D [MSGID: 0]
> >[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:07:11.932471] D [MSGID: 0]
> >[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:07:11.932494] D [MSGID: 0]
> >[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:07:11.932518] D [MSGID: 0]
> >[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
> >directory [No such file or directory]
> >[2016-11-18 16:07:11.932725] D [logging.c:1953:_gf_msg_internal]
> >0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> >About to flush least recently used log message to disk
> >[2016-11-18 16:07:11.932545] D [MSGID: 0]
> >[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
> >0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
> >[No such file or directory]
> >[2016-11-18 16:07:11.932722] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled
> >as normal fop
> >[2016-11-18 16:07:11.933008] D [MSGID: 0]
> >[dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
> >creating /cdc/kk.txt on home-lab-3-client-3
> >[2016-11-18 16:07:11.934086] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >event 19 on graph -1418558531
> >[2016-11-18 16:07:11.934323] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
> >scheduled as normal fop
> >[2016-11-18 16:07:11.934922] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
> >as fast fop
> >[2016-11-18 16:07:11.935497] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
> >scheduled as normal fop
> >[2016-11-18 16:07:11.936102] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >event 19 on graph 0
> >[2016-11-18 16:07:11.936903] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled
> >as fast fop
> >[2016-11-18 16:07:11.937540] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled
> >as slow fop
> >[2016-11-18 16:07:11.937682] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled
> >as fast fop
> >[2016-11-18 16:07:11.938267] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >event 19 on graph 0
> >[2016-11-18 16:07:11.938757] D [fuse-bridge.c:52:fuse_invalidate]
> >0-fuse: Invalidate inode id 140363155120652.
> >[2016-11-18 16:07:11.938994] I [io-stats.c:1574:io_stats_dump_fd]
> >0-home-lab-3: --- fd stats ---
> >[2016-11-18 16:07:11.939072] I [io-stats.c:1579:io_stats_dump_fd]
> >0-home-lab-3: Filename : /cdc/kk.txt
> >[2016-11-18 16:07:11.939096] I [io-stats.c:1594:io_stats_dump_fd]
> >0-home-lab-3: BytesWritten : 7 bytes
> >[2016-11-18 16:07:11.939112] I [io-stats.c:1606:io_stats_dump_fd]
> >0-home-lab-3: Write 000004b+ : 1
> >[2016-11-18 16:07:11.939448] D [MSGID: 0]
> >[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
> >scheduled as normal fop
> >[2016-11-18 16:07:11.940028] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >event 19 on graph 0
> >[2016-11-18 16:07:11.942130] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >event 19 on graph 0
> >
> >
> >
> >
> >
> >** TEST DONE as root --> NO PROBLEM
> >
> >
> >
> >
> >
> >[2016-11-18 16:07:28.794150] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
> >0-home-lab-3-client-2: returning as transport is already disconnected OR
> >there are no frames (0 || 0)
> >[2016-11-18 16:07:31.794656] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
> >0-home-lab-3-client-0: returning as transport is already disconnected OR
> >there are no frames (0 || 0)
> >[2016-11-18 16:07:31.794925] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
> >0-home-lab-3-client-1: returning as transport is already disconnected OR
> >there are no frames (0 || 0)
> >[2016-11-18 16:07:31.794984] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
> >0-home-lab-3-client-3: returning as transport is already disconnected OR
> >there are no frames (0 || 0)
> >
> >
> >** USER ROOT EXITED at this point
> >
> >El 18/11/16 a las 15:16, Miklos Szeredi escribió:
> >>On Wed, Nov 16, 2016 at 7:19 PM, Omar Walid Llorente <[email protected]>
> >>wrote:
> >>
> >>>In any case, the kernel we are using (3.13.0-96-generic #143-Ubuntu
> >>>SMP Mon
> >>>Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux) has fuse
> >>>included
> >>>(not as a module) so if it would be necessary to change the fuse mount
> >>>options some help would be needed.
> >>Can you reproduce this on latest (e.g 4.8) kernel?
> >>
> >>Also fuse debug logs (with whaterver kernel) could be useful; not sure
> >>how to enable it for glusterfs, usual convention for fuse filesystems
> >>is "-d".
> >>
> >>Thanks,
> >>Miklos
> >

2016-11-22 14:45:10

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

On 11/21/2016 08:27 PM, J. Bruce Fields wrote:
> On Mon, Nov 21, 2016 at 06:26:16PM +0530, Soumya Koduri wrote:
>> Hi,
>>
>> Sorry for the delay. I could reproduce the issue on my gluster setup
>> as well. The error is generated at posix-acl xlator layer of
>> glusterfs server.
>>
>> As Bruce mentioned in the earlier mail, as part of WRITE call,
>> NFS-server issues OPEN() with O_WRONLY flag set. Since the
>> attributes of the file "kk.txt" when created was 0400, the
>> corresponding acl maps to below ACL -
>>
>> [root@dhcp35-197 vol1]# getfacl kk.txt
>> # file: kk.txt
>> # owner: guest
>> # group: mock
>> user::r--
>> group::---
>> other::---
>>
>> And hence the permission check for WRITE access fails. Also not just
>> glusterfs, I see similar behavior with ext4 filesystem as well when
>> exported via kernel-NFS server.
>
> That definitely shouldn't be happening, how are you reproducing?

Sorry. Yesterday when I was testing, both the exports (one on glusterfs
fuse mount and another on ext4) had same fsid. So when I was trying to
mount and access the share on ext4, it had redirected to glusterfs fuse
mount and hence the test failed.

I re-checked with ext4 and I do not see any failures. So probably then
we need to put a fix in glusterfs to bypass this check for file owner
during OPEN.

@Omar,
Could you please file a bug against glusterfs component in RH bugzilla
[1] for the same?

Thanks,
Soumya

[1] https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS

>
> --b.
>
>> So I am not sure if fixing glusterfs
>> for this use-case is valid.
>>
>> Thanks,
>> Soumya
>>
>>
>> On 11/18/2016 09:33 PM, Omar Walid Llorente wrote:
>>> Yes, of course, we can reproduce the problem on latest (e.g 4.8) kernel.
>>>
>>> Tested on a server with fedora 24 (Linux cuentas11.lab.dit.upm.es
>>> 4.8.4-200.fc24.x86_64 #1 SMP Tue Oct 25 13:06:04 UTC 2016 x86_64 x86_64
>>> x86_64 GNU/Linux), glusterfs-fuse-3.8.5-1.fc24.x86_64,
>>> fuse-2.9.7-1.fc24.x86_64 and nfs-utils-1.3.4-1.rc2.fc24.x86_64 that
>>> mounts as a glusterfs-client and exports as a nfs-server a ditributed
>>> glusterfs volume (that comes from a 3.7.4 glusterfs server over Ubuntu
>>> 14.04.
>>>
>>> [root@cuentas11 ~]# cat /proc/mounts | grep home
>>> recipiente8hd:/home-lab-3.tcp /home-3 fuse.glusterfs
>>> rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072
>>> 0 0
>>> [root@cuentas11 ~]#
>>>
>>> As you may remember the schema is the one that follows (glusterfs volume
>>> name: home-lab-3, glusterfs client mount point: /home-3, nfs client
>>> mount point: /home):
>>>
>>> nfs client (v1.2.8) ---> NFSv3 ---> knfsd (v1.3.4 over a Linux v4.8.4
>>> kernel) ---> FUSE (v2.9.7) --> glusterfs client (v3.8.5) --> GLUSTER
>>> over TCP --> glusterfs server (v3.7.4)
>>>
>>> The errors at the nfs-client side are the same:
>>>
>>> cdc@l056:~$ mount | grep "/home/cdc"
>>> cuentas11:/home-3/cdc on /home/cdc type nfs
>>> (rw,noatime,nfsvers=3,vers=3,tcp,intr,fsc,nolock,rsize=262140,wsize=262140,addr=138.4.30.86)
>>>
>>> cdc@l056:~$
>>> cdc@l056:~$
>>> cdc@l056:~$
>>> cdc@l056:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444
>>> 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
>>> cp: failed to close ‘kk.txt’: Permission denied
>>> -r--r--r-- 1 cdc admincdc 7 nov 18 2016 444.txt
>>> -r--r--r-- 1 cdc admincdc 0 nov 18 2016 kk.txt
>>> cdc@l056:~$
>>> cdc@l056:~$
>>> cdc@l056:~$
>>> cdc@l056:~$ sudo su
>>> [sudo] password for cdc:
>>> root@l056:/home/cdc# rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
>>> chmod 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
>>> -r--r--r-- 1 root root 7 nov 18 2016 444.txt
>>> -r--r--r-- 1 root root 7 nov 18 2016 kk.txt
>>> root@l056:/home/cdc#
>>> root@l056:/home/cdc#
>>> root@l056:/home/cdc# exit
>>> cdc@l056:~$
>>>
>>> See at the end the logs with and without the debug option activated at
>>> the glusterfs client.
>>>
>>> Thanks again, and sorry for the long message.
>>>
>>> Omar
>>>
>>>
>>>
>>> * Logs on the glusterfs client side with NO DEBUG:
>>>
>>> [2016-11-18 15:45:43.681215] E [MSGID: 114031]
>>> [client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-3: remote
>>> operation failed. Path: /cdc/kk.txt
>>> (cc98e831-3410-4202-8254-2d4b48e8ea68) [Permission denied]
>>> [2016-11-18 15:45:43.681297] W [fuse-bridge.c:989:fuse_fd_cbk]
>>> 0-glusterfs-fuse: 399: OPEN() /cdc/kk.txt => -1 (Permission denied)
>>> [2016-11-18 15:45:55.457922] I [io-stats.c:1574:io_stats_dump_fd]
>>> 0-home-lab-3: --- fd stats ---
>>> [2016-11-18 15:45:55.457978] I [io-stats.c:1579:io_stats_dump_fd]
>>> 0-home-lab-3: Filename : /cdc/444.txt
>>> [2016-11-18 15:45:55.457997] I [io-stats.c:1594:io_stats_dump_fd]
>>> 0-home-lab-3: BytesWritten : 7 bytes
>>> [2016-11-18 15:45:55.458011] I [io-stats.c:1606:io_stats_dump_fd]
>>> 0-home-lab-3: Write 000004b+ : 1
>>> [2016-11-18 15:45:55.471933] I [io-stats.c:1574:io_stats_dump_fd]
>>> 0-home-lab-3: --- fd stats ---
>>>
>>>
>>> * Logs on the glusterfs client side with DEBUG activated:
>>>
>>> [2016-11-18 16:06:17.040092] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
>>> as fast fop
>>> [2016-11-18 16:06:17.045986] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>> scheduled as fast fop
>>> [2016-11-18 16:06:17.046132] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
>>> home-lab-3-client-3
>>> [2016-11-18 16:06:17.046833] D [MSGID: 0]
>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>> returned for /cdc/kk.txt with op_ret 0 [Invalid argument]
>>> [2016-11-18 16:06:17.047671] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK
>>> scheduled as normal fop
>>> [2016-11-18 16:06:17.050539] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>> event 19 on graph -1418558531
>>> [2016-11-18 16:06:17.050867] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>> event 19 on graph 0
>>> [2016-11-18 16:06:17.051948] D [logging.c:1953:_gf_msg_internal]
>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>> About to flush least recently used log message to disk
>>> [2016-11-18 16:06:17.051871] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>> scheduled as fast fop
>>> [2016-11-18 16:06:17.051946] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
>>> home-lab-3-client-0
>>> [2016-11-18 16:06:17.062108] D [MSGID: 0]
>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>> returned for /cdc/444.txt with op_ret 0 [Invalid argument]
>>> [2016-11-18 16:06:17.063692] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>> event 19 on graph -1418558531
>>> [2016-11-18 16:06:17.064015] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>> event 19 on graph 0
>>> [2016-11-18 16:06:17.065068] D [logging.c:1953:_gf_msg_internal]
>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>> About to flush least recently used log message to disk
>>> The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
>>> 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 4 times
>>> between [2016-11-18 16:06:17.040092] and [2016-11-18 16:06:17.064175]
>>> [2016-11-18 16:06:17.065066] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>> scheduled as fast fop
>>> [2016-11-18 16:06:17.065232] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
>>> home-lab-3-client-0
>>> [2016-11-18 16:06:17.065755] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:06:17.065897] D [MSGID: 0]
>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>> returned for /cdc/444.txt with op_ret -1 [No such file or directory]
>>> [2016-11-18 16:06:17.066016] D [MSGID: 0]
>>> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
>>> missing on subvol home-lab-3-client-0
>>> [2016-11-18 16:06:17.066104] D [MSGID: 0]
>>> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
>>> lookup call to 4 subvols
>>> [2016-11-18 16:06:17.066594] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:06:17.066669] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>> home-lab-3-client-0
>>> [2016-11-18 16:06:17.066771] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:06:17.066898] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>> home-lab-3-client-1
>>> [2016-11-18 16:06:17.066906] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:06:17.067019] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>> home-lab-3-client-2
>>> [2016-11-18 16:06:17.066915] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:06:17.067066] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>> home-lab-3-client-3
>>> [2016-11-18 16:06:17.067107] D [MSGID: 0]
>>> [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
>>> hashed_subvol home-lab-3-client-0 cached_subvol null
>>> [2016-11-18 16:06:17.067151] D [MSGID: 0]
>>> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
>>> was no cached file and unlink on hashed is not skipped /cdc/444.txt
>>> [2016-11-18 16:06:17.067179] D [MSGID: 0]
>>> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1 error: No such
>>> file or directory [No such file or directory]
>>> [2016-11-18 16:06:17.067206] D [MSGID: 0]
>>> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:06:17.067250] D [MSGID: 0]
>>> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:06:17.067275] D [MSGID: 0]
>>> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:06:17.067308] D [MSGID: 0]
>>> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:06:17.067387] D [MSGID: 0]
>>> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file or directory
>>> [No such file or directory]
>>> [2016-11-18 16:06:17.068157] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>> scheduled as fast fop
>>> [2016-11-18 16:06:17.068281] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
>>> home-lab-3-client-0
>>> [2016-11-18 16:06:17.068738] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:06:17.068781] D [MSGID: 0]
>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>> returned for /cdc/444.txt with op_ret -1 [No such file or directory]
>>> [2016-11-18 16:06:17.068805] D [MSGID: 0]
>>> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
>>> missing on subvol home-lab-3-client-0
>>> [2016-11-18 16:06:17.068827] D [MSGID: 0]
>>> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
>>> lookup call to 4 subvols
>>> [2016-11-18 16:06:17.069286] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:06:17.069345] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>> home-lab-3-client-0
>>> [2016-11-18 16:06:17.069452] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:06:17.069507] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>> home-lab-3-client-2
>>> [2016-11-18 16:06:17.069556] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:06:17.069585] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>> home-lab-3-client-3
>>> [2016-11-18 16:06:17.069939] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:06:17.069983] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>> home-lab-3-client-1
>>> [2016-11-18 16:06:17.070005] D [MSGID: 0]
>>> [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
>>> hashed_subvol home-lab-3-client-0 cached_subvol null
>>> [2016-11-18 16:06:17.070038] D [MSGID: 0]
>>> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
>>> was no cached file and unlink on hashed is not skipped /cdc/444.txt
>>> [2016-11-18 16:06:17.070098] D [MSGID: 0]
>>> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1 error: No such
>>> file or directory [No such file or directory]
>>> [2016-11-18 16:06:17.070128] D [MSGID: 0]
>>> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:06:17.070154] D [MSGID: 0]
>>> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:06:17.070178] D [MSGID: 0]
>>> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:06:17.070208] D [MSGID: 0]
>>> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:06:17.070251] D [MSGID: 0]
>>> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file or directory
>>> [No such file or directory]
>>> [2016-11-18 16:06:17.070537] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled
>>> as normal fop
>>> [2016-11-18 16:06:17.070826] D [MSGID: 0]
>>> [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
>>> creating /cdc/444.txt on home-lab-3-client-0
>>> [2016-11-18 16:06:17.071100] D [MSGID: 0]
>>> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
>>> 'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
>>> 10456189239296 and avail_inodes is: 99.00
>>> [2016-11-18 16:06:17.071142] D [MSGID: 0]
>>> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
>>> 'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
>>> 10499693871104 and avail_inodes is: 99.00
>>> [2016-11-18 16:06:17.071183] D [MSGID: 0]
>>> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
>>> 'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
>>> 10500911267840 and avail_inodes is: 99.00
>>> [2016-11-18 16:06:17.071343] D [MSGID: 0]
>>> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
>>> 'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
>>> 8600432410624 and avail_inodes is: 99.00
>>> [2016-11-18 16:06:17.073943] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>> event 19 on graph -1418558531
>>> [2016-11-18 16:06:17.074273] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
>>> as fast fop
>>> [2016-11-18 16:06:17.075035] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled
>>> as fast fop
>>> [2016-11-18 16:06:17.075913] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled
>>> as slow fop
>>> [2016-11-18 16:06:17.076205] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled
>>> as fast fop
>>> [2016-11-18 16:06:17.076636] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>> event 19 on graph 0
>>> [2016-11-18 16:06:17.077282] D [fuse-bridge.c:52:fuse_invalidate]
>>> 0-fuse: Invalidate inode id 140363155120276.
>>> [2016-11-18 16:06:17.077981] I [io-stats.c:1574:io_stats_dump_fd]
>>> 0-home-lab-3: --- fd stats ---
>>> [2016-11-18 16:06:17.078032] I [io-stats.c:1579:io_stats_dump_fd]
>>> 0-home-lab-3: Filename : /cdc/444.txt
>>> [2016-11-18 16:06:17.078049] I [io-stats.c:1594:io_stats_dump_fd]
>>> 0-home-lab-3: BytesWritten : 7 bytes
>>> [2016-11-18 16:06:17.078064] I [io-stats.c:1606:io_stats_dump_fd]
>>> 0-home-lab-3: Write 000004b+ : 1
>>> [2016-11-18 16:06:17.079156] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
>>> scheduled as normal fop
>>> [2016-11-18 16:06:17.079794] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>> event 19 on graph 0
>>> [2016-11-18 16:06:17.082196] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>> scheduled as fast fop
>>> [2016-11-18 16:06:17.082772] D [logging.c:1953:_gf_msg_internal]
>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>> About to flush least recently used log message to disk
>>> [2016-11-18 16:06:17.082318] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
>>> home-lab-3-client-3
>>> [2016-11-18 16:06:17.082770] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:06:17.083168] D [MSGID: 0]
>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>> returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
>>> [2016-11-18 16:06:17.083282] D [MSGID: 0]
>>> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
>>> missing on subvol home-lab-3-client-3
>>> [2016-11-18 16:06:17.083320] D [MSGID: 0]
>>> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
>>> lookup call to 4 subvols
>>> [2016-11-18 16:06:17.083991] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:06:17.084043] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
>>> [2016-11-18 16:06:17.084048] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:06:17.084092] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
>>> [2016-11-18 16:06:17.084114] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:06:17.084167] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
>>> [2016-11-18 16:06:17.084091] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:06:17.084377] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
>>> [2016-11-18 16:06:17.084406] D [MSGID: 0]
>>> [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
>>> hashed_subvol home-lab-3-client-3 cached_subvol null
>>> [2016-11-18 16:06:17.084511] D [MSGID: 0]
>>> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
>>> was no cached file and unlink on hashed is not skipped /cdc/kk.txt
>>> [2016-11-18 16:06:17.084534] D [MSGID: 0]
>>> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
>>> file or directory [No such file or directory]
>>> [2016-11-18 16:06:17.084683] D [MSGID: 0]
>>> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:06:17.084783] D [MSGID: 0]
>>> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:06:17.084857] D [MSGID: 0]
>>> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:06:17.084889] D [MSGID: 0]
>>> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:06:17.085001] D [MSGID: 0]
>>> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
>>> [No such file or directory]
>>> [2016-11-18 16:06:17.086098] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
>>> as fast fop
>>> [2016-11-18 16:06:17.087256] D [logging.c:1953:_gf_msg_internal]
>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>> About to flush least recently used log message to disk
>>> [2016-11-18 16:06:17.087173] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>> scheduled as fast fop
>>> [2016-11-18 16:06:17.087254] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
>>> home-lab-3-client-3
>>> [2016-11-18 16:06:17.087749] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:06:17.087795] D [MSGID: 0]
>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>> returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
>>> [2016-11-18 16:06:17.088014] D [MSGID: 0]
>>> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
>>> missing on subvol home-lab-3-client-3
>>> [2016-11-18 16:06:17.088036] D [MSGID: 0]
>>> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
>>> lookup call to 4 subvols
>>> [2016-11-18 16:06:17.088598] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:06:17.088639] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
>>> [2016-11-18 16:06:17.088804] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:06:17.088855] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
>>> [2016-11-18 16:06:17.088819] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:06:17.089069] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
>>> [2016-11-18 16:06:17.089380] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:06:17.089421] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
>>> [2016-11-18 16:06:17.089443] D [MSGID: 0]
>>> [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
>>> hashed_subvol home-lab-3-client-3 cached_subvol null
>>> [2016-11-18 16:06:17.089463] D [MSGID: 0]
>>> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
>>> was no cached file and unlink on hashed is not skipped /cdc/kk.txt
>>> [2016-11-18 16:06:17.089575] D [MSGID: 0]
>>> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
>>> file or directory [No such file or directory]
>>> [2016-11-18 16:06:17.089601] D [MSGID: 0]
>>> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:06:17.089712] D [MSGID: 0]
>>> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:06:17.089738] D [MSGID: 0]
>>> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:06:17.090006] D [MSGID: 0]
>>> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:06:17.090156] D [MSGID: 0]
>>> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
>>> [No such file or directory]
>>> [2016-11-18 16:06:17.090648] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled
>>> as normal fop
>>> [2016-11-18 16:06:17.090732] D [MSGID: 0]
>>> [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
>>> creating /cdc/kk.txt on home-lab-3-client-3
>>> [2016-11-18 16:06:17.091499] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>> event 19 on graph -1418558531
>>> [2016-11-18 16:06:17.091843] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
>>> scheduled as normal fop
>>> [2016-11-18 16:06:17.093831] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>> event 19 on graph 0
>>> [2016-11-18 16:06:17.094711] D [logging.c:1953:_gf_msg_internal]
>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>> About to flush least recently used log message to disk
>>> [2016-11-18 16:06:17.093216] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
>>> scheduled as normal fop
>>> [2016-11-18 16:06:17.094709] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled
>>> as fast fop
>>> [2016-11-18 16:06:17.095280] E [MSGID: 114031]
>>> [client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-3: remote
>>> operation failed. Path: /cdc/kk.txt
>>> (09bfb6fa-82fb-463b-abd9-bb7ef88e10c0) [Permission denied]
>>> [2016-11-18 16:06:17.095341] D [MSGID: 0]
>>> [client-rpc-fops.c:448:client3_3_open_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: Permission denied
>>> [Permission denied]
>>> [2016-11-18 16:06:17.095366] D [MSGID: 0]
>>> [dht-inode-read.c:45:dht_open_cbk] 0-home-lab-3-dht: subvolume
>>> home-lab-3-client-3 returned -1 [Permission denied]
>>> [2016-11-18 16:06:17.095480] D [MSGID: 0]
>>> [dht-inode-read.c:59:dht_open_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: Permission denied
>>> [Permission denied]
>>> [2016-11-18 16:06:17.095671] D [MSGID: 0] [read-ahead.c:101:ra_open_cbk]
>>> 0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-read-ahead
>>> returned -1 error: Permission denied [Permission denied]
>>> [2016-11-18 16:06:17.095759] D [MSGID: 0] [io-cache.c:605:ioc_open_cbk]
>>> 0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-io-cache
>>> returned -1 error: Permission denied [Permission denied]
>>> [2016-11-18 16:06:17.095963] D [MSGID: 0]
>>> [defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: Permission
>>> denied [Permission denied]
>>> [2016-11-18 16:06:17.095988] D [MSGID: 0]
>>> [defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: Permission
>>> denied [Permission denied]
>>> [2016-11-18 16:06:17.096275] D [MSGID: 0]
>>> [io-stats.c:1936:io_stats_open_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3 returned -1 error: Permission denied
>>> [Permission denied]
>>> [2016-11-18 16:06:17.096362] W [fuse-bridge.c:989:fuse_fd_cbk]
>>> 0-glusterfs-fuse: 338: OPEN() /cdc/kk.txt => -1 (Permission denied)
>>> [2016-11-18 16:06:17.097104] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
>>> scheduled as normal fop
>>> [2016-11-18 16:06:17.097674] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>> event 19 on graph 0
>>> [2016-11-18 16:06:17.099912] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>> event 19 on graph 0
>>> [2016-11-18 16:06:24.785225] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
>>> 0-home-lab-3-client-0: returning as transport is already disconnected OR
>>> there are no frames (0 || 0)
>>> [2016-11-18 16:06:24.785338] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
>>> 0-home-lab-3-client-1: returning as transport is already disconnected OR
>>> there are no frames (0 || 0)
>>> [2016-11-18 16:06:24.785359] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
>>> 0-home-lab-3-client-2: returning as transport is already disconnected OR
>>> there are no frames (0 || 0)
>>> [2016-11-18 16:06:24.785377] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
>>> 0-home-lab-3-client-3: returning as transport is already disconnected OR
>>> there are no frames (0 || 0)
>>>
>>>
>>>
>>>
>>>
>>> ** TEST DONE as user cdc --> ERROR
>>>
>>>
>>>
>>>
>>>
>>>
>>> [2016-11-18 16:06:46.737686] D
>>> [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
>>> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
>>> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
>>> (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
>>> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
>>> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
>>> ))))) 0-: 138.4.2.246:49172: ping timer event already removed
>>> [2016-11-18 16:06:49.545698] D
>>> [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
>>> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
>>> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
>>> (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
>>> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
>>> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
>>> ))))) 0-: 138.4.2.177:49157: ping timer event already removed
>>> [2016-11-18 16:06:49.546474] D
>>> [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
>>> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
>>> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
>>> (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
>>> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
>>> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
>>> ))))) 0-: 138.4.2.189:49157: ping timer event already removed
>>> [2016-11-18 16:06:49.547093] D
>>> [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
>>> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
>>> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
>>> (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
>>> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
>>> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
>>> ))))) 0-: 138.4.2.248:49157: ping timer event already removed
>>>
>>>
>>>
>>>
>>>
>>> ** user CDC becomed root at this point
>>>
>>>
>>>
>>>
>>> [2016-11-18 16:07:11.907892] D [logging.c:1953:_gf_msg_internal]
>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>> About to flush least recently used log message to disk
>>> The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
>>> 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 7 times
>>> between [2016-11-18 16:06:17.086098] and [2016-11-18 16:07:11.906944]
>>> [2016-11-18 16:07:11.907884] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>> scheduled as fast fop
>>> [2016-11-18 16:07:11.908875] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>> event 19 on graph -1418558531
>>> [2016-11-18 16:07:11.909179] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>> event 19 on graph 0
>>> [2016-11-18 16:07:11.909256] D [logging.c:1953:_gf_msg_internal]
>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>> About to flush least recently used log message to disk
>>> The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
>>> 0-home-lab-3-io-threads: UNLINK scheduled as normal fop" repeated 2
>>> times between [2016-11-18 16:06:17.047671] and [2016-11-18 16:07:11.908114]
>>> [2016-11-18 16:07:11.909254] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
>>> as fast fop
>>> [2016-11-18 16:07:11.911109] D [logging.c:1953:_gf_msg_internal]
>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>> About to flush least recently used log message to disk
>>> [2016-11-18 16:07:11.911018] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>> scheduled as fast fop
>>> [2016-11-18 16:07:11.911107] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK
>>> scheduled as normal fop
>>> [2016-11-18 16:07:11.911871] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>> event 19 on graph -1418558531
>>> [2016-11-18 16:07:11.912045] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>> event 19 on graph 0
>>> [2016-11-18 16:07:11.913067] D [logging.c:1953:_gf_msg_internal]
>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>> About to flush least recently used log message to disk
>>> The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
>>> 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 2 times
>>> between [2016-11-18 16:07:11.909254] and [2016-11-18 16:07:11.912165]
>>> [2016-11-18 16:07:11.913065] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>> scheduled as fast fop
>>> [2016-11-18 16:07:11.913193] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
>>> home-lab-3-client-0
>>> [2016-11-18 16:07:11.913710] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:07:11.913757] D [MSGID: 0]
>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>> returned for /cdc/444.txt with op_ret -1 [No such file or directory]
>>> [2016-11-18 16:07:11.913788] D [MSGID: 0]
>>> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
>>> missing on subvol home-lab-3-client-0
>>> [2016-11-18 16:07:11.913810] D [MSGID: 0]
>>> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
>>> lookup call to 4 subvols
>>> [2016-11-18 16:07:11.914336] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:07:11.914379] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>> home-lab-3-client-0
>>> [2016-11-18 16:07:11.914424] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:07:11.914448] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>> home-lab-3-client-1
>>> [2016-11-18 16:07:11.914501] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:07:11.914617] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>> home-lab-3-client-3
>>> [2016-11-18 16:07:11.914649] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:07:11.914698] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>> home-lab-3-client-2
>>> [2016-11-18 16:07:11.914724] D [MSGID: 0]
>>> [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
>>> hashed_subvol home-lab-3-client-0 cached_subvol null
>>> [2016-11-18 16:07:11.914743] D [MSGID: 0]
>>> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
>>> was no cached file and unlink on hashed is not skipped /cdc/444.txt
>>> [2016-11-18 16:07:11.914764] D [MSGID: 0]
>>> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
>>> file or directory [No such file or directory]
>>> [2016-11-18 16:07:11.914791] D [MSGID: 0]
>>> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:07:11.914825] D [MSGID: 0]
>>> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:07:11.914855] D [MSGID: 0]
>>> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:07:11.914880] D [MSGID: 0]
>>> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:07:11.914947] D [MSGID: 0]
>>> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
>>> [No such file or directory]
>>> [2016-11-18 16:07:11.915569] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>> scheduled as fast fop
>>> [2016-11-18 16:07:11.915644] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
>>> home-lab-3-client-0
>>> [2016-11-18 16:07:11.916090] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:07:11.916134] D [MSGID: 0]
>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>> returned for /cdc/444.txt with op_ret -1 [No such file or directory]
>>> [2016-11-18 16:07:11.916158] D [MSGID: 0]
>>> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
>>> missing on subvol home-lab-3-client-0
>>> [2016-11-18 16:07:11.916185] D [MSGID: 0]
>>> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
>>> lookup call to 4 subvols
>>> [2016-11-18 16:07:11.916593] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:07:11.916780] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>> home-lab-3-client-0
>>> [2016-11-18 16:07:11.916878] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:07:11.916994] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>> home-lab-3-client-3
>>> [2016-11-18 16:07:11.917076] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:07:11.916906] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:07:11.917117] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>> home-lab-3-client-2
>>> [2016-11-18 16:07:11.917153] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>> home-lab-3-client-1
>>> [2016-11-18 16:07:11.917205] D [MSGID: 0]
>>> [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
>>> hashed_subvol home-lab-3-client-0 cached_subvol null
>>> [2016-11-18 16:07:11.917368] D [MSGID: 0]
>>> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
>>> was no cached file and unlink on hashed is not skipped /cdc/444.txt
>>> [2016-11-18 16:07:11.917437] D [MSGID: 0]
>>> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
>>> file or directory [No such file or directory]
>>> [2016-11-18 16:07:11.917467] D [MSGID: 0]
>>> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:07:11.917496] D [MSGID: 0]
>>> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:07:11.917521] D [MSGID: 0]
>>> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:07:11.917551] D [MSGID: 0]
>>> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:07:11.917726] D [MSGID: 0]
>>> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
>>> [No such file or directory]
>>> [2016-11-18 16:07:11.917958] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled
>>> as normal fop
>>> [2016-11-18 16:07:11.918230] D [MSGID: 0]
>>> [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
>>> creating /cdc/444.txt on home-lab-3-client-0
>>> [2016-11-18 16:07:11.918488] D [MSGID: 0]
>>> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
>>> 'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
>>> 10456166432768 and avail_inodes is: 99.00
>>> [2016-11-18 16:07:11.918570] D [MSGID: 0]
>>> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
>>> 'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
>>> 8600417075200 and avail_inodes is: 99.00
>>> [2016-11-18 16:07:11.918603] D [MSGID: 0]
>>> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
>>> 'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
>>> 10500905500672 and avail_inodes is: 99.00
>>> [2016-11-18 16:07:11.918675] D [MSGID: 0]
>>> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
>>> 'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
>>> 10499686531072 and avail_inodes is: 99.00
>>> [2016-11-18 16:07:11.919095] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>> event 19 on graph -1418558531
>>> [2016-11-18 16:07:11.919377] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
>>> as fast fop
>>> [2016-11-18 16:07:11.920040] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled
>>> as fast fop
>>> [2016-11-18 16:07:11.920889] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled
>>> as slow fop
>>> [2016-11-18 16:07:11.921051] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled
>>> as fast fop
>>> [2016-11-18 16:07:11.921583] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>> event 19 on graph 0
>>> [2016-11-18 16:07:11.922068] D [fuse-bridge.c:52:fuse_invalidate]
>>> 0-fuse: Invalidate inode id 140363155120276.
>>> [2016-11-18 16:07:11.922936] I [io-stats.c:1574:io_stats_dump_fd]
>>> 0-home-lab-3: --- fd stats ---
>>> [2016-11-18 16:07:11.922988] I [io-stats.c:1579:io_stats_dump_fd]
>>> 0-home-lab-3: Filename : /cdc/444.txt
>>> [2016-11-18 16:07:11.923004] I [io-stats.c:1594:io_stats_dump_fd]
>>> 0-home-lab-3: BytesWritten : 7 bytes
>>> [2016-11-18 16:07:11.923026] I [io-stats.c:1606:io_stats_dump_fd]
>>> 0-home-lab-3: Write 000004b+ : 1
>>> [2016-11-18 16:07:11.923779] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
>>> as fast fop
>>> [2016-11-18 16:07:11.924504] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>> event 19 on graph 0
>>> [2016-11-18 16:07:11.926692] D [logging.c:1953:_gf_msg_internal]
>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>> About to flush least recently used log message to disk
>>> The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
>>> 0-home-lab-3-io-threads: SETATTR scheduled as normal fop" repeated 2
>>> times between [2016-11-18 16:06:17.097104] and [2016-11-18 16:07:11.923946]
>>> [2016-11-18 16:07:11.926690] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>> scheduled as fast fop
>>> [2016-11-18 16:07:11.926858] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
>>> home-lab-3-client-3
>>> [2016-11-18 16:07:11.927327] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:07:11.927376] D [MSGID: 0]
>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>> returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
>>> [2016-11-18 16:07:11.927401] D [MSGID: 0]
>>> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
>>> missing on subvol home-lab-3-client-3
>>> [2016-11-18 16:07:11.927422] D [MSGID: 0]
>>> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
>>> lookup call to 4 subvols
>>> [2016-11-18 16:07:11.927940] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:07:11.927987] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
>>> [2016-11-18 16:07:11.928043] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:07:11.928055] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:07:11.928058] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:07:11.928170] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
>>> [2016-11-18 16:07:11.928174] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
>>> [2016-11-18 16:07:11.928277] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
>>> [2016-11-18 16:07:11.928360] D [MSGID: 0]
>>> [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
>>> hashed_subvol home-lab-3-client-3 cached_subvol null
>>> [2016-11-18 16:07:11.928381] D [MSGID: 0]
>>> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
>>> was no cached file and unlink on hashed is not skipped /cdc/kk.txt
>>> [2016-11-18 16:07:11.928403] D [MSGID: 0]
>>> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
>>> file or directory [No such file or directory]
>>> [2016-11-18 16:07:11.928431] D [MSGID: 0]
>>> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:07:11.928456] D [MSGID: 0]
>>> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:07:11.928480] D [MSGID: 0]
>>> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:07:11.928511] D [MSGID: 0]
>>> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:07:11.928753] D [MSGID: 0]
>>> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
>>> [No such file or directory]
>>> [2016-11-18 16:07:11.930485] D [logging.c:1953:_gf_msg_internal]
>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>> About to flush least recently used log message to disk
>>> [2016-11-18 16:07:11.929862] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
>>> as fast fop
>>> [2016-11-18 16:07:11.930483] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>> scheduled as fast fop
>>> [2016-11-18 16:07:11.930602] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
>>> home-lab-3-client-3
>>> [2016-11-18 16:07:11.931071] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:07:11.931124] D [MSGID: 0]
>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>> returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
>>> [2016-11-18 16:07:11.931148] D [MSGID: 0]
>>> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
>>> missing on subvol home-lab-3-client-3
>>> [2016-11-18 16:07:11.931348] D [MSGID: 0]
>>> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
>>> lookup call to 4 subvols
>>> [2016-11-18 16:07:11.931880] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:07:11.931927] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
>>> [2016-11-18 16:07:11.932002] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:07:11.931994] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:07:11.931991] D [MSGID: 0]
>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
>>> such file or directory [No such file or directory]
>>> [2016-11-18 16:07:11.932078] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
>>> [2016-11-18 16:07:11.932116] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
>>> [2016-11-18 16:07:11.932315] D [MSGID: 0]
>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
>>> [2016-11-18 16:07:11.932363] D [MSGID: 0]
>>> [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
>>> hashed_subvol home-lab-3-client-3 cached_subvol null
>>> [2016-11-18 16:07:11.932383] D [MSGID: 0]
>>> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
>>> was no cached file and unlink on hashed is not skipped /cdc/kk.txt
>>> [2016-11-18 16:07:11.932407] D [MSGID: 0]
>>> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
>>> stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such
>>> file or directory [No such file or directory]
>>> [2016-11-18 16:07:11.932446] D [MSGID: 0]
>>> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:07:11.932471] D [MSGID: 0]
>>> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:07:11.932494] D [MSGID: 0]
>>> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:07:11.932518] D [MSGID: 0]
>>> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
>>> directory [No such file or directory]
>>> [2016-11-18 16:07:11.932725] D [logging.c:1953:_gf_msg_internal]
>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>> About to flush least recently used log message to disk
>>> [2016-11-18 16:07:11.932545] D [MSGID: 0]
>>> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
>>> 0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
>>> [No such file or directory]
>>> [2016-11-18 16:07:11.932722] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled
>>> as normal fop
>>> [2016-11-18 16:07:11.933008] D [MSGID: 0]
>>> [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
>>> creating /cdc/kk.txt on home-lab-3-client-3
>>> [2016-11-18 16:07:11.934086] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>> event 19 on graph -1418558531
>>> [2016-11-18 16:07:11.934323] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
>>> scheduled as normal fop
>>> [2016-11-18 16:07:11.934922] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled
>>> as fast fop
>>> [2016-11-18 16:07:11.935497] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
>>> scheduled as normal fop
>>> [2016-11-18 16:07:11.936102] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>> event 19 on graph 0
>>> [2016-11-18 16:07:11.936903] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled
>>> as fast fop
>>> [2016-11-18 16:07:11.937540] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled
>>> as slow fop
>>> [2016-11-18 16:07:11.937682] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled
>>> as fast fop
>>> [2016-11-18 16:07:11.938267] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>> event 19 on graph 0
>>> [2016-11-18 16:07:11.938757] D [fuse-bridge.c:52:fuse_invalidate]
>>> 0-fuse: Invalidate inode id 140363155120652.
>>> [2016-11-18 16:07:11.938994] I [io-stats.c:1574:io_stats_dump_fd]
>>> 0-home-lab-3: --- fd stats ---
>>> [2016-11-18 16:07:11.939072] I [io-stats.c:1579:io_stats_dump_fd]
>>> 0-home-lab-3: Filename : /cdc/kk.txt
>>> [2016-11-18 16:07:11.939096] I [io-stats.c:1594:io_stats_dump_fd]
>>> 0-home-lab-3: BytesWritten : 7 bytes
>>> [2016-11-18 16:07:11.939112] I [io-stats.c:1606:io_stats_dump_fd]
>>> 0-home-lab-3: Write 000004b+ : 1
>>> [2016-11-18 16:07:11.939448] D [MSGID: 0]
>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
>>> scheduled as normal fop
>>> [2016-11-18 16:07:11.940028] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>> event 19 on graph 0
>>> [2016-11-18 16:07:11.942130] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>> event 19 on graph 0
>>>
>>>
>>>
>>>
>>>
>>> ** TEST DONE as root --> NO PROBLEM
>>>
>>>
>>>
>>>
>>>
>>> [2016-11-18 16:07:28.794150] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
>>> 0-home-lab-3-client-2: returning as transport is already disconnected OR
>>> there are no frames (0 || 0)
>>> [2016-11-18 16:07:31.794656] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
>>> 0-home-lab-3-client-0: returning as transport is already disconnected OR
>>> there are no frames (0 || 0)
>>> [2016-11-18 16:07:31.794925] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
>>> 0-home-lab-3-client-1: returning as transport is already disconnected OR
>>> there are no frames (0 || 0)
>>> [2016-11-18 16:07:31.794984] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
>>> 0-home-lab-3-client-3: returning as transport is already disconnected OR
>>> there are no frames (0 || 0)
>>>
>>>
>>> ** USER ROOT EXITED at this point
>>>
>>> El 18/11/16 a las 15:16, Miklos Szeredi escribió:
>>>> On Wed, Nov 16, 2016 at 7:19 PM, Omar Walid Llorente <[email protected]>
>>>> wrote:
>>>>
>>>>> In any case, the kernel we are using (3.13.0-96-generic #143-Ubuntu
>>>>> SMP Mon
>>>>> Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux) has fuse
>>>>> included
>>>>> (not as a module) so if it would be necessary to change the fuse mount
>>>>> options some help would be needed.
>>>> Can you reproduce this on latest (e.g 4.8) kernel?
>>>>
>>>> Also fuse debug logs (with whaterver kernel) could be useful; not sure
>>>> how to enable it for glusterfs, usual convention for fuse filesystems
>>>> is "-d".
>>>>
>>>> Thanks,
>>>> Miklos
>>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>

2016-11-28 18:04:09

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

Hi Soumya, all.

Sorry for the delay answering.

Following your sugestion, we are planning to file a bug at bugzilla [1],
but in order to minimize the content of the explanation, we wanted to
include a link to the file archive of this nfs-kernel-server list, but
we only find a page that doesn't work (from
http://dir.gmane.org/gmane.linux.nfs we go to
http://news.gmane.org/gmane.linux.nfs, but it gives a "Page Not Found"
answer).

Does anybody know if is this link available anywhere else or if it can
be repaired?

Thanks a lot to everyone. We hope we can file the bug this week.

Omar

El 22/11/16 a las 15:45, Soumya Koduri escribió:
>
>
> On 11/21/2016 08:27 PM, J. Bruce Fields wrote:
>> On Mon, Nov 21, 2016 at 06:26:16PM +0530, Soumya Koduri wrote:
>>> Hi,
>>>
>>> Sorry for the delay. I could reproduce the issue on my gluster setup
>>> as well. The error is generated at posix-acl xlator layer of
>>> glusterfs server.
>>>
>>> As Bruce mentioned in the earlier mail, as part of WRITE call,
>>> NFS-server issues OPEN() with O_WRONLY flag set. Since the
>>> attributes of the file "kk.txt" when created was 0400, the
>>> corresponding acl maps to below ACL -
>>>
>>> [root@dhcp35-197 vol1]# getfacl kk.txt
>>> # file: kk.txt
>>> # owner: guest
>>> # group: mock
>>> user::r--
>>> group::---
>>> other::---
>>>
>>> And hence the permission check for WRITE access fails. Also not just
>>> glusterfs, I see similar behavior with ext4 filesystem as well when
>>> exported via kernel-NFS server.
>>
>> That definitely shouldn't be happening, how are you reproducing?
>
> Sorry. Yesterday when I was testing, both the exports (one on
> glusterfs fuse mount and another on ext4) had same fsid. So when I was
> trying to mount and access the share on ext4, it had redirected to
> glusterfs fuse mount and hence the test failed.
>
> I re-checked with ext4 and I do not see any failures. So probably then
> we need to put a fix in glusterfs to bypass this check for file owner
> during OPEN.
>
> @Omar,
> Could you please file a bug against glusterfs component in RH bugzilla
> [1] for the same?
>
> Thanks,
> Soumya
>
> [1] https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
>
>>
>> --b.
>>
>>> So I am not sure if fixing glusterfs
>>> for this use-case is valid.
>>>
>>> Thanks,
>>> Soumya
>>>
>>>
>>> On 11/18/2016 09:33 PM, Omar Walid Llorente wrote:
>>>> Yes, of course, we can reproduce the problem on latest (e.g 4.8)
>>>> kernel.
>>>>
>>>> Tested on a server with fedora 24 (Linux cuentas11.lab.dit.upm.es
>>>> 4.8.4-200.fc24.x86_64 #1 SMP Tue Oct 25 13:06:04 UTC 2016 x86_64
>>>> x86_64
>>>> x86_64 GNU/Linux), glusterfs-fuse-3.8.5-1.fc24.x86_64,
>>>> fuse-2.9.7-1.fc24.x86_64 and nfs-utils-1.3.4-1.rc2.fc24.x86_64 that
>>>> mounts as a glusterfs-client and exports as a nfs-server a ditributed
>>>> glusterfs volume (that comes from a 3.7.4 glusterfs server over Ubuntu
>>>> 14.04.
>>>>
>>>> [root@cuentas11 ~]# cat /proc/mounts | grep home
>>>> recipiente8hd:/home-lab-3.tcp /home-3 fuse.glusterfs
>>>> rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072
>>>>
>>>> 0 0
>>>> [root@cuentas11 ~]#
>>>>
>>>> As you may remember the schema is the one that follows (glusterfs
>>>> volume
>>>> name: home-lab-3, glusterfs client mount point: /home-3, nfs client
>>>> mount point: /home):
>>>>
>>>> nfs client (v1.2.8) ---> NFSv3 ---> knfsd (v1.3.4 over a Linux v4.8.4
>>>> kernel) ---> FUSE (v2.9.7) --> glusterfs client (v3.8.5) --> GLUSTER
>>>> over TCP --> glusterfs server (v3.7.4)
>>>>
>>>> The errors at the nfs-client side are the same:
>>>>
>>>> cdc@l056:~$ mount | grep "/home/cdc"
>>>> cuentas11:/home-3/cdc on /home/cdc type nfs
>>>> (rw,noatime,nfsvers=3,vers=3,tcp,intr,fsc,nolock,rsize=262140,wsize=262140,addr=138.4.30.86)
>>>>
>>>>
>>>> cdc@l056:~$
>>>> cdc@l056:~$
>>>> cdc@l056:~$
>>>> cdc@l056:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444
>>>> 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
>>>> cp: failed to close ‘kk.txt’: Permission denied
>>>> -r--r--r-- 1 cdc admincdc 7 nov 18 2016 444.txt
>>>> -r--r--r-- 1 cdc admincdc 0 nov 18 2016 kk.txt
>>>> cdc@l056:~$
>>>> cdc@l056:~$
>>>> cdc@l056:~$
>>>> cdc@l056:~$ sudo su
>>>> [sudo] password for cdc:
>>>> root@l056:/home/cdc# rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
>>>> chmod 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
>>>> -r--r--r-- 1 root root 7 nov 18 2016 444.txt
>>>> -r--r--r-- 1 root root 7 nov 18 2016 kk.txt
>>>> root@l056:/home/cdc#
>>>> root@l056:/home/cdc#
>>>> root@l056:/home/cdc# exit
>>>> cdc@l056:~$
>>>>
>>>> See at the end the logs with and without the debug option activated at
>>>> the glusterfs client.
>>>>
>>>> Thanks again, and sorry for the long message.
>>>>
>>>> Omar
>>>>
>>>>
>>>>
>>>> * Logs on the glusterfs client side with NO DEBUG:
>>>>
>>>> [2016-11-18 15:45:43.681215] E [MSGID: 114031]
>>>> [client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-3:
>>>> remote
>>>> operation failed. Path: /cdc/kk.txt
>>>> (cc98e831-3410-4202-8254-2d4b48e8ea68) [Permission denied]
>>>> [2016-11-18 15:45:43.681297] W [fuse-bridge.c:989:fuse_fd_cbk]
>>>> 0-glusterfs-fuse: 399: OPEN() /cdc/kk.txt => -1 (Permission denied)
>>>> [2016-11-18 15:45:55.457922] I [io-stats.c:1574:io_stats_dump_fd]
>>>> 0-home-lab-3: --- fd stats ---
>>>> [2016-11-18 15:45:55.457978] I [io-stats.c:1579:io_stats_dump_fd]
>>>> 0-home-lab-3: Filename : /cdc/444.txt
>>>> [2016-11-18 15:45:55.457997] I [io-stats.c:1594:io_stats_dump_fd]
>>>> 0-home-lab-3: BytesWritten : 7 bytes
>>>> [2016-11-18 15:45:55.458011] I [io-stats.c:1606:io_stats_dump_fd]
>>>> 0-home-lab-3: Write 000004b+ : 1
>>>> [2016-11-18 15:45:55.471933] I [io-stats.c:1574:io_stats_dump_fd]
>>>> 0-home-lab-3: --- fd stats ---
>>>>
>>>>
>>>> * Logs on the glusterfs client side with DEBUG activated:
>>>>
>>>> [2016-11-18 16:06:17.040092] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT
>>>> scheduled
>>>> as fast fop
>>>> [2016-11-18 16:06:17.045986] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>>> scheduled as fast fop
>>>> [2016-11-18 16:06:17.046132] D [MSGID: 0]
>>>> [dht-common.c:2684:dht_lookup]
>>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
>>>> home-lab-3-client-3
>>>> [2016-11-18 16:06:17.046833] D [MSGID: 0]
>>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>>> returned for /cdc/kk.txt with op_ret 0 [Invalid argument]
>>>> [2016-11-18 16:06:17.047671] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK
>>>> scheduled as normal fop
>>>> [2016-11-18 16:06:17.050539] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>> event 19 on graph -1418558531
>>>> [2016-11-18 16:06:17.050867] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>> event 19 on graph 0
>>>> [2016-11-18 16:06:17.051948] D [logging.c:1953:_gf_msg_internal]
>>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>>> About to flush least recently used log message to disk
>>>> [2016-11-18 16:06:17.051871] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>>> scheduled as fast fop
>>>> [2016-11-18 16:06:17.051946] D [MSGID: 0]
>>>> [dht-common.c:2684:dht_lookup]
>>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
>>>> home-lab-3-client-0
>>>> [2016-11-18 16:06:17.062108] D [MSGID: 0]
>>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>>> returned for /cdc/444.txt with op_ret 0 [Invalid argument]
>>>> [2016-11-18 16:06:17.063692] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>> event 19 on graph -1418558531
>>>> [2016-11-18 16:06:17.064015] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>> event 19 on graph 0
>>>> [2016-11-18 16:06:17.065068] D [logging.c:1953:_gf_msg_internal]
>>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>>> About to flush least recently used log message to disk
>>>> The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
>>>> 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 4 times
>>>> between [2016-11-18 16:06:17.040092] and [2016-11-18 16:06:17.064175]
>>>> [2016-11-18 16:06:17.065066] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>>> scheduled as fast fop
>>>> [2016-11-18 16:06:17.065232] D [MSGID: 0]
>>>> [dht-common.c:2684:dht_lookup]
>>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
>>>> home-lab-3-client-0
>>>> [2016-11-18 16:06:17.065755] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:06:17.065897] D [MSGID: 0]
>>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>>> returned for /cdc/444.txt with op_ret -1 [No such file or directory]
>>>> [2016-11-18 16:06:17.066016] D [MSGID: 0]
>>>> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry
>>>> /cdc/444.txt
>>>> missing on subvol home-lab-3-client-0
>>>> [2016-11-18 16:06:17.066104] D [MSGID: 0]
>>>> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
>>>> lookup call to 4 subvols
>>>> [2016-11-18 16:06:17.066594] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:06:17.066669] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>> home-lab-3-client-0
>>>> [2016-11-18 16:06:17.066771] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:06:17.066898] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>> home-lab-3-client-1
>>>> [2016-11-18 16:06:17.066906] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:06:17.067019] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>> home-lab-3-client-2
>>>> [2016-11-18 16:06:17.066915] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:06:17.067066] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>> home-lab-3-client-3
>>>> [2016-11-18 16:06:17.067107] D [MSGID: 0]
>>>> [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht:
>>>> STATUS:
>>>> hashed_subvol home-lab-3-client-0 cached_subvol null
>>>> [2016-11-18 16:06:17.067151] D [MSGID: 0]
>>>> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
>>>> was no cached file and unlink on hashed is not skipped /cdc/444.txt
>>>> [2016-11-18 16:06:17.067179] D [MSGID: 0]
>>>> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1 error: No
>>>> such
>>>> file or directory [No such file or directory]
>>>> [2016-11-18 16:06:17.067206] D [MSGID: 0]
>>>> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:06:17.067250] D [MSGID: 0]
>>>> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No such
>>>> file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:06:17.067275] D [MSGID: 0]
>>>> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:06:17.067308] D [MSGID: 0]
>>>> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No such
>>>> file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:06:17.067387] D [MSGID: 0]
>>>> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file or
>>>> directory
>>>> [No such file or directory]
>>>> [2016-11-18 16:06:17.068157] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>>> scheduled as fast fop
>>>> [2016-11-18 16:06:17.068281] D [MSGID: 0]
>>>> [dht-common.c:2684:dht_lookup]
>>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
>>>> home-lab-3-client-0
>>>> [2016-11-18 16:06:17.068738] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:06:17.068781] D [MSGID: 0]
>>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>>> returned for /cdc/444.txt with op_ret -1 [No such file or directory]
>>>> [2016-11-18 16:06:17.068805] D [MSGID: 0]
>>>> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry
>>>> /cdc/444.txt
>>>> missing on subvol home-lab-3-client-0
>>>> [2016-11-18 16:06:17.068827] D [MSGID: 0]
>>>> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
>>>> lookup call to 4 subvols
>>>> [2016-11-18 16:06:17.069286] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:06:17.069345] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>> home-lab-3-client-0
>>>> [2016-11-18 16:06:17.069452] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:06:17.069507] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>> home-lab-3-client-2
>>>> [2016-11-18 16:06:17.069556] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:06:17.069585] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>> home-lab-3-client-3
>>>> [2016-11-18 16:06:17.069939] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:06:17.069983] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>> home-lab-3-client-1
>>>> [2016-11-18 16:06:17.070005] D [MSGID: 0]
>>>> [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht:
>>>> STATUS:
>>>> hashed_subvol home-lab-3-client-0 cached_subvol null
>>>> [2016-11-18 16:06:17.070038] D [MSGID: 0]
>>>> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
>>>> was no cached file and unlink on hashed is not skipped /cdc/444.txt
>>>> [2016-11-18 16:06:17.070098] D [MSGID: 0]
>>>> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1 error: No
>>>> such
>>>> file or directory [No such file or directory]
>>>> [2016-11-18 16:06:17.070128] D [MSGID: 0]
>>>> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:06:17.070154] D [MSGID: 0]
>>>> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No such
>>>> file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:06:17.070178] D [MSGID: 0]
>>>> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:06:17.070208] D [MSGID: 0]
>>>> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No such
>>>> file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:06:17.070251] D [MSGID: 0]
>>>> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file or
>>>> directory
>>>> [No such file or directory]
>>>> [2016-11-18 16:06:17.070537] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD
>>>> scheduled
>>>> as normal fop
>>>> [2016-11-18 16:06:17.070826] D [MSGID: 0]
>>>> [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
>>>> creating /cdc/444.txt on home-lab-3-client-0
>>>> [2016-11-18 16:06:17.071100] D [MSGID: 0]
>>>> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
>>>> 'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
>>>> 10456189239296 and avail_inodes is: 99.00
>>>> [2016-11-18 16:06:17.071142] D [MSGID: 0]
>>>> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
>>>> 'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
>>>> 10499693871104 and avail_inodes is: 99.00
>>>> [2016-11-18 16:06:17.071183] D [MSGID: 0]
>>>> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
>>>> 'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
>>>> 10500911267840 and avail_inodes is: 99.00
>>>> [2016-11-18 16:06:17.071343] D [MSGID: 0]
>>>> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
>>>> 'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
>>>> 8600432410624 and avail_inodes is: 99.00
>>>> [2016-11-18 16:06:17.073943] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>> event 19 on graph -1418558531
>>>> [2016-11-18 16:06:17.074273] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT
>>>> scheduled
>>>> as fast fop
>>>> [2016-11-18 16:06:17.075035] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN
>>>> scheduled
>>>> as fast fop
>>>> [2016-11-18 16:06:17.075913] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE
>>>> scheduled
>>>> as slow fop
>>>> [2016-11-18 16:06:17.076205] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT
>>>> scheduled
>>>> as fast fop
>>>> [2016-11-18 16:06:17.076636] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>> event 19 on graph 0
>>>> [2016-11-18 16:06:17.077282] D [fuse-bridge.c:52:fuse_invalidate]
>>>> 0-fuse: Invalidate inode id 140363155120276.
>>>> [2016-11-18 16:06:17.077981] I [io-stats.c:1574:io_stats_dump_fd]
>>>> 0-home-lab-3: --- fd stats ---
>>>> [2016-11-18 16:06:17.078032] I [io-stats.c:1579:io_stats_dump_fd]
>>>> 0-home-lab-3: Filename : /cdc/444.txt
>>>> [2016-11-18 16:06:17.078049] I [io-stats.c:1594:io_stats_dump_fd]
>>>> 0-home-lab-3: BytesWritten : 7 bytes
>>>> [2016-11-18 16:06:17.078064] I [io-stats.c:1606:io_stats_dump_fd]
>>>> 0-home-lab-3: Write 000004b+ : 1
>>>> [2016-11-18 16:06:17.079156] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
>>>> scheduled as normal fop
>>>> [2016-11-18 16:06:17.079794] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>> event 19 on graph 0
>>>> [2016-11-18 16:06:17.082196] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>>> scheduled as fast fop
>>>> [2016-11-18 16:06:17.082772] D [logging.c:1953:_gf_msg_internal]
>>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>>> About to flush least recently used log message to disk
>>>> [2016-11-18 16:06:17.082318] D [MSGID: 0]
>>>> [dht-common.c:2684:dht_lookup]
>>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
>>>> home-lab-3-client-3
>>>> [2016-11-18 16:06:17.082770] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:06:17.083168] D [MSGID: 0]
>>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>>> returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
>>>> [2016-11-18 16:06:17.083282] D [MSGID: 0]
>>>> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
>>>> missing on subvol home-lab-3-client-3
>>>> [2016-11-18 16:06:17.083320] D [MSGID: 0]
>>>> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
>>>> lookup call to 4 subvols
>>>> [2016-11-18 16:06:17.083991] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:06:17.084043] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>> home-lab-3-client-0
>>>> [2016-11-18 16:06:17.084048] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:06:17.084092] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>> home-lab-3-client-3
>>>> [2016-11-18 16:06:17.084114] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:06:17.084167] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>> home-lab-3-client-1
>>>> [2016-11-18 16:06:17.084091] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:06:17.084377] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>> home-lab-3-client-2
>>>> [2016-11-18 16:06:17.084406] D [MSGID: 0]
>>>> [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht:
>>>> STATUS:
>>>> hashed_subvol home-lab-3-client-3 cached_subvol null
>>>> [2016-11-18 16:06:17.084511] D [MSGID: 0]
>>>> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
>>>> was no cached file and unlink on hashed is not skipped /cdc/kk.txt
>>>> [2016-11-18 16:06:17.084534] D [MSGID: 0]
>>>> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No
>>>> such
>>>> file or directory [No such file or directory]
>>>> [2016-11-18 16:06:17.084683] D [MSGID: 0]
>>>> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:06:17.084783] D [MSGID: 0]
>>>> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such
>>>> file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:06:17.084857] D [MSGID: 0]
>>>> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:06:17.084889] D [MSGID: 0]
>>>> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such
>>>> file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:06:17.085001] D [MSGID: 0]
>>>> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or
>>>> directory
>>>> [No such file or directory]
>>>> [2016-11-18 16:06:17.086098] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT
>>>> scheduled
>>>> as fast fop
>>>> [2016-11-18 16:06:17.087256] D [logging.c:1953:_gf_msg_internal]
>>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>>> About to flush least recently used log message to disk
>>>> [2016-11-18 16:06:17.087173] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>>> scheduled as fast fop
>>>> [2016-11-18 16:06:17.087254] D [MSGID: 0]
>>>> [dht-common.c:2684:dht_lookup]
>>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
>>>> home-lab-3-client-3
>>>> [2016-11-18 16:06:17.087749] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:06:17.087795] D [MSGID: 0]
>>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>>> returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
>>>> [2016-11-18 16:06:17.088014] D [MSGID: 0]
>>>> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
>>>> missing on subvol home-lab-3-client-3
>>>> [2016-11-18 16:06:17.088036] D [MSGID: 0]
>>>> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
>>>> lookup call to 4 subvols
>>>> [2016-11-18 16:06:17.088598] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:06:17.088639] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>> home-lab-3-client-0
>>>> [2016-11-18 16:06:17.088804] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:06:17.088855] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>> home-lab-3-client-2
>>>> [2016-11-18 16:06:17.088819] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:06:17.089069] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>> home-lab-3-client-3
>>>> [2016-11-18 16:06:17.089380] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:06:17.089421] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>> home-lab-3-client-1
>>>> [2016-11-18 16:06:17.089443] D [MSGID: 0]
>>>> [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht:
>>>> STATUS:
>>>> hashed_subvol home-lab-3-client-3 cached_subvol null
>>>> [2016-11-18 16:06:17.089463] D [MSGID: 0]
>>>> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
>>>> was no cached file and unlink on hashed is not skipped /cdc/kk.txt
>>>> [2016-11-18 16:06:17.089575] D [MSGID: 0]
>>>> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No
>>>> such
>>>> file or directory [No such file or directory]
>>>> [2016-11-18 16:06:17.089601] D [MSGID: 0]
>>>> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:06:17.089712] D [MSGID: 0]
>>>> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such
>>>> file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:06:17.089738] D [MSGID: 0]
>>>> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:06:17.090006] D [MSGID: 0]
>>>> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such
>>>> file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:06:17.090156] D [MSGID: 0]
>>>> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or
>>>> directory
>>>> [No such file or directory]
>>>> [2016-11-18 16:06:17.090648] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD
>>>> scheduled
>>>> as normal fop
>>>> [2016-11-18 16:06:17.090732] D [MSGID: 0]
>>>> [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
>>>> creating /cdc/kk.txt on home-lab-3-client-3
>>>> [2016-11-18 16:06:17.091499] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>> event 19 on graph -1418558531
>>>> [2016-11-18 16:06:17.091843] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
>>>> scheduled as normal fop
>>>> [2016-11-18 16:06:17.093831] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>> event 19 on graph 0
>>>> [2016-11-18 16:06:17.094711] D [logging.c:1953:_gf_msg_internal]
>>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>>> About to flush least recently used log message to disk
>>>> [2016-11-18 16:06:17.093216] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
>>>> scheduled as normal fop
>>>> [2016-11-18 16:06:17.094709] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN
>>>> scheduled
>>>> as fast fop
>>>> [2016-11-18 16:06:17.095280] E [MSGID: 114031]
>>>> [client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-3:
>>>> remote
>>>> operation failed. Path: /cdc/kk.txt
>>>> (09bfb6fa-82fb-463b-abd9-bb7ef88e10c0) [Permission denied]
>>>> [2016-11-18 16:06:17.095341] D [MSGID: 0]
>>>> [client-rpc-fops.c:448:client3_3_open_cbk] 0-stack-trace:
>>>> stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: Permission
>>>> denied
>>>> [Permission denied]
>>>> [2016-11-18 16:06:17.095366] D [MSGID: 0]
>>>> [dht-inode-read.c:45:dht_open_cbk] 0-home-lab-3-dht: subvolume
>>>> home-lab-3-client-3 returned -1 [Permission denied]
>>>> [2016-11-18 16:06:17.095480] D [MSGID: 0]
>>>> [dht-inode-read.c:59:dht_open_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: Permission denied
>>>> [Permission denied]
>>>> [2016-11-18 16:06:17.095671] D [MSGID: 0]
>>>> [read-ahead.c:101:ra_open_cbk]
>>>> 0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-read-ahead
>>>> returned -1 error: Permission denied [Permission denied]
>>>> [2016-11-18 16:06:17.095759] D [MSGID: 0]
>>>> [io-cache.c:605:ioc_open_cbk]
>>>> 0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-io-cache
>>>> returned -1 error: Permission denied [Permission denied]
>>>> [2016-11-18 16:06:17.095963] D [MSGID: 0]
>>>> [defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: Permission
>>>> denied [Permission denied]
>>>> [2016-11-18 16:06:17.095988] D [MSGID: 0]
>>>> [defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: Permission
>>>> denied [Permission denied]
>>>> [2016-11-18 16:06:17.096275] D [MSGID: 0]
>>>> [io-stats.c:1936:io_stats_open_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3 returned -1 error: Permission denied
>>>> [Permission denied]
>>>> [2016-11-18 16:06:17.096362] W [fuse-bridge.c:989:fuse_fd_cbk]
>>>> 0-glusterfs-fuse: 338: OPEN() /cdc/kk.txt => -1 (Permission denied)
>>>> [2016-11-18 16:06:17.097104] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
>>>> scheduled as normal fop
>>>> [2016-11-18 16:06:17.097674] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>> event 19 on graph 0
>>>> [2016-11-18 16:06:17.099912] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>> event 19 on graph 0
>>>> [2016-11-18 16:06:24.785225] D
>>>> [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
>>>> 0-home-lab-3-client-0: returning as transport is already
>>>> disconnected OR
>>>> there are no frames (0 || 0)
>>>> [2016-11-18 16:06:24.785338] D
>>>> [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
>>>> 0-home-lab-3-client-1: returning as transport is already
>>>> disconnected OR
>>>> there are no frames (0 || 0)
>>>> [2016-11-18 16:06:24.785359] D
>>>> [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
>>>> 0-home-lab-3-client-2: returning as transport is already
>>>> disconnected OR
>>>> there are no frames (0 || 0)
>>>> [2016-11-18 16:06:24.785377] D
>>>> [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
>>>> 0-home-lab-3-client-3: returning as transport is already
>>>> disconnected OR
>>>> there are no frames (0 || 0)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ** TEST DONE as user cdc --> ERROR
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> [2016-11-18 16:06:46.737686] D
>>>> [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
>>>> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
>>>> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
>>>>
>>>> (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
>>>> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
>>>> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
>>>>
>>>> ))))) 0-: 138.4.2.246:49172: ping timer event already removed
>>>> [2016-11-18 16:06:49.545698] D
>>>> [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
>>>> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
>>>> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
>>>>
>>>> (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
>>>> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
>>>> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
>>>>
>>>> ))))) 0-: 138.4.2.177:49157: ping timer event already removed
>>>> [2016-11-18 16:06:49.546474] D
>>>> [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
>>>> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
>>>> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
>>>>
>>>> (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
>>>> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
>>>> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
>>>>
>>>> ))))) 0-: 138.4.2.189:49157: ping timer event already removed
>>>> [2016-11-18 16:06:49.547093] D
>>>> [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
>>>> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
>>>> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
>>>>
>>>> (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
>>>> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
>>>> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
>>>>
>>>> ))))) 0-: 138.4.2.248:49157: ping timer event already removed
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ** user CDC becomed root at this point
>>>>
>>>>
>>>>
>>>>
>>>> [2016-11-18 16:07:11.907892] D [logging.c:1953:_gf_msg_internal]
>>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>>> About to flush least recently used log message to disk
>>>> The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
>>>> 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 7 times
>>>> between [2016-11-18 16:06:17.086098] and [2016-11-18 16:07:11.906944]
>>>> [2016-11-18 16:07:11.907884] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>>> scheduled as fast fop
>>>> [2016-11-18 16:07:11.908875] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>> event 19 on graph -1418558531
>>>> [2016-11-18 16:07:11.909179] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>> event 19 on graph 0
>>>> [2016-11-18 16:07:11.909256] D [logging.c:1953:_gf_msg_internal]
>>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>>> About to flush least recently used log message to disk
>>>> The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
>>>> 0-home-lab-3-io-threads: UNLINK scheduled as normal fop" repeated 2
>>>> times between [2016-11-18 16:06:17.047671] and [2016-11-18
>>>> 16:07:11.908114]
>>>> [2016-11-18 16:07:11.909254] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT
>>>> scheduled
>>>> as fast fop
>>>> [2016-11-18 16:07:11.911109] D [logging.c:1953:_gf_msg_internal]
>>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>>> About to flush least recently used log message to disk
>>>> [2016-11-18 16:07:11.911018] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>>> scheduled as fast fop
>>>> [2016-11-18 16:07:11.911107] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK
>>>> scheduled as normal fop
>>>> [2016-11-18 16:07:11.911871] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>> event 19 on graph -1418558531
>>>> [2016-11-18 16:07:11.912045] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>> event 19 on graph 0
>>>> [2016-11-18 16:07:11.913067] D [logging.c:1953:_gf_msg_internal]
>>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>>> About to flush least recently used log message to disk
>>>> The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
>>>> 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 2 times
>>>> between [2016-11-18 16:07:11.909254] and [2016-11-18 16:07:11.912165]
>>>> [2016-11-18 16:07:11.913065] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>>> scheduled as fast fop
>>>> [2016-11-18 16:07:11.913193] D [MSGID: 0]
>>>> [dht-common.c:2684:dht_lookup]
>>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
>>>> home-lab-3-client-0
>>>> [2016-11-18 16:07:11.913710] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:07:11.913757] D [MSGID: 0]
>>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>>> returned for /cdc/444.txt with op_ret -1 [No such file or directory]
>>>> [2016-11-18 16:07:11.913788] D [MSGID: 0]
>>>> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry
>>>> /cdc/444.txt
>>>> missing on subvol home-lab-3-client-0
>>>> [2016-11-18 16:07:11.913810] D [MSGID: 0]
>>>> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
>>>> lookup call to 4 subvols
>>>> [2016-11-18 16:07:11.914336] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:07:11.914379] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>> home-lab-3-client-0
>>>> [2016-11-18 16:07:11.914424] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:07:11.914448] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>> home-lab-3-client-1
>>>> [2016-11-18 16:07:11.914501] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:07:11.914617] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>> home-lab-3-client-3
>>>> [2016-11-18 16:07:11.914649] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:07:11.914698] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>> home-lab-3-client-2
>>>> [2016-11-18 16:07:11.914724] D [MSGID: 0]
>>>> [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht:
>>>> STATUS:
>>>> hashed_subvol home-lab-3-client-0 cached_subvol null
>>>> [2016-11-18 16:07:11.914743] D [MSGID: 0]
>>>> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
>>>> was no cached file and unlink on hashed is not skipped /cdc/444.txt
>>>> [2016-11-18 16:07:11.914764] D [MSGID: 0]
>>>> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No
>>>> such
>>>> file or directory [No such file or directory]
>>>> [2016-11-18 16:07:11.914791] D [MSGID: 0]
>>>> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:07:11.914825] D [MSGID: 0]
>>>> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such
>>>> file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:07:11.914855] D [MSGID: 0]
>>>> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:07:11.914880] D [MSGID: 0]
>>>> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such
>>>> file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:07:11.914947] D [MSGID: 0]
>>>> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or
>>>> directory
>>>> [No such file or directory]
>>>> [2016-11-18 16:07:11.915569] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>>> scheduled as fast fop
>>>> [2016-11-18 16:07:11.915644] D [MSGID: 0]
>>>> [dht-common.c:2684:dht_lookup]
>>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
>>>> home-lab-3-client-0
>>>> [2016-11-18 16:07:11.916090] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:07:11.916134] D [MSGID: 0]
>>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>>> returned for /cdc/444.txt with op_ret -1 [No such file or directory]
>>>> [2016-11-18 16:07:11.916158] D [MSGID: 0]
>>>> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry
>>>> /cdc/444.txt
>>>> missing on subvol home-lab-3-client-0
>>>> [2016-11-18 16:07:11.916185] D [MSGID: 0]
>>>> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
>>>> lookup call to 4 subvols
>>>> [2016-11-18 16:07:11.916593] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:07:11.916780] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>> home-lab-3-client-0
>>>> [2016-11-18 16:07:11.916878] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:07:11.916994] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>> home-lab-3-client-3
>>>> [2016-11-18 16:07:11.917076] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:07:11.916906] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:07:11.917117] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>> home-lab-3-client-2
>>>> [2016-11-18 16:07:11.917153] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>> home-lab-3-client-1
>>>> [2016-11-18 16:07:11.917205] D [MSGID: 0]
>>>> [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht:
>>>> STATUS:
>>>> hashed_subvol home-lab-3-client-0 cached_subvol null
>>>> [2016-11-18 16:07:11.917368] D [MSGID: 0]
>>>> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
>>>> was no cached file and unlink on hashed is not skipped /cdc/444.txt
>>>> [2016-11-18 16:07:11.917437] D [MSGID: 0]
>>>> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No
>>>> such
>>>> file or directory [No such file or directory]
>>>> [2016-11-18 16:07:11.917467] D [MSGID: 0]
>>>> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:07:11.917496] D [MSGID: 0]
>>>> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such
>>>> file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:07:11.917521] D [MSGID: 0]
>>>> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:07:11.917551] D [MSGID: 0]
>>>> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such
>>>> file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:07:11.917726] D [MSGID: 0]
>>>> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or
>>>> directory
>>>> [No such file or directory]
>>>> [2016-11-18 16:07:11.917958] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD
>>>> scheduled
>>>> as normal fop
>>>> [2016-11-18 16:07:11.918230] D [MSGID: 0]
>>>> [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
>>>> creating /cdc/444.txt on home-lab-3-client-0
>>>> [2016-11-18 16:07:11.918488] D [MSGID: 0]
>>>> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
>>>> 'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
>>>> 10456166432768 and avail_inodes is: 99.00
>>>> [2016-11-18 16:07:11.918570] D [MSGID: 0]
>>>> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
>>>> 'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
>>>> 8600417075200 and avail_inodes is: 99.00
>>>> [2016-11-18 16:07:11.918603] D [MSGID: 0]
>>>> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
>>>> 'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
>>>> 10500905500672 and avail_inodes is: 99.00
>>>> [2016-11-18 16:07:11.918675] D [MSGID: 0]
>>>> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
>>>> 'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
>>>> 10499686531072 and avail_inodes is: 99.00
>>>> [2016-11-18 16:07:11.919095] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>> event 19 on graph -1418558531
>>>> [2016-11-18 16:07:11.919377] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT
>>>> scheduled
>>>> as fast fop
>>>> [2016-11-18 16:07:11.920040] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN
>>>> scheduled
>>>> as fast fop
>>>> [2016-11-18 16:07:11.920889] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE
>>>> scheduled
>>>> as slow fop
>>>> [2016-11-18 16:07:11.921051] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT
>>>> scheduled
>>>> as fast fop
>>>> [2016-11-18 16:07:11.921583] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>> event 19 on graph 0
>>>> [2016-11-18 16:07:11.922068] D [fuse-bridge.c:52:fuse_invalidate]
>>>> 0-fuse: Invalidate inode id 140363155120276.
>>>> [2016-11-18 16:07:11.922936] I [io-stats.c:1574:io_stats_dump_fd]
>>>> 0-home-lab-3: --- fd stats ---
>>>> [2016-11-18 16:07:11.922988] I [io-stats.c:1579:io_stats_dump_fd]
>>>> 0-home-lab-3: Filename : /cdc/444.txt
>>>> [2016-11-18 16:07:11.923004] I [io-stats.c:1594:io_stats_dump_fd]
>>>> 0-home-lab-3: BytesWritten : 7 bytes
>>>> [2016-11-18 16:07:11.923026] I [io-stats.c:1606:io_stats_dump_fd]
>>>> 0-home-lab-3: Write 000004b+ : 1
>>>> [2016-11-18 16:07:11.923779] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT
>>>> scheduled
>>>> as fast fop
>>>> [2016-11-18 16:07:11.924504] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>> event 19 on graph 0
>>>> [2016-11-18 16:07:11.926692] D [logging.c:1953:_gf_msg_internal]
>>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>>> About to flush least recently used log message to disk
>>>> The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
>>>> 0-home-lab-3-io-threads: SETATTR scheduled as normal fop" repeated 2
>>>> times between [2016-11-18 16:06:17.097104] and [2016-11-18
>>>> 16:07:11.923946]
>>>> [2016-11-18 16:07:11.926690] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>>> scheduled as fast fop
>>>> [2016-11-18 16:07:11.926858] D [MSGID: 0]
>>>> [dht-common.c:2684:dht_lookup]
>>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
>>>> home-lab-3-client-3
>>>> [2016-11-18 16:07:11.927327] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:07:11.927376] D [MSGID: 0]
>>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>>> returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
>>>> [2016-11-18 16:07:11.927401] D [MSGID: 0]
>>>> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
>>>> missing on subvol home-lab-3-client-3
>>>> [2016-11-18 16:07:11.927422] D [MSGID: 0]
>>>> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
>>>> lookup call to 4 subvols
>>>> [2016-11-18 16:07:11.927940] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:07:11.927987] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>> home-lab-3-client-0
>>>> [2016-11-18 16:07:11.928043] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:07:11.928055] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:07:11.928058] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:07:11.928170] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>> home-lab-3-client-1
>>>> [2016-11-18 16:07:11.928174] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>> home-lab-3-client-3
>>>> [2016-11-18 16:07:11.928277] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>> home-lab-3-client-2
>>>> [2016-11-18 16:07:11.928360] D [MSGID: 0]
>>>> [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht:
>>>> STATUS:
>>>> hashed_subvol home-lab-3-client-3 cached_subvol null
>>>> [2016-11-18 16:07:11.928381] D [MSGID: 0]
>>>> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
>>>> was no cached file and unlink on hashed is not skipped /cdc/kk.txt
>>>> [2016-11-18 16:07:11.928403] D [MSGID: 0]
>>>> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No
>>>> such
>>>> file or directory [No such file or directory]
>>>> [2016-11-18 16:07:11.928431] D [MSGID: 0]
>>>> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:07:11.928456] D [MSGID: 0]
>>>> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such
>>>> file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:07:11.928480] D [MSGID: 0]
>>>> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:07:11.928511] D [MSGID: 0]
>>>> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such
>>>> file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:07:11.928753] D [MSGID: 0]
>>>> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or
>>>> directory
>>>> [No such file or directory]
>>>> [2016-11-18 16:07:11.930485] D [logging.c:1953:_gf_msg_internal]
>>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>>> About to flush least recently used log message to disk
>>>> [2016-11-18 16:07:11.929862] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT
>>>> scheduled
>>>> as fast fop
>>>> [2016-11-18 16:07:11.930483] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>>> scheduled as fast fop
>>>> [2016-11-18 16:07:11.930602] D [MSGID: 0]
>>>> [dht-common.c:2684:dht_lookup]
>>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
>>>> home-lab-3-client-3
>>>> [2016-11-18 16:07:11.931071] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:07:11.931124] D [MSGID: 0]
>>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>>> returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
>>>> [2016-11-18 16:07:11.931148] D [MSGID: 0]
>>>> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
>>>> missing on subvol home-lab-3-client-3
>>>> [2016-11-18 16:07:11.931348] D [MSGID: 0]
>>>> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
>>>> lookup call to 4 subvols
>>>> [2016-11-18 16:07:11.931880] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:07:11.931927] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>> home-lab-3-client-0
>>>> [2016-11-18 16:07:11.932002] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:07:11.931994] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:07:11.931991] D [MSGID: 0]
>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1
>>>> error: No
>>>> such file or directory [No such file or directory]
>>>> [2016-11-18 16:07:11.932078] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>> home-lab-3-client-2
>>>> [2016-11-18 16:07:11.932116] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>> home-lab-3-client-1
>>>> [2016-11-18 16:07:11.932315] D [MSGID: 0]
>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht:
>>>> returned
>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>> home-lab-3-client-3
>>>> [2016-11-18 16:07:11.932363] D [MSGID: 0]
>>>> [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht:
>>>> STATUS:
>>>> hashed_subvol home-lab-3-client-3 cached_subvol null
>>>> [2016-11-18 16:07:11.932383] D [MSGID: 0]
>>>> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
>>>> was no cached file and unlink on hashed is not skipped /cdc/kk.txt
>>>> [2016-11-18 16:07:11.932407] D [MSGID: 0]
>>>> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No
>>>> such
>>>> file or directory [No such file or directory]
>>>> [2016-11-18 16:07:11.932446] D [MSGID: 0]
>>>> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:07:11.932471] D [MSGID: 0]
>>>> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such
>>>> file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:07:11.932494] D [MSGID: 0]
>>>> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:07:11.932518] D [MSGID: 0]
>>>> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such
>>>> file or
>>>> directory [No such file or directory]
>>>> [2016-11-18 16:07:11.932725] D [logging.c:1953:_gf_msg_internal]
>>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>>> About to flush least recently used log message to disk
>>>> [2016-11-18 16:07:11.932545] D [MSGID: 0]
>>>> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
>>>> 0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or
>>>> directory
>>>> [No such file or directory]
>>>> [2016-11-18 16:07:11.932722] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD
>>>> scheduled
>>>> as normal fop
>>>> [2016-11-18 16:07:11.933008] D [MSGID: 0]
>>>> [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
>>>> creating /cdc/kk.txt on home-lab-3-client-3
>>>> [2016-11-18 16:07:11.934086] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>> event 19 on graph -1418558531
>>>> [2016-11-18 16:07:11.934323] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
>>>> scheduled as normal fop
>>>> [2016-11-18 16:07:11.934922] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT
>>>> scheduled
>>>> as fast fop
>>>> [2016-11-18 16:07:11.935497] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
>>>> scheduled as normal fop
>>>> [2016-11-18 16:07:11.936102] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>> event 19 on graph 0
>>>> [2016-11-18 16:07:11.936903] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN
>>>> scheduled
>>>> as fast fop
>>>> [2016-11-18 16:07:11.937540] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE
>>>> scheduled
>>>> as slow fop
>>>> [2016-11-18 16:07:11.937682] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT
>>>> scheduled
>>>> as fast fop
>>>> [2016-11-18 16:07:11.938267] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>> event 19 on graph 0
>>>> [2016-11-18 16:07:11.938757] D [fuse-bridge.c:52:fuse_invalidate]
>>>> 0-fuse: Invalidate inode id 140363155120652.
>>>> [2016-11-18 16:07:11.938994] I [io-stats.c:1574:io_stats_dump_fd]
>>>> 0-home-lab-3: --- fd stats ---
>>>> [2016-11-18 16:07:11.939072] I [io-stats.c:1579:io_stats_dump_fd]
>>>> 0-home-lab-3: Filename : /cdc/kk.txt
>>>> [2016-11-18 16:07:11.939096] I [io-stats.c:1594:io_stats_dump_fd]
>>>> 0-home-lab-3: BytesWritten : 7 bytes
>>>> [2016-11-18 16:07:11.939112] I [io-stats.c:1606:io_stats_dump_fd]
>>>> 0-home-lab-3: Write 000004b+ : 1
>>>> [2016-11-18 16:07:11.939448] D [MSGID: 0]
>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
>>>> scheduled as normal fop
>>>> [2016-11-18 16:07:11.940028] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>> event 19 on graph 0
>>>> [2016-11-18 16:07:11.942130] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>> event 19 on graph 0
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ** TEST DONE as root --> NO PROBLEM
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> [2016-11-18 16:07:28.794150] D
>>>> [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
>>>> 0-home-lab-3-client-2: returning as transport is already
>>>> disconnected OR
>>>> there are no frames (0 || 0)
>>>> [2016-11-18 16:07:31.794656] D
>>>> [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
>>>> 0-home-lab-3-client-0: returning as transport is already
>>>> disconnected OR
>>>> there are no frames (0 || 0)
>>>> [2016-11-18 16:07:31.794925] D
>>>> [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
>>>> 0-home-lab-3-client-1: returning as transport is already
>>>> disconnected OR
>>>> there are no frames (0 || 0)
>>>> [2016-11-18 16:07:31.794984] D
>>>> [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
>>>> 0-home-lab-3-client-3: returning as transport is already
>>>> disconnected OR
>>>> there are no frames (0 || 0)
>>>>
>>>>
>>>> ** USER ROOT EXITED at this point
>>>>
>>>> El 18/11/16 a las 15:16, Miklos Szeredi escribió:
>>>>> On Wed, Nov 16, 2016 at 7:19 PM, Omar Walid Llorente
>>>>> <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> In any case, the kernel we are using (3.13.0-96-generic #143-Ubuntu
>>>>>> SMP Mon
>>>>>> Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux) has fuse
>>>>>> included
>>>>>> (not as a module) so if it would be necessary to change the fuse
>>>>>> mount
>>>>>> options some help would be needed.
>>>>> Can you reproduce this on latest (e.g 4.8) kernel?
>>>>>
>>>>> Also fuse debug logs (with whaterver kernel) could be useful; not
>>>>> sure
>>>>> how to enable it for glusterfs, usual convention for fuse filesystems
>>>>> is "-d".
>>>>>
>>>>> Thanks,
>>>>> Miklos
>>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to [email protected]
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>

--
----------------------------------------------------------------
Centro de Cálculo Depto. Ingeniería Sistemas Telemáticos
E-mail: [email protected] Universidad Politécnica de Madrid
Fax:(+34) 913367333 E.T.S. Ing. Telecomunicación
Tel:(+34) 915495700-Ext.3005 28040 Madrid (Spain)
Tel:(+34) 915495762-Ext.3005
Tel:(+34) 913367366-Ext.3005
----------------------------------------------------------------

2016-11-28 18:25:07

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

On Mon, Nov 28, 2016 at 07:03:48PM +0100, Omar Walid Llorente wrote:
>
> Hi Soumya, all.
>
> Sorry for the delay answering.
>
> Following your sugestion, we are planning to file a bug at bugzilla
> [1], but in order to minimize the content of the explanation, we
> wanted to include a link to the file archive of this
> nfs-kernel-server list, but we only find a page that doesn't work
> (from http://dir.gmane.org/gmane.linux.nfs we go to
> http://news.gmane.org/gmane.linux.nfs, but it gives a "Page Not
> Found" answer).
>
> Does anybody know if is this link available anywhere else or if it
> can be repaired?
>
> Thanks a lot to everyone. We hope we can file the bug this week.

http://marc.info/?t=144801803400001&r=1&w=2

Or you can link by message id like:

http://marc.info/[email protected]

which I sometimes try to for posterity's sake since the message id
should still be useful even if the one archive dies.

--b.

>
> Omar
>
> El 22/11/16 a las 15:45, Soumya Koduri escribió:
> >
> >
> >On 11/21/2016 08:27 PM, J. Bruce Fields wrote:
> >>On Mon, Nov 21, 2016 at 06:26:16PM +0530, Soumya Koduri wrote:
> >>>Hi,
> >>>
> >>>Sorry for the delay. I could reproduce the issue on my gluster setup
> >>>as well. The error is generated at posix-acl xlator layer of
> >>>glusterfs server.
> >>>
> >>>As Bruce mentioned in the earlier mail, as part of WRITE call,
> >>>NFS-server issues OPEN() with O_WRONLY flag set. Since the
> >>>attributes of the file "kk.txt" when created was 0400, the
> >>>corresponding acl maps to below ACL -
> >>>
> >>>[root@dhcp35-197 vol1]# getfacl kk.txt
> >>># file: kk.txt
> >>># owner: guest
> >>># group: mock
> >>>user::r--
> >>>group::---
> >>>other::---
> >>>
> >>>And hence the permission check for WRITE access fails. Also not just
> >>>glusterfs, I see similar behavior with ext4 filesystem as well when
> >>>exported via kernel-NFS server.
> >>
> >>That definitely shouldn't be happening, how are you reproducing?
> >
> >Sorry. Yesterday when I was testing, both the exports (one on
> >glusterfs fuse mount and another on ext4) had same fsid. So when I
> >was trying to mount and access the share on ext4, it had
> >redirected to glusterfs fuse mount and hence the test failed.
> >
> >I re-checked with ext4 and I do not see any failures. So probably
> >then we need to put a fix in glusterfs to bypass this check for
> >file owner during OPEN.
> >
> >@Omar,
> >Could you please file a bug against glusterfs component in RH
> >bugzilla [1] for the same?
> >
> >Thanks,
> >Soumya
> >
> >[1] https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
> >
> >>
> >>--b.
> >>
> >>>So I am not sure if fixing glusterfs
> >>>for this use-case is valid.
> >>>
> >>>Thanks,
> >>>Soumya
> >>>
> >>>
> >>>On 11/18/2016 09:33 PM, Omar Walid Llorente wrote:
> >>>>Yes, of course, we can reproduce the problem on latest (e.g
> >>>>4.8) kernel.
> >>>>
> >>>>Tested on a server with fedora 24 (Linux cuentas11.lab.dit.upm.es
> >>>>4.8.4-200.fc24.x86_64 #1 SMP Tue Oct 25 13:06:04 UTC 2016
> >>>>x86_64 x86_64
> >>>>x86_64 GNU/Linux), glusterfs-fuse-3.8.5-1.fc24.x86_64,
> >>>>fuse-2.9.7-1.fc24.x86_64 and nfs-utils-1.3.4-1.rc2.fc24.x86_64 that
> >>>>mounts as a glusterfs-client and exports as a nfs-server a ditributed
> >>>>glusterfs volume (that comes from a 3.7.4 glusterfs server over Ubuntu
> >>>>14.04.
> >>>>
> >>>>[root@cuentas11 ~]# cat /proc/mounts | grep home
> >>>>recipiente8hd:/home-lab-3.tcp /home-3 fuse.glusterfs
> >>>>rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072
> >>>>
> >>>>0 0
> >>>>[root@cuentas11 ~]#
> >>>>
> >>>>As you may remember the schema is the one that follows
> >>>>(glusterfs volume
> >>>>name: home-lab-3, glusterfs client mount point: /home-3, nfs client
> >>>>mount point: /home):
> >>>>
> >>>>nfs client (v1.2.8) ---> NFSv3 ---> knfsd (v1.3.4 over a Linux v4.8.4
> >>>>kernel) ---> FUSE (v2.9.7) --> glusterfs client (v3.8.5) --> GLUSTER
> >>>>over TCP --> glusterfs server (v3.7.4)
> >>>>
> >>>>The errors at the nfs-client side are the same:
> >>>>
> >>>>cdc@l056:~$ mount | grep "/home/cdc"
> >>>>cuentas11:/home-3/cdc on /home/cdc type nfs
> >>>>(rw,noatime,nfsvers=3,vers=3,tcp,intr,fsc,nolock,rsize=262140,wsize=262140,addr=138.4.30.86)
> >>>>
> >>>>
> >>>>cdc@l056:~$
> >>>>cdc@l056:~$
> >>>>cdc@l056:~$
> >>>>cdc@l056:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444
> >>>>444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
> >>>>cp: failed to close ‘kk.txt’: Permission denied
> >>>>-r--r--r-- 1 cdc admincdc 7 nov 18 2016 444.txt
> >>>>-r--r--r-- 1 cdc admincdc 0 nov 18 2016 kk.txt
> >>>>cdc@l056:~$
> >>>>cdc@l056:~$
> >>>>cdc@l056:~$
> >>>>cdc@l056:~$ sudo su
> >>>>[sudo] password for cdc:
> >>>>root@l056:/home/cdc# rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
> >>>>chmod 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
> >>>>-r--r--r-- 1 root root 7 nov 18 2016 444.txt
> >>>>-r--r--r-- 1 root root 7 nov 18 2016 kk.txt
> >>>>root@l056:/home/cdc#
> >>>>root@l056:/home/cdc#
> >>>>root@l056:/home/cdc# exit
> >>>>cdc@l056:~$
> >>>>
> >>>>See at the end the logs with and without the debug option activated at
> >>>>the glusterfs client.
> >>>>
> >>>>Thanks again, and sorry for the long message.
> >>>>
> >>>>Omar
> >>>>
> >>>>
> >>>>
> >>>>* Logs on the glusterfs client side with NO DEBUG:
> >>>>
> >>>>[2016-11-18 15:45:43.681215] E [MSGID: 114031]
> >>>>[client-rpc-fops.c:444:client3_3_open_cbk]
> >>>>0-home-lab-3-client-3: remote
> >>>>operation failed. Path: /cdc/kk.txt
> >>>>(cc98e831-3410-4202-8254-2d4b48e8ea68) [Permission denied]
> >>>>[2016-11-18 15:45:43.681297] W [fuse-bridge.c:989:fuse_fd_cbk]
> >>>>0-glusterfs-fuse: 399: OPEN() /cdc/kk.txt => -1 (Permission denied)
> >>>>[2016-11-18 15:45:55.457922] I [io-stats.c:1574:io_stats_dump_fd]
> >>>>0-home-lab-3: --- fd stats ---
> >>>>[2016-11-18 15:45:55.457978] I [io-stats.c:1579:io_stats_dump_fd]
> >>>>0-home-lab-3: Filename : /cdc/444.txt
> >>>>[2016-11-18 15:45:55.457997] I [io-stats.c:1594:io_stats_dump_fd]
> >>>>0-home-lab-3: BytesWritten : 7 bytes
> >>>>[2016-11-18 15:45:55.458011] I [io-stats.c:1606:io_stats_dump_fd]
> >>>>0-home-lab-3: Write 000004b+ : 1
> >>>>[2016-11-18 15:45:55.471933] I [io-stats.c:1574:io_stats_dump_fd]
> >>>>0-home-lab-3: --- fd stats ---
> >>>>
> >>>>
> >>>>* Logs on the glusterfs client side with DEBUG activated:
> >>>>
> >>>>[2016-11-18 16:06:17.040092] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
> >>>>STAT scheduled
> >>>>as fast fop
> >>>>[2016-11-18 16:06:17.045986] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> >>>>scheduled as fast fop
> >>>>[2016-11-18 16:06:17.046132] D [MSGID: 0]
> >>>>[dht-common.c:2684:dht_lookup]
> >>>>0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
> >>>>home-lab-3-client-3
> >>>>[2016-11-18 16:06:17.046833] D [MSGID: 0]
> >>>>[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> >>>>returned for /cdc/kk.txt with op_ret 0 [Invalid argument]
> >>>>[2016-11-18 16:06:17.047671] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK
> >>>>scheduled as normal fop
> >>>>[2016-11-18 16:06:17.050539] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >>>>event 19 on graph -1418558531
> >>>>[2016-11-18 16:06:17.050867] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >>>>event 19 on graph 0
> >>>>[2016-11-18 16:06:17.051948] D [logging.c:1953:_gf_msg_internal]
> >>>>0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> >>>>About to flush least recently used log message to disk
> >>>>[2016-11-18 16:06:17.051871] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> >>>>scheduled as fast fop
> >>>>[2016-11-18 16:06:17.051946] D [MSGID: 0]
> >>>>[dht-common.c:2684:dht_lookup]
> >>>>0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
> >>>>home-lab-3-client-0
> >>>>[2016-11-18 16:06:17.062108] D [MSGID: 0]
> >>>>[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> >>>>returned for /cdc/444.txt with op_ret 0 [Invalid argument]
> >>>>[2016-11-18 16:06:17.063692] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >>>>event 19 on graph -1418558531
> >>>>[2016-11-18 16:06:17.064015] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >>>>event 19 on graph 0
> >>>>[2016-11-18 16:06:17.065068] D [logging.c:1953:_gf_msg_internal]
> >>>>0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> >>>>About to flush least recently used log message to disk
> >>>>The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
> >>>>0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 4 times
> >>>>between [2016-11-18 16:06:17.040092] and [2016-11-18 16:06:17.064175]
> >>>>[2016-11-18 16:06:17.065066] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> >>>>scheduled as fast fop
> >>>>[2016-11-18 16:06:17.065232] D [MSGID: 0]
> >>>>[dht-common.c:2684:dht_lookup]
> >>>>0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
> >>>>home-lab-3-client-0
> >>>>[2016-11-18 16:06:17.065755] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.065897] D [MSGID: 0]
> >>>>[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> >>>>returned for /cdc/444.txt with op_ret -1 [No such file or directory]
> >>>>[2016-11-18 16:06:17.066016] D [MSGID: 0]
> >>>>[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry
> >>>>/cdc/444.txt
> >>>>missing on subvol home-lab-3-client-0
> >>>>[2016-11-18 16:06:17.066104] D [MSGID: 0]
> >>>>[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
> >>>>lookup call to 4 subvols
> >>>>[2016-11-18 16:06:17.066594] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.066669] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >>>>home-lab-3-client-0
> >>>>[2016-11-18 16:06:17.066771] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.066898] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >>>>home-lab-3-client-1
> >>>>[2016-11-18 16:06:17.066906] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.067019] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >>>>home-lab-3-client-2
> >>>>[2016-11-18 16:06:17.066915] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.067066] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >>>>home-lab-3-client-3
> >>>>[2016-11-18 16:06:17.067107] D [MSGID: 0]
> >>>>[dht-common.c:1527:dht_lookup_everywhere_done]
> >>>>0-home-lab-3-dht: STATUS:
> >>>>hashed_subvol home-lab-3-client-0 cached_subvol null
> >>>>[2016-11-18 16:06:17.067151] D [MSGID: 0]
> >>>>[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
> >>>>was no cached file and unlink on hashed is not skipped /cdc/444.txt
> >>>>[2016-11-18 16:06:17.067179] D [MSGID: 0]
> >>>>[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1
> >>>>error: No such
> >>>>file or directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.067206] D [MSGID: 0]
> >>>>[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.067250] D [MSGID: 0]
> >>>>[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No
> >>>>such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.067275] D [MSGID: 0]
> >>>>[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.067308] D [MSGID: 0]
> >>>>[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No
> >>>>such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.067387] D [MSGID: 0]
> >>>>[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file
> >>>>or directory
> >>>>[No such file or directory]
> >>>>[2016-11-18 16:06:17.068157] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> >>>>scheduled as fast fop
> >>>>[2016-11-18 16:06:17.068281] D [MSGID: 0]
> >>>>[dht-common.c:2684:dht_lookup]
> >>>>0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
> >>>>home-lab-3-client-0
> >>>>[2016-11-18 16:06:17.068738] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.068781] D [MSGID: 0]
> >>>>[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> >>>>returned for /cdc/444.txt with op_ret -1 [No such file or directory]
> >>>>[2016-11-18 16:06:17.068805] D [MSGID: 0]
> >>>>[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry
> >>>>/cdc/444.txt
> >>>>missing on subvol home-lab-3-client-0
> >>>>[2016-11-18 16:06:17.068827] D [MSGID: 0]
> >>>>[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
> >>>>lookup call to 4 subvols
> >>>>[2016-11-18 16:06:17.069286] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.069345] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >>>>home-lab-3-client-0
> >>>>[2016-11-18 16:06:17.069452] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.069507] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >>>>home-lab-3-client-2
> >>>>[2016-11-18 16:06:17.069556] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.069585] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >>>>home-lab-3-client-3
> >>>>[2016-11-18 16:06:17.069939] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.069983] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >>>>home-lab-3-client-1
> >>>>[2016-11-18 16:06:17.070005] D [MSGID: 0]
> >>>>[dht-common.c:1527:dht_lookup_everywhere_done]
> >>>>0-home-lab-3-dht: STATUS:
> >>>>hashed_subvol home-lab-3-client-0 cached_subvol null
> >>>>[2016-11-18 16:06:17.070038] D [MSGID: 0]
> >>>>[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
> >>>>was no cached file and unlink on hashed is not skipped /cdc/444.txt
> >>>>[2016-11-18 16:06:17.070098] D [MSGID: 0]
> >>>>[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1
> >>>>error: No such
> >>>>file or directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.070128] D [MSGID: 0]
> >>>>[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.070154] D [MSGID: 0]
> >>>>[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No
> >>>>such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.070178] D [MSGID: 0]
> >>>>[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.070208] D [MSGID: 0]
> >>>>[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No
> >>>>such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.070251] D [MSGID: 0]
> >>>>[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file
> >>>>or directory
> >>>>[No such file or directory]
> >>>>[2016-11-18 16:06:17.070537] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
> >>>>MKNOD scheduled
> >>>>as normal fop
> >>>>[2016-11-18 16:06:17.070826] D [MSGID: 0]
> >>>>[dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
> >>>>creating /cdc/444.txt on home-lab-3-client-0
> >>>>[2016-11-18 16:06:17.071100] D [MSGID: 0]
> >>>>[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
> >>>>'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
> >>>>10456189239296 and avail_inodes is: 99.00
> >>>>[2016-11-18 16:06:17.071142] D [MSGID: 0]
> >>>>[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
> >>>>'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
> >>>>10499693871104 and avail_inodes is: 99.00
> >>>>[2016-11-18 16:06:17.071183] D [MSGID: 0]
> >>>>[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
> >>>>'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
> >>>>10500911267840 and avail_inodes is: 99.00
> >>>>[2016-11-18 16:06:17.071343] D [MSGID: 0]
> >>>>[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
> >>>>'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
> >>>>8600432410624 and avail_inodes is: 99.00
> >>>>[2016-11-18 16:06:17.073943] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >>>>event 19 on graph -1418558531
> >>>>[2016-11-18 16:06:17.074273] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
> >>>>STAT scheduled
> >>>>as fast fop
> >>>>[2016-11-18 16:06:17.075035] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
> >>>>OPEN scheduled
> >>>>as fast fop
> >>>>[2016-11-18 16:06:17.075913] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
> >>>>WRITE scheduled
> >>>>as slow fop
> >>>>[2016-11-18 16:06:17.076205] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
> >>>>FSTAT scheduled
> >>>>as fast fop
> >>>>[2016-11-18 16:06:17.076636] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >>>>event 19 on graph 0
> >>>>[2016-11-18 16:06:17.077282] D [fuse-bridge.c:52:fuse_invalidate]
> >>>>0-fuse: Invalidate inode id 140363155120276.
> >>>>[2016-11-18 16:06:17.077981] I [io-stats.c:1574:io_stats_dump_fd]
> >>>>0-home-lab-3: --- fd stats ---
> >>>>[2016-11-18 16:06:17.078032] I [io-stats.c:1579:io_stats_dump_fd]
> >>>>0-home-lab-3: Filename : /cdc/444.txt
> >>>>[2016-11-18 16:06:17.078049] I [io-stats.c:1594:io_stats_dump_fd]
> >>>>0-home-lab-3: BytesWritten : 7 bytes
> >>>>[2016-11-18 16:06:17.078064] I [io-stats.c:1606:io_stats_dump_fd]
> >>>>0-home-lab-3: Write 000004b+ : 1
> >>>>[2016-11-18 16:06:17.079156] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
> >>>>scheduled as normal fop
> >>>>[2016-11-18 16:06:17.079794] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >>>>event 19 on graph 0
> >>>>[2016-11-18 16:06:17.082196] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> >>>>scheduled as fast fop
> >>>>[2016-11-18 16:06:17.082772] D [logging.c:1953:_gf_msg_internal]
> >>>>0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> >>>>About to flush least recently used log message to disk
> >>>>[2016-11-18 16:06:17.082318] D [MSGID: 0]
> >>>>[dht-common.c:2684:dht_lookup]
> >>>>0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
> >>>>home-lab-3-client-3
> >>>>[2016-11-18 16:06:17.082770] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.083168] D [MSGID: 0]
> >>>>[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> >>>>returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
> >>>>[2016-11-18 16:06:17.083282] D [MSGID: 0]
> >>>>[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
> >>>>missing on subvol home-lab-3-client-3
> >>>>[2016-11-18 16:06:17.083320] D [MSGID: 0]
> >>>>[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
> >>>>lookup call to 4 subvols
> >>>>[2016-11-18 16:06:17.083991] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.084043] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
> >>>>home-lab-3-client-0
> >>>>[2016-11-18 16:06:17.084048] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.084092] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
> >>>>home-lab-3-client-3
> >>>>[2016-11-18 16:06:17.084114] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.084167] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
> >>>>home-lab-3-client-1
> >>>>[2016-11-18 16:06:17.084091] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.084377] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
> >>>>home-lab-3-client-2
> >>>>[2016-11-18 16:06:17.084406] D [MSGID: 0]
> >>>>[dht-common.c:1527:dht_lookup_everywhere_done]
> >>>>0-home-lab-3-dht: STATUS:
> >>>>hashed_subvol home-lab-3-client-3 cached_subvol null
> >>>>[2016-11-18 16:06:17.084511] D [MSGID: 0]
> >>>>[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
> >>>>was no cached file and unlink on hashed is not skipped /cdc/kk.txt
> >>>>[2016-11-18 16:06:17.084534] D [MSGID: 0]
> >>>>[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1
> >>>>error: No such
> >>>>file or directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.084683] D [MSGID: 0]
> >>>>[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.084783] D [MSGID: 0]
> >>>>[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No
> >>>>such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.084857] D [MSGID: 0]
> >>>>[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.084889] D [MSGID: 0]
> >>>>[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No
> >>>>such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.085001] D [MSGID: 0]
> >>>>[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file
> >>>>or directory
> >>>>[No such file or directory]
> >>>>[2016-11-18 16:06:17.086098] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
> >>>>STAT scheduled
> >>>>as fast fop
> >>>>[2016-11-18 16:06:17.087256] D [logging.c:1953:_gf_msg_internal]
> >>>>0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> >>>>About to flush least recently used log message to disk
> >>>>[2016-11-18 16:06:17.087173] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> >>>>scheduled as fast fop
> >>>>[2016-11-18 16:06:17.087254] D [MSGID: 0]
> >>>>[dht-common.c:2684:dht_lookup]
> >>>>0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
> >>>>home-lab-3-client-3
> >>>>[2016-11-18 16:06:17.087749] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.087795] D [MSGID: 0]
> >>>>[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> >>>>returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
> >>>>[2016-11-18 16:06:17.088014] D [MSGID: 0]
> >>>>[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
> >>>>missing on subvol home-lab-3-client-3
> >>>>[2016-11-18 16:06:17.088036] D [MSGID: 0]
> >>>>[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
> >>>>lookup call to 4 subvols
> >>>>[2016-11-18 16:06:17.088598] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.088639] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
> >>>>home-lab-3-client-0
> >>>>[2016-11-18 16:06:17.088804] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.088855] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
> >>>>home-lab-3-client-2
> >>>>[2016-11-18 16:06:17.088819] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.089069] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
> >>>>home-lab-3-client-3
> >>>>[2016-11-18 16:06:17.089380] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.089421] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
> >>>>home-lab-3-client-1
> >>>>[2016-11-18 16:06:17.089443] D [MSGID: 0]
> >>>>[dht-common.c:1527:dht_lookup_everywhere_done]
> >>>>0-home-lab-3-dht: STATUS:
> >>>>hashed_subvol home-lab-3-client-3 cached_subvol null
> >>>>[2016-11-18 16:06:17.089463] D [MSGID: 0]
> >>>>[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
> >>>>was no cached file and unlink on hashed is not skipped /cdc/kk.txt
> >>>>[2016-11-18 16:06:17.089575] D [MSGID: 0]
> >>>>[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1
> >>>>error: No such
> >>>>file or directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.089601] D [MSGID: 0]
> >>>>[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.089712] D [MSGID: 0]
> >>>>[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No
> >>>>such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.089738] D [MSGID: 0]
> >>>>[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.090006] D [MSGID: 0]
> >>>>[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No
> >>>>such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:06:17.090156] D [MSGID: 0]
> >>>>[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file
> >>>>or directory
> >>>>[No such file or directory]
> >>>>[2016-11-18 16:06:17.090648] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
> >>>>MKNOD scheduled
> >>>>as normal fop
> >>>>[2016-11-18 16:06:17.090732] D [MSGID: 0]
> >>>>[dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
> >>>>creating /cdc/kk.txt on home-lab-3-client-3
> >>>>[2016-11-18 16:06:17.091499] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >>>>event 19 on graph -1418558531
> >>>>[2016-11-18 16:06:17.091843] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
> >>>>scheduled as normal fop
> >>>>[2016-11-18 16:06:17.093831] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >>>>event 19 on graph 0
> >>>>[2016-11-18 16:06:17.094711] D [logging.c:1953:_gf_msg_internal]
> >>>>0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> >>>>About to flush least recently used log message to disk
> >>>>[2016-11-18 16:06:17.093216] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
> >>>>scheduled as normal fop
> >>>>[2016-11-18 16:06:17.094709] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
> >>>>OPEN scheduled
> >>>>as fast fop
> >>>>[2016-11-18 16:06:17.095280] E [MSGID: 114031]
> >>>>[client-rpc-fops.c:444:client3_3_open_cbk]
> >>>>0-home-lab-3-client-3: remote
> >>>>operation failed. Path: /cdc/kk.txt
> >>>>(09bfb6fa-82fb-463b-abd9-bb7ef88e10c0) [Permission denied]
> >>>>[2016-11-18 16:06:17.095341] D [MSGID: 0]
> >>>>[client-rpc-fops.c:448:client3_3_open_cbk] 0-stack-trace:
> >>>>stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error:
> >>>>Permission denied
> >>>>[Permission denied]
> >>>>[2016-11-18 16:06:17.095366] D [MSGID: 0]
> >>>>[dht-inode-read.c:45:dht_open_cbk] 0-home-lab-3-dht: subvolume
> >>>>home-lab-3-client-3 returned -1 [Permission denied]
> >>>>[2016-11-18 16:06:17.095480] D [MSGID: 0]
> >>>>[dht-inode-read.c:59:dht_open_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-dht returned -1 error: Permission denied
> >>>>[Permission denied]
> >>>>[2016-11-18 16:06:17.095671] D [MSGID: 0]
> >>>>[read-ahead.c:101:ra_open_cbk]
> >>>>0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-read-ahead
> >>>>returned -1 error: Permission denied [Permission denied]
> >>>>[2016-11-18 16:06:17.095759] D [MSGID: 0]
> >>>>[io-cache.c:605:ioc_open_cbk]
> >>>>0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-io-cache
> >>>>returned -1 error: Permission denied [Permission denied]
> >>>>[2016-11-18 16:06:17.095963] D [MSGID: 0]
> >>>>[defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: Permission
> >>>>denied [Permission denied]
> >>>>[2016-11-18 16:06:17.095988] D [MSGID: 0]
> >>>>[defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: Permission
> >>>>denied [Permission denied]
> >>>>[2016-11-18 16:06:17.096275] D [MSGID: 0]
> >>>>[io-stats.c:1936:io_stats_open_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3 returned -1 error: Permission denied
> >>>>[Permission denied]
> >>>>[2016-11-18 16:06:17.096362] W [fuse-bridge.c:989:fuse_fd_cbk]
> >>>>0-glusterfs-fuse: 338: OPEN() /cdc/kk.txt => -1 (Permission denied)
> >>>>[2016-11-18 16:06:17.097104] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
> >>>>scheduled as normal fop
> >>>>[2016-11-18 16:06:17.097674] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >>>>event 19 on graph 0
> >>>>[2016-11-18 16:06:17.099912] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >>>>event 19 on graph 0
> >>>>[2016-11-18 16:06:24.785225] D
> >>>>[rpc-clnt-ping.c:295:rpc_clnt_start_ping]
> >>>>0-home-lab-3-client-0: returning as transport is already
> >>>>disconnected OR
> >>>>there are no frames (0 || 0)
> >>>>[2016-11-18 16:06:24.785338] D
> >>>>[rpc-clnt-ping.c:295:rpc_clnt_start_ping]
> >>>>0-home-lab-3-client-1: returning as transport is already
> >>>>disconnected OR
> >>>>there are no frames (0 || 0)
> >>>>[2016-11-18 16:06:24.785359] D
> >>>>[rpc-clnt-ping.c:295:rpc_clnt_start_ping]
> >>>>0-home-lab-3-client-2: returning as transport is already
> >>>>disconnected OR
> >>>>there are no frames (0 || 0)
> >>>>[2016-11-18 16:06:24.785377] D
> >>>>[rpc-clnt-ping.c:295:rpc_clnt_start_ping]
> >>>>0-home-lab-3-client-3: returning as transport is already
> >>>>disconnected OR
> >>>>there are no frames (0 || 0)
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>** TEST DONE as user cdc --> ERROR
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>[2016-11-18 16:06:46.737686] D
> >>>>[rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
> >>>>/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
> >>>>/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
> >>>>
> >>>>(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
> >>>>/lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
> >>>>/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
> >>>>
> >>>>))))) 0-: 138.4.2.246:49172: ping timer event already removed
> >>>>[2016-11-18 16:06:49.545698] D
> >>>>[rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
> >>>>/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
> >>>>/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
> >>>>
> >>>>(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
> >>>>/lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
> >>>>/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
> >>>>
> >>>>))))) 0-: 138.4.2.177:49157: ping timer event already removed
> >>>>[2016-11-18 16:06:49.546474] D
> >>>>[rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
> >>>>/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
> >>>>/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
> >>>>
> >>>>(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
> >>>>/lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
> >>>>/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
> >>>>
> >>>>))))) 0-: 138.4.2.189:49157: ping timer event already removed
> >>>>[2016-11-18 16:06:49.547093] D
> >>>>[rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
> >>>>/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
> >>>>/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
> >>>>
> >>>>(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
> >>>>/lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
> >>>>/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
> >>>>
> >>>>))))) 0-: 138.4.2.248:49157: ping timer event already removed
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>** user CDC becomed root at this point
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>[2016-11-18 16:07:11.907892] D [logging.c:1953:_gf_msg_internal]
> >>>>0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> >>>>About to flush least recently used log message to disk
> >>>>The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
> >>>>0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 7 times
> >>>>between [2016-11-18 16:06:17.086098] and [2016-11-18 16:07:11.906944]
> >>>>[2016-11-18 16:07:11.907884] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> >>>>scheduled as fast fop
> >>>>[2016-11-18 16:07:11.908875] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >>>>event 19 on graph -1418558531
> >>>>[2016-11-18 16:07:11.909179] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >>>>event 19 on graph 0
> >>>>[2016-11-18 16:07:11.909256] D [logging.c:1953:_gf_msg_internal]
> >>>>0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> >>>>About to flush least recently used log message to disk
> >>>>The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
> >>>>0-home-lab-3-io-threads: UNLINK scheduled as normal fop" repeated 2
> >>>>times between [2016-11-18 16:06:17.047671] and [2016-11-18
> >>>>16:07:11.908114]
> >>>>[2016-11-18 16:07:11.909254] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
> >>>>STAT scheduled
> >>>>as fast fop
> >>>>[2016-11-18 16:07:11.911109] D [logging.c:1953:_gf_msg_internal]
> >>>>0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> >>>>About to flush least recently used log message to disk
> >>>>[2016-11-18 16:07:11.911018] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> >>>>scheduled as fast fop
> >>>>[2016-11-18 16:07:11.911107] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK
> >>>>scheduled as normal fop
> >>>>[2016-11-18 16:07:11.911871] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >>>>event 19 on graph -1418558531
> >>>>[2016-11-18 16:07:11.912045] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >>>>event 19 on graph 0
> >>>>[2016-11-18 16:07:11.913067] D [logging.c:1953:_gf_msg_internal]
> >>>>0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> >>>>About to flush least recently used log message to disk
> >>>>The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
> >>>>0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 2 times
> >>>>between [2016-11-18 16:07:11.909254] and [2016-11-18 16:07:11.912165]
> >>>>[2016-11-18 16:07:11.913065] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> >>>>scheduled as fast fop
> >>>>[2016-11-18 16:07:11.913193] D [MSGID: 0]
> >>>>[dht-common.c:2684:dht_lookup]
> >>>>0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
> >>>>home-lab-3-client-0
> >>>>[2016-11-18 16:07:11.913710] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.913757] D [MSGID: 0]
> >>>>[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> >>>>returned for /cdc/444.txt with op_ret -1 [No such file or directory]
> >>>>[2016-11-18 16:07:11.913788] D [MSGID: 0]
> >>>>[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry
> >>>>/cdc/444.txt
> >>>>missing on subvol home-lab-3-client-0
> >>>>[2016-11-18 16:07:11.913810] D [MSGID: 0]
> >>>>[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
> >>>>lookup call to 4 subvols
> >>>>[2016-11-18 16:07:11.914336] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.914379] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >>>>home-lab-3-client-0
> >>>>[2016-11-18 16:07:11.914424] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.914448] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >>>>home-lab-3-client-1
> >>>>[2016-11-18 16:07:11.914501] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.914617] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >>>>home-lab-3-client-3
> >>>>[2016-11-18 16:07:11.914649] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.914698] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >>>>home-lab-3-client-2
> >>>>[2016-11-18 16:07:11.914724] D [MSGID: 0]
> >>>>[dht-common.c:1527:dht_lookup_everywhere_done]
> >>>>0-home-lab-3-dht: STATUS:
> >>>>hashed_subvol home-lab-3-client-0 cached_subvol null
> >>>>[2016-11-18 16:07:11.914743] D [MSGID: 0]
> >>>>[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
> >>>>was no cached file and unlink on hashed is not skipped /cdc/444.txt
> >>>>[2016-11-18 16:07:11.914764] D [MSGID: 0]
> >>>>[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1
> >>>>error: No such
> >>>>file or directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.914791] D [MSGID: 0]
> >>>>[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.914825] D [MSGID: 0]
> >>>>[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No
> >>>>such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.914855] D [MSGID: 0]
> >>>>[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.914880] D [MSGID: 0]
> >>>>[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No
> >>>>such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.914947] D [MSGID: 0]
> >>>>[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file
> >>>>or directory
> >>>>[No such file or directory]
> >>>>[2016-11-18 16:07:11.915569] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> >>>>scheduled as fast fop
> >>>>[2016-11-18 16:07:11.915644] D [MSGID: 0]
> >>>>[dht-common.c:2684:dht_lookup]
> >>>>0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
> >>>>home-lab-3-client-0
> >>>>[2016-11-18 16:07:11.916090] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.916134] D [MSGID: 0]
> >>>>[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> >>>>returned for /cdc/444.txt with op_ret -1 [No such file or directory]
> >>>>[2016-11-18 16:07:11.916158] D [MSGID: 0]
> >>>>[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry
> >>>>/cdc/444.txt
> >>>>missing on subvol home-lab-3-client-0
> >>>>[2016-11-18 16:07:11.916185] D [MSGID: 0]
> >>>>[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
> >>>>lookup call to 4 subvols
> >>>>[2016-11-18 16:07:11.916593] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.916780] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >>>>home-lab-3-client-0
> >>>>[2016-11-18 16:07:11.916878] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.916994] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >>>>home-lab-3-client-3
> >>>>[2016-11-18 16:07:11.917076] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.916906] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.917117] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >>>>home-lab-3-client-2
> >>>>[2016-11-18 16:07:11.917153] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
> >>>>home-lab-3-client-1
> >>>>[2016-11-18 16:07:11.917205] D [MSGID: 0]
> >>>>[dht-common.c:1527:dht_lookup_everywhere_done]
> >>>>0-home-lab-3-dht: STATUS:
> >>>>hashed_subvol home-lab-3-client-0 cached_subvol null
> >>>>[2016-11-18 16:07:11.917368] D [MSGID: 0]
> >>>>[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
> >>>>was no cached file and unlink on hashed is not skipped /cdc/444.txt
> >>>>[2016-11-18 16:07:11.917437] D [MSGID: 0]
> >>>>[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1
> >>>>error: No such
> >>>>file or directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.917467] D [MSGID: 0]
> >>>>[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.917496] D [MSGID: 0]
> >>>>[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No
> >>>>such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.917521] D [MSGID: 0]
> >>>>[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.917551] D [MSGID: 0]
> >>>>[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No
> >>>>such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.917726] D [MSGID: 0]
> >>>>[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file
> >>>>or directory
> >>>>[No such file or directory]
> >>>>[2016-11-18 16:07:11.917958] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
> >>>>MKNOD scheduled
> >>>>as normal fop
> >>>>[2016-11-18 16:07:11.918230] D [MSGID: 0]
> >>>>[dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
> >>>>creating /cdc/444.txt on home-lab-3-client-0
> >>>>[2016-11-18 16:07:11.918488] D [MSGID: 0]
> >>>>[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
> >>>>'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
> >>>>10456166432768 and avail_inodes is: 99.00
> >>>>[2016-11-18 16:07:11.918570] D [MSGID: 0]
> >>>>[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
> >>>>'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
> >>>>8600417075200 and avail_inodes is: 99.00
> >>>>[2016-11-18 16:07:11.918603] D [MSGID: 0]
> >>>>[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
> >>>>'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
> >>>>10500905500672 and avail_inodes is: 99.00
> >>>>[2016-11-18 16:07:11.918675] D [MSGID: 0]
> >>>>[dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
> >>>>'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
> >>>>10499686531072 and avail_inodes is: 99.00
> >>>>[2016-11-18 16:07:11.919095] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >>>>event 19 on graph -1418558531
> >>>>[2016-11-18 16:07:11.919377] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
> >>>>STAT scheduled
> >>>>as fast fop
> >>>>[2016-11-18 16:07:11.920040] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
> >>>>OPEN scheduled
> >>>>as fast fop
> >>>>[2016-11-18 16:07:11.920889] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
> >>>>WRITE scheduled
> >>>>as slow fop
> >>>>[2016-11-18 16:07:11.921051] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
> >>>>FSTAT scheduled
> >>>>as fast fop
> >>>>[2016-11-18 16:07:11.921583] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >>>>event 19 on graph 0
> >>>>[2016-11-18 16:07:11.922068] D [fuse-bridge.c:52:fuse_invalidate]
> >>>>0-fuse: Invalidate inode id 140363155120276.
> >>>>[2016-11-18 16:07:11.922936] I [io-stats.c:1574:io_stats_dump_fd]
> >>>>0-home-lab-3: --- fd stats ---
> >>>>[2016-11-18 16:07:11.922988] I [io-stats.c:1579:io_stats_dump_fd]
> >>>>0-home-lab-3: Filename : /cdc/444.txt
> >>>>[2016-11-18 16:07:11.923004] I [io-stats.c:1594:io_stats_dump_fd]
> >>>>0-home-lab-3: BytesWritten : 7 bytes
> >>>>[2016-11-18 16:07:11.923026] I [io-stats.c:1606:io_stats_dump_fd]
> >>>>0-home-lab-3: Write 000004b+ : 1
> >>>>[2016-11-18 16:07:11.923779] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
> >>>>STAT scheduled
> >>>>as fast fop
> >>>>[2016-11-18 16:07:11.924504] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >>>>event 19 on graph 0
> >>>>[2016-11-18 16:07:11.926692] D [logging.c:1953:_gf_msg_internal]
> >>>>0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> >>>>About to flush least recently used log message to disk
> >>>>The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
> >>>>0-home-lab-3-io-threads: SETATTR scheduled as normal fop" repeated 2
> >>>>times between [2016-11-18 16:06:17.097104] and [2016-11-18
> >>>>16:07:11.923946]
> >>>>[2016-11-18 16:07:11.926690] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> >>>>scheduled as fast fop
> >>>>[2016-11-18 16:07:11.926858] D [MSGID: 0]
> >>>>[dht-common.c:2684:dht_lookup]
> >>>>0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
> >>>>home-lab-3-client-3
> >>>>[2016-11-18 16:07:11.927327] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.927376] D [MSGID: 0]
> >>>>[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> >>>>returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
> >>>>[2016-11-18 16:07:11.927401] D [MSGID: 0]
> >>>>[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
> >>>>missing on subvol home-lab-3-client-3
> >>>>[2016-11-18 16:07:11.927422] D [MSGID: 0]
> >>>>[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
> >>>>lookup call to 4 subvols
> >>>>[2016-11-18 16:07:11.927940] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.927987] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
> >>>>home-lab-3-client-0
> >>>>[2016-11-18 16:07:11.928043] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.928055] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.928058] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.928170] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
> >>>>home-lab-3-client-1
> >>>>[2016-11-18 16:07:11.928174] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
> >>>>home-lab-3-client-3
> >>>>[2016-11-18 16:07:11.928277] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
> >>>>home-lab-3-client-2
> >>>>[2016-11-18 16:07:11.928360] D [MSGID: 0]
> >>>>[dht-common.c:1527:dht_lookup_everywhere_done]
> >>>>0-home-lab-3-dht: STATUS:
> >>>>hashed_subvol home-lab-3-client-3 cached_subvol null
> >>>>[2016-11-18 16:07:11.928381] D [MSGID: 0]
> >>>>[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
> >>>>was no cached file and unlink on hashed is not skipped /cdc/kk.txt
> >>>>[2016-11-18 16:07:11.928403] D [MSGID: 0]
> >>>>[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1
> >>>>error: No such
> >>>>file or directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.928431] D [MSGID: 0]
> >>>>[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.928456] D [MSGID: 0]
> >>>>[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No
> >>>>such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.928480] D [MSGID: 0]
> >>>>[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.928511] D [MSGID: 0]
> >>>>[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No
> >>>>such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.928753] D [MSGID: 0]
> >>>>[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file
> >>>>or directory
> >>>>[No such file or directory]
> >>>>[2016-11-18 16:07:11.930485] D [logging.c:1953:_gf_msg_internal]
> >>>>0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> >>>>About to flush least recently used log message to disk
> >>>>[2016-11-18 16:07:11.929862] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
> >>>>STAT scheduled
> >>>>as fast fop
> >>>>[2016-11-18 16:07:11.930483] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
> >>>>scheduled as fast fop
> >>>>[2016-11-18 16:07:11.930602] D [MSGID: 0]
> >>>>[dht-common.c:2684:dht_lookup]
> >>>>0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
> >>>>home-lab-3-client-3
> >>>>[2016-11-18 16:07:11.931071] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.931124] D [MSGID: 0]
> >>>>[dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
> >>>>returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
> >>>>[2016-11-18 16:07:11.931148] D [MSGID: 0]
> >>>>[dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
> >>>>missing on subvol home-lab-3-client-3
> >>>>[2016-11-18 16:07:11.931348] D [MSGID: 0]
> >>>>[dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
> >>>>lookup call to 4 subvols
> >>>>[2016-11-18 16:07:11.931880] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.931927] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
> >>>>home-lab-3-client-0
> >>>>[2016-11-18 16:07:11.932002] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.931994] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.931991] D [MSGID: 0]
> >>>>[client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned
> >>>>-1 error: No
> >>>>such file or directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.932078] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
> >>>>home-lab-3-client-2
> >>>>[2016-11-18 16:07:11.932116] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
> >>>>home-lab-3-client-1
> >>>>[2016-11-18 16:07:11.932315] D [MSGID: 0]
> >>>>[dht-common.c:1862:dht_lookup_everywhere_cbk]
> >>>>0-home-lab-3-dht: returned
> >>>>with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
> >>>>home-lab-3-client-3
> >>>>[2016-11-18 16:07:11.932363] D [MSGID: 0]
> >>>>[dht-common.c:1527:dht_lookup_everywhere_done]
> >>>>0-home-lab-3-dht: STATUS:
> >>>>hashed_subvol home-lab-3-client-3 cached_subvol null
> >>>>[2016-11-18 16:07:11.932383] D [MSGID: 0]
> >>>>[dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
> >>>>was no cached file and unlink on hashed is not skipped /cdc/kk.txt
> >>>>[2016-11-18 16:07:11.932407] D [MSGID: 0]
> >>>>[dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
> >>>>stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1
> >>>>error: No such
> >>>>file or directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.932446] D [MSGID: 0]
> >>>>[io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.932471] D [MSGID: 0]
> >>>>[quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No
> >>>>such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.932494] D [MSGID: 0]
> >>>>[md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.932518] D [MSGID: 0]
> >>>>[defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No
> >>>>such file or
> >>>>directory [No such file or directory]
> >>>>[2016-11-18 16:07:11.932725] D [logging.c:1953:_gf_msg_internal]
> >>>>0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> >>>>About to flush least recently used log message to disk
> >>>>[2016-11-18 16:07:11.932545] D [MSGID: 0]
> >>>>[io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
> >>>>0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file
> >>>>or directory
> >>>>[No such file or directory]
> >>>>[2016-11-18 16:07:11.932722] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
> >>>>MKNOD scheduled
> >>>>as normal fop
> >>>>[2016-11-18 16:07:11.933008] D [MSGID: 0]
> >>>>[dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
> >>>>creating /cdc/kk.txt on home-lab-3-client-3
> >>>>[2016-11-18 16:07:11.934086] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >>>>event 19 on graph -1418558531
> >>>>[2016-11-18 16:07:11.934323] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
> >>>>scheduled as normal fop
> >>>>[2016-11-18 16:07:11.934922] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
> >>>>STAT scheduled
> >>>>as fast fop
> >>>>[2016-11-18 16:07:11.935497] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
> >>>>scheduled as normal fop
> >>>>[2016-11-18 16:07:11.936102] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >>>>event 19 on graph 0
> >>>>[2016-11-18 16:07:11.936903] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
> >>>>OPEN scheduled
> >>>>as fast fop
> >>>>[2016-11-18 16:07:11.937540] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
> >>>>WRITE scheduled
> >>>>as slow fop
> >>>>[2016-11-18 16:07:11.937682] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
> >>>>FSTAT scheduled
> >>>>as fast fop
> >>>>[2016-11-18 16:07:11.938267] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >>>>event 19 on graph 0
> >>>>[2016-11-18 16:07:11.938757] D [fuse-bridge.c:52:fuse_invalidate]
> >>>>0-fuse: Invalidate inode id 140363155120652.
> >>>>[2016-11-18 16:07:11.938994] I [io-stats.c:1574:io_stats_dump_fd]
> >>>>0-home-lab-3: --- fd stats ---
> >>>>[2016-11-18 16:07:11.939072] I [io-stats.c:1579:io_stats_dump_fd]
> >>>>0-home-lab-3: Filename : /cdc/kk.txt
> >>>>[2016-11-18 16:07:11.939096] I [io-stats.c:1594:io_stats_dump_fd]
> >>>>0-home-lab-3: BytesWritten : 7 bytes
> >>>>[2016-11-18 16:07:11.939112] I [io-stats.c:1606:io_stats_dump_fd]
> >>>>0-home-lab-3: Write 000004b+ : 1
> >>>>[2016-11-18 16:07:11.939448] D [MSGID: 0]
> >>>>[io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
> >>>>scheduled as normal fop
> >>>>[2016-11-18 16:07:11.940028] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >>>>event 19 on graph 0
> >>>>[2016-11-18 16:07:11.942130] D [fuse-bridge.c:5264:notify] 0-fuse: got
> >>>>event 19 on graph 0
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>** TEST DONE as root --> NO PROBLEM
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>[2016-11-18 16:07:28.794150] D
> >>>>[rpc-clnt-ping.c:295:rpc_clnt_start_ping]
> >>>>0-home-lab-3-client-2: returning as transport is already
> >>>>disconnected OR
> >>>>there are no frames (0 || 0)
> >>>>[2016-11-18 16:07:31.794656] D
> >>>>[rpc-clnt-ping.c:295:rpc_clnt_start_ping]
> >>>>0-home-lab-3-client-0: returning as transport is already
> >>>>disconnected OR
> >>>>there are no frames (0 || 0)
> >>>>[2016-11-18 16:07:31.794925] D
> >>>>[rpc-clnt-ping.c:295:rpc_clnt_start_ping]
> >>>>0-home-lab-3-client-1: returning as transport is already
> >>>>disconnected OR
> >>>>there are no frames (0 || 0)
> >>>>[2016-11-18 16:07:31.794984] D
> >>>>[rpc-clnt-ping.c:295:rpc_clnt_start_ping]
> >>>>0-home-lab-3-client-3: returning as transport is already
> >>>>disconnected OR
> >>>>there are no frames (0 || 0)
> >>>>
> >>>>
> >>>>** USER ROOT EXITED at this point
> >>>>
> >>>>El 18/11/16 a las 15:16, Miklos Szeredi escribió:
> >>>>>On Wed, Nov 16, 2016 at 7:19 PM, Omar Walid Llorente
> >>>>><[email protected]>
> >>>>>wrote:
> >>>>>
> >>>>>>In any case, the kernel we are using (3.13.0-96-generic #143-Ubuntu
> >>>>>>SMP Mon
> >>>>>>Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux) has fuse
> >>>>>>included
> >>>>>>(not as a module) so if it would be necessary to change
> >>>>>>the fuse mount
> >>>>>>options some help would be needed.
> >>>>>Can you reproduce this on latest (e.g 4.8) kernel?
> >>>>>
> >>>>>Also fuse debug logs (with whaterver kernel) could be
> >>>>>useful; not sure
> >>>>>how to enable it for glusterfs, usual convention for fuse filesystems
> >>>>>is "-d".
> >>>>>
> >>>>>Thanks,
> >>>>>Miklos
> >>>>
> >>--
> >>To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> >>the body of a message to [email protected]
> >>More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>
>
> --
> ----------------------------------------------------------------
> Centro de Cálculo Depto. Ingeniería Sistemas Telemáticos
> E-mail: [email protected] Universidad Politécnica de Madrid
> Fax:(+34) 913367333 E.T.S. Ing. Telecomunicación
> Tel:(+34) 915495700-Ext.3005 28040 Madrid (Spain)
> Tel:(+34) 915495762-Ext.3005
> Tel:(+34) 913367366-Ext.3005
> ----------------------------------------------------------------

2016-12-15 17:09:19

[permalink] [raw]

Subject: Re: possible bug in nfs-kernel-server

Thanks Bruce, all.

I just filed a bug in bugzilla.redhat.com.

See https://bugzilla.redhat.com/show_bug.cgi?id=1405147

I hope that soon we have this problem solved.

Omar

CONTENT of bugzilla bug report:

Description of problem:
When using a fuse-mounted glusterfs volume, an error is found if the
owner of a file (that is not root) tries to write in its own file when
it has read-only permissions.

The problem does not exist if the user is root.

Version-Release number of selected component (if applicable):
All versions tested (glusterfs-3.7.{4-16}, glusterfs-3.8.{4-5})

How reproducible:
Always

Steps to Reproduce:

1. Fuse mount a glusterfs volume to be the home of the user, cd to home
dir and check that the file we will use for testing does not exist:
user@computer:~$ ls -l 444.txt
ls: cannot access 444.txt: No such file or directory
user@computer:~$

2. Execute command (error comes at echo, not before):
user@computer:~$ rm -f 444.txt; touch 444.txt; chmod 444 444.txt; echo
test > 444.txt
-bash: 444.txt: Permission denied

3. Check result: File is empty.
user@computer:~$ ls -l 444.txt
-r--r--r-- 1 user user 0 dic 15 2016 444.txt
user@computer:~$ rm -f 444.txt
user@computer:~$

4. Check that the problem dissapears with root privileges:
user@computer:~$ sudo su
root@computer:/home/user# rm -f 444.txt; touch 444.txt; chmod 444
444.txt; echo test > 444.txt
root@computer:/home/user# ls -l 444.txt
-r--r--r-- 1 root root 5 dic 15 2016 444.txt
root@computer:/home/user# rm -f 444.txt
root@computer:/home/user#

Actual results:
A user who owns a file that is read-only cannot write into it.

Expected results:
A user who owns a file that is read-only should be able to write into it
even if it is read-only.

Additional info:
Special thanks to Bruce Fields and other components of nfs-kernel-server
team because initially we thought it was their fault. The same to Soumya
Koduri and Miklos Szeredi, that helped to confirm the problem.

See full previous thread on nfs-kernel-server list with history about
the issue:
http://marc.info/?t=144801803400001 (for posterity's sake, permanent
link: http://marc.info/[email protected]).

El 28/11/16 a las 19:25, J. Bruce Fields escribió:
> On Mon, Nov 28, 2016 at 07:03:48PM +0100, Omar Walid Llorente wrote:
>> Hi Soumya, all.
>>
>> Sorry for the delay answering.
>>
>> Following your sugestion, we are planning to file a bug at bugzilla
>> [1], but in order to minimize the content of the explanation, we
>> wanted to include a link to the file archive of this
>> nfs-kernel-server list, but we only find a page that doesn't work
>> (from http://dir.gmane.org/gmane.linux.nfs we go to
>> http://news.gmane.org/gmane.linux.nfs, but it gives a "Page Not
>> Found" answer).
>>
>> Does anybody know if is this link available anywhere else or if it
>> can be repaired?
>>
>> Thanks a lot to everyone. We hope we can file the bug this week.
> http://marc.info/?t=144801803400001&r=1&w=2
>
> Or you can link by message id like:
>
> http://marc.info/[email protected]
>
> which I sometimes try to for posterity's sake since the message id
> should still be useful even if the one archive dies.
>
> --b.
>
>
>> Omar
>>
>> El 22/11/16 a las 15:45, Soumya Koduri escribió:
>>>
>>> On 11/21/2016 08:27 PM, J. Bruce Fields wrote:
>>>> On Mon, Nov 21, 2016 at 06:26:16PM +0530, Soumya Koduri wrote:
>>>>> Hi,
>>>>>
>>>>> Sorry for the delay. I could reproduce the issue on my gluster setup
>>>>> as well. The error is generated at posix-acl xlator layer of
>>>>> glusterfs server.
>>>>>
>>>>> As Bruce mentioned in the earlier mail, as part of WRITE call,
>>>>> NFS-server issues OPEN() with O_WRONLY flag set. Since the
>>>>> attributes of the file "kk.txt" when created was 0400, the
>>>>> corresponding acl maps to below ACL -
>>>>>
>>>>> [root@dhcp35-197 vol1]# getfacl kk.txt
>>>>> # file: kk.txt
>>>>> # owner: guest
>>>>> # group: mock
>>>>> user::r--
>>>>> group::---
>>>>> other::---
>>>>>
>>>>> And hence the permission check for WRITE access fails. Also not just
>>>>> glusterfs, I see similar behavior with ext4 filesystem as well when
>>>>> exported via kernel-NFS server.
>>>> That definitely shouldn't be happening, how are you reproducing?
>>> Sorry. Yesterday when I was testing, both the exports (one on
>>> glusterfs fuse mount and another on ext4) had same fsid. So when I
>>> was trying to mount and access the share on ext4, it had
>>> redirected to glusterfs fuse mount and hence the test failed.
>>>
>>> I re-checked with ext4 and I do not see any failures. So probably
>>> then we need to put a fix in glusterfs to bypass this check for
>>> file owner during OPEN.
>>>
>>> @Omar,
>>> Could you please file a bug against glusterfs component in RH
>>> bugzilla [1] for the same?
>>>
>>> Thanks,
>>> Soumya
>>>
>>> [1] https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
>>>
>>>> --b.
>>>>
>>>>> So I am not sure if fixing glusterfs
>>>>> for this use-case is valid.
>>>>>
>>>>> Thanks,
>>>>> Soumya
>>>>>
>>>>>
>>>>> On 11/18/2016 09:33 PM, Omar Walid Llorente wrote:
>>>>>> Yes, of course, we can reproduce the problem on latest (e.g
>>>>>> 4.8) kernel.
>>>>>>
>>>>>> Tested on a server with fedora 24 (Linux cuentas11.lab.dit.upm.es
>>>>>> 4.8.4-200.fc24.x86_64 #1 SMP Tue Oct 25 13:06:04 UTC 2016
>>>>>> x86_64 x86_64
>>>>>> x86_64 GNU/Linux), glusterfs-fuse-3.8.5-1.fc24.x86_64,
>>>>>> fuse-2.9.7-1.fc24.x86_64 and nfs-utils-1.3.4-1.rc2.fc24.x86_64 that
>>>>>> mounts as a glusterfs-client and exports as a nfs-server a ditributed
>>>>>> glusterfs volume (that comes from a 3.7.4 glusterfs server over Ubuntu
>>>>>> 14.04.
>>>>>>
>>>>>> [root@cuentas11 ~]# cat /proc/mounts | grep home
>>>>>> recipiente8hd:/home-lab-3.tcp /home-3 fuse.glusterfs
>>>>>> rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072
>>>>>>
>>>>>> 0 0
>>>>>> [root@cuentas11 ~]#
>>>>>>
>>>>>> As you may remember the schema is the one that follows
>>>>>> (glusterfs volume
>>>>>> name: home-lab-3, glusterfs client mount point: /home-3, nfs client
>>>>>> mount point: /home):
>>>>>>
>>>>>> nfs client (v1.2.8) ---> NFSv3 ---> knfsd (v1.3.4 over a Linux v4.8.4
>>>>>> kernel) ---> FUSE (v2.9.7) --> glusterfs client (v3.8.5) --> GLUSTER
>>>>>> over TCP --> glusterfs server (v3.7.4)
>>>>>>
>>>>>> The errors at the nfs-client side are the same:
>>>>>>
>>>>>> cdc@l056:~$ mount | grep "/home/cdc"
>>>>>> cuentas11:/home-3/cdc on /home/cdc type nfs
>>>>>> (rw,noatime,nfsvers=3,vers=3,tcp,intr,fsc,nolock,rsize=262140,wsize=262140,addr=138.4.30.86)
>>>>>>
>>>>>>
>>>>>> cdc@l056:~$
>>>>>> cdc@l056:~$
>>>>>> cdc@l056:~$
>>>>>> cdc@l056:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444
>>>>>> 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
>>>>>> cp: failed to close ‘kk.txt’: Permission denied
>>>>>> -r--r--r-- 1 cdc admincdc 7 nov 18 2016 444.txt
>>>>>> -r--r--r-- 1 cdc admincdc 0 nov 18 2016 kk.txt
>>>>>> cdc@l056:~$
>>>>>> cdc@l056:~$
>>>>>> cdc@l056:~$
>>>>>> cdc@l056:~$ sudo su
>>>>>> [sudo] password for cdc:
>>>>>> root@l056:/home/cdc# rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
>>>>>> chmod 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
>>>>>> -r--r--r-- 1 root root 7 nov 18 2016 444.txt
>>>>>> -r--r--r-- 1 root root 7 nov 18 2016 kk.txt
>>>>>> root@l056:/home/cdc#
>>>>>> root@l056:/home/cdc#
>>>>>> root@l056:/home/cdc# exit
>>>>>> cdc@l056:~$
>>>>>>
>>>>>> See at the end the logs with and without the debug option activated at
>>>>>> the glusterfs client.
>>>>>>
>>>>>> Thanks again, and sorry for the long message.
>>>>>>
>>>>>> Omar
>>>>>>
>>>>>>
>>>>>>
>>>>>> * Logs on the glusterfs client side with NO DEBUG:
>>>>>>
>>>>>> [2016-11-18 15:45:43.681215] E [MSGID: 114031]
>>>>>> [client-rpc-fops.c:444:client3_3_open_cbk]
>>>>>> 0-home-lab-3-client-3: remote
>>>>>> operation failed. Path: /cdc/kk.txt
>>>>>> (cc98e831-3410-4202-8254-2d4b48e8ea68) [Permission denied]
>>>>>> [2016-11-18 15:45:43.681297] W [fuse-bridge.c:989:fuse_fd_cbk]
>>>>>> 0-glusterfs-fuse: 399: OPEN() /cdc/kk.txt => -1 (Permission denied)
>>>>>> [2016-11-18 15:45:55.457922] I [io-stats.c:1574:io_stats_dump_fd]
>>>>>> 0-home-lab-3: --- fd stats ---
>>>>>> [2016-11-18 15:45:55.457978] I [io-stats.c:1579:io_stats_dump_fd]
>>>>>> 0-home-lab-3: Filename : /cdc/444.txt
>>>>>> [2016-11-18 15:45:55.457997] I [io-stats.c:1594:io_stats_dump_fd]
>>>>>> 0-home-lab-3: BytesWritten : 7 bytes
>>>>>> [2016-11-18 15:45:55.458011] I [io-stats.c:1606:io_stats_dump_fd]
>>>>>> 0-home-lab-3: Write 000004b+ : 1
>>>>>> [2016-11-18 15:45:55.471933] I [io-stats.c:1574:io_stats_dump_fd]
>>>>>> 0-home-lab-3: --- fd stats ---
>>>>>>
>>>>>>
>>>>>> * Logs on the glusterfs client side with DEBUG activated:
>>>>>>
>>>>>> [2016-11-18 16:06:17.040092] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
>>>>>> STAT scheduled
>>>>>> as fast fop
>>>>>> [2016-11-18 16:06:17.045986] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>>>>> scheduled as fast fop
>>>>>> [2016-11-18 16:06:17.046132] D [MSGID: 0]
>>>>>> [dht-common.c:2684:dht_lookup]
>>>>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
>>>>>> home-lab-3-client-3
>>>>>> [2016-11-18 16:06:17.046833] D [MSGID: 0]
>>>>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>>>>> returned for /cdc/kk.txt with op_ret 0 [Invalid argument]
>>>>>> [2016-11-18 16:06:17.047671] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK
>>>>>> scheduled as normal fop
>>>>>> [2016-11-18 16:06:17.050539] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>>>> event 19 on graph -1418558531
>>>>>> [2016-11-18 16:06:17.050867] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>>>> event 19 on graph 0
>>>>>> [2016-11-18 16:06:17.051948] D [logging.c:1953:_gf_msg_internal]
>>>>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>>>>> About to flush least recently used log message to disk
>>>>>> [2016-11-18 16:06:17.051871] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>>>>> scheduled as fast fop
>>>>>> [2016-11-18 16:06:17.051946] D [MSGID: 0]
>>>>>> [dht-common.c:2684:dht_lookup]
>>>>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
>>>>>> home-lab-3-client-0
>>>>>> [2016-11-18 16:06:17.062108] D [MSGID: 0]
>>>>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>>>>> returned for /cdc/444.txt with op_ret 0 [Invalid argument]
>>>>>> [2016-11-18 16:06:17.063692] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>>>> event 19 on graph -1418558531
>>>>>> [2016-11-18 16:06:17.064015] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>>>> event 19 on graph 0
>>>>>> [2016-11-18 16:06:17.065068] D [logging.c:1953:_gf_msg_internal]
>>>>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>>>>> About to flush least recently used log message to disk
>>>>>> The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
>>>>>> 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 4 times
>>>>>> between [2016-11-18 16:06:17.040092] and [2016-11-18 16:06:17.064175]
>>>>>> [2016-11-18 16:06:17.065066] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>>>>> scheduled as fast fop
>>>>>> [2016-11-18 16:06:17.065232] D [MSGID: 0]
>>>>>> [dht-common.c:2684:dht_lookup]
>>>>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
>>>>>> home-lab-3-client-0
>>>>>> [2016-11-18 16:06:17.065755] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.065897] D [MSGID: 0]
>>>>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>>>>> returned for /cdc/444.txt with op_ret -1 [No such file or directory]
>>>>>> [2016-11-18 16:06:17.066016] D [MSGID: 0]
>>>>>> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry
>>>>>> /cdc/444.txt
>>>>>> missing on subvol home-lab-3-client-0
>>>>>> [2016-11-18 16:06:17.066104] D [MSGID: 0]
>>>>>> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
>>>>>> lookup call to 4 subvols
>>>>>> [2016-11-18 16:06:17.066594] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.066669] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>>>> home-lab-3-client-0
>>>>>> [2016-11-18 16:06:17.066771] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.066898] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>>>> home-lab-3-client-1
>>>>>> [2016-11-18 16:06:17.066906] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.067019] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>>>> home-lab-3-client-2
>>>>>> [2016-11-18 16:06:17.066915] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.067066] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>>>> home-lab-3-client-3
>>>>>> [2016-11-18 16:06:17.067107] D [MSGID: 0]
>>>>>> [dht-common.c:1527:dht_lookup_everywhere_done]
>>>>>> 0-home-lab-3-dht: STATUS:
>>>>>> hashed_subvol home-lab-3-client-0 cached_subvol null
>>>>>> [2016-11-18 16:06:17.067151] D [MSGID: 0]
>>>>>> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
>>>>>> was no cached file and unlink on hashed is not skipped /cdc/444.txt
>>>>>> [2016-11-18 16:06:17.067179] D [MSGID: 0]
>>>>>> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1
>>>>>> error: No such
>>>>>> file or directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.067206] D [MSGID: 0]
>>>>>> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.067250] D [MSGID: 0]
>>>>>> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No
>>>>>> such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.067275] D [MSGID: 0]
>>>>>> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.067308] D [MSGID: 0]
>>>>>> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No
>>>>>> such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.067387] D [MSGID: 0]
>>>>>> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file
>>>>>> or directory
>>>>>> [No such file or directory]
>>>>>> [2016-11-18 16:06:17.068157] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>>>>> scheduled as fast fop
>>>>>> [2016-11-18 16:06:17.068281] D [MSGID: 0]
>>>>>> [dht-common.c:2684:dht_lookup]
>>>>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
>>>>>> home-lab-3-client-0
>>>>>> [2016-11-18 16:06:17.068738] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.068781] D [MSGID: 0]
>>>>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>>>>> returned for /cdc/444.txt with op_ret -1 [No such file or directory]
>>>>>> [2016-11-18 16:06:17.068805] D [MSGID: 0]
>>>>>> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry
>>>>>> /cdc/444.txt
>>>>>> missing on subvol home-lab-3-client-0
>>>>>> [2016-11-18 16:06:17.068827] D [MSGID: 0]
>>>>>> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
>>>>>> lookup call to 4 subvols
>>>>>> [2016-11-18 16:06:17.069286] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.069345] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>>>> home-lab-3-client-0
>>>>>> [2016-11-18 16:06:17.069452] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.069507] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>>>> home-lab-3-client-2
>>>>>> [2016-11-18 16:06:17.069556] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.069585] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>>>> home-lab-3-client-3
>>>>>> [2016-11-18 16:06:17.069939] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.069983] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>>>> home-lab-3-client-1
>>>>>> [2016-11-18 16:06:17.070005] D [MSGID: 0]
>>>>>> [dht-common.c:1527:dht_lookup_everywhere_done]
>>>>>> 0-home-lab-3-dht: STATUS:
>>>>>> hashed_subvol home-lab-3-client-0 cached_subvol null
>>>>>> [2016-11-18 16:06:17.070038] D [MSGID: 0]
>>>>>> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
>>>>>> was no cached file and unlink on hashed is not skipped /cdc/444.txt
>>>>>> [2016-11-18 16:06:17.070098] D [MSGID: 0]
>>>>>> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1
>>>>>> error: No such
>>>>>> file or directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.070128] D [MSGID: 0]
>>>>>> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.070154] D [MSGID: 0]
>>>>>> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No
>>>>>> such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.070178] D [MSGID: 0]
>>>>>> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.070208] D [MSGID: 0]
>>>>>> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No
>>>>>> such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.070251] D [MSGID: 0]
>>>>>> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file
>>>>>> or directory
>>>>>> [No such file or directory]
>>>>>> [2016-11-18 16:06:17.070537] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
>>>>>> MKNOD scheduled
>>>>>> as normal fop
>>>>>> [2016-11-18 16:06:17.070826] D [MSGID: 0]
>>>>>> [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
>>>>>> creating /cdc/444.txt on home-lab-3-client-0
>>>>>> [2016-11-18 16:06:17.071100] D [MSGID: 0]
>>>>>> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
>>>>>> 'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
>>>>>> 10456189239296 and avail_inodes is: 99.00
>>>>>> [2016-11-18 16:06:17.071142] D [MSGID: 0]
>>>>>> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
>>>>>> 'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
>>>>>> 10499693871104 and avail_inodes is: 99.00
>>>>>> [2016-11-18 16:06:17.071183] D [MSGID: 0]
>>>>>> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
>>>>>> 'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
>>>>>> 10500911267840 and avail_inodes is: 99.00
>>>>>> [2016-11-18 16:06:17.071343] D [MSGID: 0]
>>>>>> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
>>>>>> 'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
>>>>>> 8600432410624 and avail_inodes is: 99.00
>>>>>> [2016-11-18 16:06:17.073943] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>>>> event 19 on graph -1418558531
>>>>>> [2016-11-18 16:06:17.074273] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
>>>>>> STAT scheduled
>>>>>> as fast fop
>>>>>> [2016-11-18 16:06:17.075035] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
>>>>>> OPEN scheduled
>>>>>> as fast fop
>>>>>> [2016-11-18 16:06:17.075913] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
>>>>>> WRITE scheduled
>>>>>> as slow fop
>>>>>> [2016-11-18 16:06:17.076205] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
>>>>>> FSTAT scheduled
>>>>>> as fast fop
>>>>>> [2016-11-18 16:06:17.076636] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>>>> event 19 on graph 0
>>>>>> [2016-11-18 16:06:17.077282] D [fuse-bridge.c:52:fuse_invalidate]
>>>>>> 0-fuse: Invalidate inode id 140363155120276.
>>>>>> [2016-11-18 16:06:17.077981] I [io-stats.c:1574:io_stats_dump_fd]
>>>>>> 0-home-lab-3: --- fd stats ---
>>>>>> [2016-11-18 16:06:17.078032] I [io-stats.c:1579:io_stats_dump_fd]
>>>>>> 0-home-lab-3: Filename : /cdc/444.txt
>>>>>> [2016-11-18 16:06:17.078049] I [io-stats.c:1594:io_stats_dump_fd]
>>>>>> 0-home-lab-3: BytesWritten : 7 bytes
>>>>>> [2016-11-18 16:06:17.078064] I [io-stats.c:1606:io_stats_dump_fd]
>>>>>> 0-home-lab-3: Write 000004b+ : 1
>>>>>> [2016-11-18 16:06:17.079156] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
>>>>>> scheduled as normal fop
>>>>>> [2016-11-18 16:06:17.079794] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>>>> event 19 on graph 0
>>>>>> [2016-11-18 16:06:17.082196] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>>>>> scheduled as fast fop
>>>>>> [2016-11-18 16:06:17.082772] D [logging.c:1953:_gf_msg_internal]
>>>>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>>>>> About to flush least recently used log message to disk
>>>>>> [2016-11-18 16:06:17.082318] D [MSGID: 0]
>>>>>> [dht-common.c:2684:dht_lookup]
>>>>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
>>>>>> home-lab-3-client-3
>>>>>> [2016-11-18 16:06:17.082770] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.083168] D [MSGID: 0]
>>>>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>>>>> returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
>>>>>> [2016-11-18 16:06:17.083282] D [MSGID: 0]
>>>>>> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
>>>>>> missing on subvol home-lab-3-client-3
>>>>>> [2016-11-18 16:06:17.083320] D [MSGID: 0]
>>>>>> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
>>>>>> lookup call to 4 subvols
>>>>>> [2016-11-18 16:06:17.083991] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.084043] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>>>> home-lab-3-client-0
>>>>>> [2016-11-18 16:06:17.084048] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.084092] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>>>> home-lab-3-client-3
>>>>>> [2016-11-18 16:06:17.084114] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.084167] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>>>> home-lab-3-client-1
>>>>>> [2016-11-18 16:06:17.084091] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.084377] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>>>> home-lab-3-client-2
>>>>>> [2016-11-18 16:06:17.084406] D [MSGID: 0]
>>>>>> [dht-common.c:1527:dht_lookup_everywhere_done]
>>>>>> 0-home-lab-3-dht: STATUS:
>>>>>> hashed_subvol home-lab-3-client-3 cached_subvol null
>>>>>> [2016-11-18 16:06:17.084511] D [MSGID: 0]
>>>>>> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
>>>>>> was no cached file and unlink on hashed is not skipped /cdc/kk.txt
>>>>>> [2016-11-18 16:06:17.084534] D [MSGID: 0]
>>>>>> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1
>>>>>> error: No such
>>>>>> file or directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.084683] D [MSGID: 0]
>>>>>> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.084783] D [MSGID: 0]
>>>>>> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No
>>>>>> such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.084857] D [MSGID: 0]
>>>>>> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.084889] D [MSGID: 0]
>>>>>> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No
>>>>>> such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.085001] D [MSGID: 0]
>>>>>> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file
>>>>>> or directory
>>>>>> [No such file or directory]
>>>>>> [2016-11-18 16:06:17.086098] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
>>>>>> STAT scheduled
>>>>>> as fast fop
>>>>>> [2016-11-18 16:06:17.087256] D [logging.c:1953:_gf_msg_internal]
>>>>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>>>>> About to flush least recently used log message to disk
>>>>>> [2016-11-18 16:06:17.087173] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>>>>> scheduled as fast fop
>>>>>> [2016-11-18 16:06:17.087254] D [MSGID: 0]
>>>>>> [dht-common.c:2684:dht_lookup]
>>>>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
>>>>>> home-lab-3-client-3
>>>>>> [2016-11-18 16:06:17.087749] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.087795] D [MSGID: 0]
>>>>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>>>>> returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
>>>>>> [2016-11-18 16:06:17.088014] D [MSGID: 0]
>>>>>> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
>>>>>> missing on subvol home-lab-3-client-3
>>>>>> [2016-11-18 16:06:17.088036] D [MSGID: 0]
>>>>>> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
>>>>>> lookup call to 4 subvols
>>>>>> [2016-11-18 16:06:17.088598] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.088639] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>>>> home-lab-3-client-0
>>>>>> [2016-11-18 16:06:17.088804] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.088855] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>>>> home-lab-3-client-2
>>>>>> [2016-11-18 16:06:17.088819] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.089069] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>>>> home-lab-3-client-3
>>>>>> [2016-11-18 16:06:17.089380] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.089421] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>>>> home-lab-3-client-1
>>>>>> [2016-11-18 16:06:17.089443] D [MSGID: 0]
>>>>>> [dht-common.c:1527:dht_lookup_everywhere_done]
>>>>>> 0-home-lab-3-dht: STATUS:
>>>>>> hashed_subvol home-lab-3-client-3 cached_subvol null
>>>>>> [2016-11-18 16:06:17.089463] D [MSGID: 0]
>>>>>> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
>>>>>> was no cached file and unlink on hashed is not skipped /cdc/kk.txt
>>>>>> [2016-11-18 16:06:17.089575] D [MSGID: 0]
>>>>>> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1
>>>>>> error: No such
>>>>>> file or directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.089601] D [MSGID: 0]
>>>>>> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.089712] D [MSGID: 0]
>>>>>> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No
>>>>>> such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.089738] D [MSGID: 0]
>>>>>> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.090006] D [MSGID: 0]
>>>>>> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No
>>>>>> such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:06:17.090156] D [MSGID: 0]
>>>>>> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file
>>>>>> or directory
>>>>>> [No such file or directory]
>>>>>> [2016-11-18 16:06:17.090648] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
>>>>>> MKNOD scheduled
>>>>>> as normal fop
>>>>>> [2016-11-18 16:06:17.090732] D [MSGID: 0]
>>>>>> [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
>>>>>> creating /cdc/kk.txt on home-lab-3-client-3
>>>>>> [2016-11-18 16:06:17.091499] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>>>> event 19 on graph -1418558531
>>>>>> [2016-11-18 16:06:17.091843] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
>>>>>> scheduled as normal fop
>>>>>> [2016-11-18 16:06:17.093831] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>>>> event 19 on graph 0
>>>>>> [2016-11-18 16:06:17.094711] D [logging.c:1953:_gf_msg_internal]
>>>>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>>>>> About to flush least recently used log message to disk
>>>>>> [2016-11-18 16:06:17.093216] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
>>>>>> scheduled as normal fop
>>>>>> [2016-11-18 16:06:17.094709] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
>>>>>> OPEN scheduled
>>>>>> as fast fop
>>>>>> [2016-11-18 16:06:17.095280] E [MSGID: 114031]
>>>>>> [client-rpc-fops.c:444:client3_3_open_cbk]
>>>>>> 0-home-lab-3-client-3: remote
>>>>>> operation failed. Path: /cdc/kk.txt
>>>>>> (09bfb6fa-82fb-463b-abd9-bb7ef88e10c0) [Permission denied]
>>>>>> [2016-11-18 16:06:17.095341] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:448:client3_3_open_cbk] 0-stack-trace:
>>>>>> stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error:
>>>>>> Permission denied
>>>>>> [Permission denied]
>>>>>> [2016-11-18 16:06:17.095366] D [MSGID: 0]
>>>>>> [dht-inode-read.c:45:dht_open_cbk] 0-home-lab-3-dht: subvolume
>>>>>> home-lab-3-client-3 returned -1 [Permission denied]
>>>>>> [2016-11-18 16:06:17.095480] D [MSGID: 0]
>>>>>> [dht-inode-read.c:59:dht_open_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: Permission denied
>>>>>> [Permission denied]
>>>>>> [2016-11-18 16:06:17.095671] D [MSGID: 0]
>>>>>> [read-ahead.c:101:ra_open_cbk]
>>>>>> 0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-read-ahead
>>>>>> returned -1 error: Permission denied [Permission denied]
>>>>>> [2016-11-18 16:06:17.095759] D [MSGID: 0]
>>>>>> [io-cache.c:605:ioc_open_cbk]
>>>>>> 0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-io-cache
>>>>>> returned -1 error: Permission denied [Permission denied]
>>>>>> [2016-11-18 16:06:17.095963] D [MSGID: 0]
>>>>>> [defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: Permission
>>>>>> denied [Permission denied]
>>>>>> [2016-11-18 16:06:17.095988] D [MSGID: 0]
>>>>>> [defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: Permission
>>>>>> denied [Permission denied]
>>>>>> [2016-11-18 16:06:17.096275] D [MSGID: 0]
>>>>>> [io-stats.c:1936:io_stats_open_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3 returned -1 error: Permission denied
>>>>>> [Permission denied]
>>>>>> [2016-11-18 16:06:17.096362] W [fuse-bridge.c:989:fuse_fd_cbk]
>>>>>> 0-glusterfs-fuse: 338: OPEN() /cdc/kk.txt => -1 (Permission denied)
>>>>>> [2016-11-18 16:06:17.097104] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
>>>>>> scheduled as normal fop
>>>>>> [2016-11-18 16:06:17.097674] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>>>> event 19 on graph 0
>>>>>> [2016-11-18 16:06:17.099912] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>>>> event 19 on graph 0
>>>>>> [2016-11-18 16:06:24.785225] D
>>>>>> [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
>>>>>> 0-home-lab-3-client-0: returning as transport is already
>>>>>> disconnected OR
>>>>>> there are no frames (0 || 0)
>>>>>> [2016-11-18 16:06:24.785338] D
>>>>>> [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
>>>>>> 0-home-lab-3-client-1: returning as transport is already
>>>>>> disconnected OR
>>>>>> there are no frames (0 || 0)
>>>>>> [2016-11-18 16:06:24.785359] D
>>>>>> [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
>>>>>> 0-home-lab-3-client-2: returning as transport is already
>>>>>> disconnected OR
>>>>>> there are no frames (0 || 0)
>>>>>> [2016-11-18 16:06:24.785377] D
>>>>>> [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
>>>>>> 0-home-lab-3-client-3: returning as transport is already
>>>>>> disconnected OR
>>>>>> there are no frames (0 || 0)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ** TEST DONE as user cdc --> ERROR
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> [2016-11-18 16:06:46.737686] D
>>>>>> [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
>>>>>> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
>>>>>> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
>>>>>>
>>>>>> (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
>>>>>> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
>>>>>> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
>>>>>>
>>>>>> ))))) 0-: 138.4.2.246:49172: ping timer event already removed
>>>>>> [2016-11-18 16:06:49.545698] D
>>>>>> [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
>>>>>> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
>>>>>> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
>>>>>>
>>>>>> (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
>>>>>> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
>>>>>> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
>>>>>>
>>>>>> ))))) 0-: 138.4.2.177:49157: ping timer event already removed
>>>>>> [2016-11-18 16:06:49.546474] D
>>>>>> [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
>>>>>> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
>>>>>> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
>>>>>>
>>>>>> (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
>>>>>> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
>>>>>> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
>>>>>>
>>>>>> ))))) 0-: 138.4.2.189:49157: ping timer event already removed
>>>>>> [2016-11-18 16:06:49.547093] D
>>>>>> [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
>>>>>> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (-->
>>>>>> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]
>>>>>>
>>>>>> (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (-->
>>>>>> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (-->
>>>>>> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]
>>>>>>
>>>>>> ))))) 0-: 138.4.2.248:49157: ping timer event already removed
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ** user CDC becomed root at this point
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> [2016-11-18 16:07:11.907892] D [logging.c:1953:_gf_msg_internal]
>>>>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>>>>> About to flush least recently used log message to disk
>>>>>> The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
>>>>>> 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 7 times
>>>>>> between [2016-11-18 16:06:17.086098] and [2016-11-18 16:07:11.906944]
>>>>>> [2016-11-18 16:07:11.907884] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>>>>> scheduled as fast fop
>>>>>> [2016-11-18 16:07:11.908875] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>>>> event 19 on graph -1418558531
>>>>>> [2016-11-18 16:07:11.909179] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>>>> event 19 on graph 0
>>>>>> [2016-11-18 16:07:11.909256] D [logging.c:1953:_gf_msg_internal]
>>>>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>>>>> About to flush least recently used log message to disk
>>>>>> The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
>>>>>> 0-home-lab-3-io-threads: UNLINK scheduled as normal fop" repeated 2
>>>>>> times between [2016-11-18 16:06:17.047671] and [2016-11-18
>>>>>> 16:07:11.908114]
>>>>>> [2016-11-18 16:07:11.909254] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
>>>>>> STAT scheduled
>>>>>> as fast fop
>>>>>> [2016-11-18 16:07:11.911109] D [logging.c:1953:_gf_msg_internal]
>>>>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>>>>> About to flush least recently used log message to disk
>>>>>> [2016-11-18 16:07:11.911018] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>>>>> scheduled as fast fop
>>>>>> [2016-11-18 16:07:11.911107] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK
>>>>>> scheduled as normal fop
>>>>>> [2016-11-18 16:07:11.911871] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>>>> event 19 on graph -1418558531
>>>>>> [2016-11-18 16:07:11.912045] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>>>> event 19 on graph 0
>>>>>> [2016-11-18 16:07:11.913067] D [logging.c:1953:_gf_msg_internal]
>>>>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>>>>> About to flush least recently used log message to disk
>>>>>> The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
>>>>>> 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 2 times
>>>>>> between [2016-11-18 16:07:11.909254] and [2016-11-18 16:07:11.912165]
>>>>>> [2016-11-18 16:07:11.913065] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>>>>> scheduled as fast fop
>>>>>> [2016-11-18 16:07:11.913193] D [MSGID: 0]
>>>>>> [dht-common.c:2684:dht_lookup]
>>>>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
>>>>>> home-lab-3-client-0
>>>>>> [2016-11-18 16:07:11.913710] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.913757] D [MSGID: 0]
>>>>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>>>>> returned for /cdc/444.txt with op_ret -1 [No such file or directory]
>>>>>> [2016-11-18 16:07:11.913788] D [MSGID: 0]
>>>>>> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry
>>>>>> /cdc/444.txt
>>>>>> missing on subvol home-lab-3-client-0
>>>>>> [2016-11-18 16:07:11.913810] D [MSGID: 0]
>>>>>> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
>>>>>> lookup call to 4 subvols
>>>>>> [2016-11-18 16:07:11.914336] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.914379] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>>>> home-lab-3-client-0
>>>>>> [2016-11-18 16:07:11.914424] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.914448] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>>>> home-lab-3-client-1
>>>>>> [2016-11-18 16:07:11.914501] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.914617] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>>>> home-lab-3-client-3
>>>>>> [2016-11-18 16:07:11.914649] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.914698] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>>>> home-lab-3-client-2
>>>>>> [2016-11-18 16:07:11.914724] D [MSGID: 0]
>>>>>> [dht-common.c:1527:dht_lookup_everywhere_done]
>>>>>> 0-home-lab-3-dht: STATUS:
>>>>>> hashed_subvol home-lab-3-client-0 cached_subvol null
>>>>>> [2016-11-18 16:07:11.914743] D [MSGID: 0]
>>>>>> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
>>>>>> was no cached file and unlink on hashed is not skipped /cdc/444.txt
>>>>>> [2016-11-18 16:07:11.914764] D [MSGID: 0]
>>>>>> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1
>>>>>> error: No such
>>>>>> file or directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.914791] D [MSGID: 0]
>>>>>> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.914825] D [MSGID: 0]
>>>>>> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No
>>>>>> such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.914855] D [MSGID: 0]
>>>>>> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.914880] D [MSGID: 0]
>>>>>> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No
>>>>>> such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.914947] D [MSGID: 0]
>>>>>> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file
>>>>>> or directory
>>>>>> [No such file or directory]
>>>>>> [2016-11-18 16:07:11.915569] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>>>>> scheduled as fast fop
>>>>>> [2016-11-18 16:07:11.915644] D [MSGID: 0]
>>>>>> [dht-common.c:2684:dht_lookup]
>>>>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on
>>>>>> home-lab-3-client-0
>>>>>> [2016-11-18 16:07:11.916090] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.916134] D [MSGID: 0]
>>>>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>>>>> returned for /cdc/444.txt with op_ret -1 [No such file or directory]
>>>>>> [2016-11-18 16:07:11.916158] D [MSGID: 0]
>>>>>> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry
>>>>>> /cdc/444.txt
>>>>>> missing on subvol home-lab-3-client-0
>>>>>> [2016-11-18 16:07:11.916185] D [MSGID: 0]
>>>>>> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
>>>>>> lookup call to 4 subvols
>>>>>> [2016-11-18 16:07:11.916593] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.916780] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>>>> home-lab-3-client-0
>>>>>> [2016-11-18 16:07:11.916878] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.916994] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>>>> home-lab-3-client-3
>>>>>> [2016-11-18 16:07:11.917076] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.916906] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.917117] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>>>> home-lab-3-client-2
>>>>>> [2016-11-18 16:07:11.917153] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol
>>>>>> home-lab-3-client-1
>>>>>> [2016-11-18 16:07:11.917205] D [MSGID: 0]
>>>>>> [dht-common.c:1527:dht_lookup_everywhere_done]
>>>>>> 0-home-lab-3-dht: STATUS:
>>>>>> hashed_subvol home-lab-3-client-0 cached_subvol null
>>>>>> [2016-11-18 16:07:11.917368] D [MSGID: 0]
>>>>>> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
>>>>>> was no cached file and unlink on hashed is not skipped /cdc/444.txt
>>>>>> [2016-11-18 16:07:11.917437] D [MSGID: 0]
>>>>>> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1
>>>>>> error: No such
>>>>>> file or directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.917467] D [MSGID: 0]
>>>>>> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.917496] D [MSGID: 0]
>>>>>> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No
>>>>>> such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.917521] D [MSGID: 0]
>>>>>> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.917551] D [MSGID: 0]
>>>>>> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No
>>>>>> such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.917726] D [MSGID: 0]
>>>>>> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file
>>>>>> or directory
>>>>>> [No such file or directory]
>>>>>> [2016-11-18 16:07:11.917958] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
>>>>>> MKNOD scheduled
>>>>>> as normal fop
>>>>>> [2016-11-18 16:07:11.918230] D [MSGID: 0]
>>>>>> [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
>>>>>> creating /cdc/444.txt on home-lab-3-client-0
>>>>>> [2016-11-18 16:07:11.918488] D [MSGID: 0]
>>>>>> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
>>>>>> 'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
>>>>>> 10456166432768 and avail_inodes is: 99.00
>>>>>> [2016-11-18 16:07:11.918570] D [MSGID: 0]
>>>>>> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
>>>>>> 'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
>>>>>> 8600417075200 and avail_inodes is: 99.00
>>>>>> [2016-11-18 16:07:11.918603] D [MSGID: 0]
>>>>>> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
>>>>>> 'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
>>>>>> 10500905500672 and avail_inodes is: 99.00
>>>>>> [2016-11-18 16:07:11.918675] D [MSGID: 0]
>>>>>> [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume
>>>>>> 'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
>>>>>> 10499686531072 and avail_inodes is: 99.00
>>>>>> [2016-11-18 16:07:11.919095] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>>>> event 19 on graph -1418558531
>>>>>> [2016-11-18 16:07:11.919377] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
>>>>>> STAT scheduled
>>>>>> as fast fop
>>>>>> [2016-11-18 16:07:11.920040] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
>>>>>> OPEN scheduled
>>>>>> as fast fop
>>>>>> [2016-11-18 16:07:11.920889] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
>>>>>> WRITE scheduled
>>>>>> as slow fop
>>>>>> [2016-11-18 16:07:11.921051] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
>>>>>> FSTAT scheduled
>>>>>> as fast fop
>>>>>> [2016-11-18 16:07:11.921583] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>>>> event 19 on graph 0
>>>>>> [2016-11-18 16:07:11.922068] D [fuse-bridge.c:52:fuse_invalidate]
>>>>>> 0-fuse: Invalidate inode id 140363155120276.
>>>>>> [2016-11-18 16:07:11.922936] I [io-stats.c:1574:io_stats_dump_fd]
>>>>>> 0-home-lab-3: --- fd stats ---
>>>>>> [2016-11-18 16:07:11.922988] I [io-stats.c:1579:io_stats_dump_fd]
>>>>>> 0-home-lab-3: Filename : /cdc/444.txt
>>>>>> [2016-11-18 16:07:11.923004] I [io-stats.c:1594:io_stats_dump_fd]
>>>>>> 0-home-lab-3: BytesWritten : 7 bytes
>>>>>> [2016-11-18 16:07:11.923026] I [io-stats.c:1606:io_stats_dump_fd]
>>>>>> 0-home-lab-3: Write 000004b+ : 1
>>>>>> [2016-11-18 16:07:11.923779] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
>>>>>> STAT scheduled
>>>>>> as fast fop
>>>>>> [2016-11-18 16:07:11.924504] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>>>> event 19 on graph 0
>>>>>> [2016-11-18 16:07:11.926692] D [logging.c:1953:_gf_msg_internal]
>>>>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>>>>> About to flush least recently used log message to disk
>>>>>> The message "D [MSGID: 0] [io-threads.c:349:iot_schedule]
>>>>>> 0-home-lab-3-io-threads: SETATTR scheduled as normal fop" repeated 2
>>>>>> times between [2016-11-18 16:06:17.097104] and [2016-11-18
>>>>>> 16:07:11.923946]
>>>>>> [2016-11-18 16:07:11.926690] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>>>>> scheduled as fast fop
>>>>>> [2016-11-18 16:07:11.926858] D [MSGID: 0]
>>>>>> [dht-common.c:2684:dht_lookup]
>>>>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
>>>>>> home-lab-3-client-3
>>>>>> [2016-11-18 16:07:11.927327] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.927376] D [MSGID: 0]
>>>>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>>>>> returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
>>>>>> [2016-11-18 16:07:11.927401] D [MSGID: 0]
>>>>>> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
>>>>>> missing on subvol home-lab-3-client-3
>>>>>> [2016-11-18 16:07:11.927422] D [MSGID: 0]
>>>>>> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
>>>>>> lookup call to 4 subvols
>>>>>> [2016-11-18 16:07:11.927940] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.927987] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>>>> home-lab-3-client-0
>>>>>> [2016-11-18 16:07:11.928043] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.928055] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.928058] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.928170] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>>>> home-lab-3-client-1
>>>>>> [2016-11-18 16:07:11.928174] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>>>> home-lab-3-client-3
>>>>>> [2016-11-18 16:07:11.928277] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>>>> home-lab-3-client-2
>>>>>> [2016-11-18 16:07:11.928360] D [MSGID: 0]
>>>>>> [dht-common.c:1527:dht_lookup_everywhere_done]
>>>>>> 0-home-lab-3-dht: STATUS:
>>>>>> hashed_subvol home-lab-3-client-3 cached_subvol null
>>>>>> [2016-11-18 16:07:11.928381] D [MSGID: 0]
>>>>>> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
>>>>>> was no cached file and unlink on hashed is not skipped /cdc/kk.txt
>>>>>> [2016-11-18 16:07:11.928403] D [MSGID: 0]
>>>>>> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1
>>>>>> error: No such
>>>>>> file or directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.928431] D [MSGID: 0]
>>>>>> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.928456] D [MSGID: 0]
>>>>>> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No
>>>>>> such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.928480] D [MSGID: 0]
>>>>>> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.928511] D [MSGID: 0]
>>>>>> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No
>>>>>> such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.928753] D [MSGID: 0]
>>>>>> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file
>>>>>> or directory
>>>>>> [No such file or directory]
>>>>>> [2016-11-18 16:07:11.930485] D [logging.c:1953:_gf_msg_internal]
>>>>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>>>>> About to flush least recently used log message to disk
>>>>>> [2016-11-18 16:07:11.929862] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
>>>>>> STAT scheduled
>>>>>> as fast fop
>>>>>> [2016-11-18 16:07:11.930483] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP
>>>>>> scheduled as fast fop
>>>>>> [2016-11-18 16:07:11.930602] D [MSGID: 0]
>>>>>> [dht-common.c:2684:dht_lookup]
>>>>>> 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on
>>>>>> home-lab-3-client-3
>>>>>> [2016-11-18 16:07:11.931071] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.931124] D [MSGID: 0]
>>>>>> [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup
>>>>>> returned for /cdc/kk.txt with op_ret -1 [No such file or directory]
>>>>>> [2016-11-18 16:07:11.931148] D [MSGID: 0]
>>>>>> [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt
>>>>>> missing on subvol home-lab-3-client-3
>>>>>> [2016-11-18 16:07:11.931348] D [MSGID: 0]
>>>>>> [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding
>>>>>> lookup call to 4 subvols
>>>>>> [2016-11-18 16:07:11.931880] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.931927] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>>>> home-lab-3-client-0
>>>>>> [2016-11-18 16:07:11.932002] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.931994] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.931991] D [MSGID: 0]
>>>>>> [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned
>>>>>> -1 error: No
>>>>>> such file or directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.932078] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>>>> home-lab-3-client-2
>>>>>> [2016-11-18 16:07:11.932116] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>>>> home-lab-3-client-1
>>>>>> [2016-11-18 16:07:11.932315] D [MSGID: 0]
>>>>>> [dht-common.c:1862:dht_lookup_everywhere_cbk]
>>>>>> 0-home-lab-3-dht: returned
>>>>>> with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol
>>>>>> home-lab-3-client-3
>>>>>> [2016-11-18 16:07:11.932363] D [MSGID: 0]
>>>>>> [dht-common.c:1527:dht_lookup_everywhere_done]
>>>>>> 0-home-lab-3-dht: STATUS:
>>>>>> hashed_subvol home-lab-3-client-3 cached_subvol null
>>>>>> [2016-11-18 16:07:11.932383] D [MSGID: 0]
>>>>>> [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There
>>>>>> was no cached file and unlink on hashed is not skipped /cdc/kk.txt
>>>>>> [2016-11-18 16:07:11.932407] D [MSGID: 0]
>>>>>> [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
>>>>>> stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1
>>>>>> error: No such
>>>>>> file or directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.932446] D [MSGID: 0]
>>>>>> [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.932471] D [MSGID: 0]
>>>>>> [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No
>>>>>> such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.932494] D [MSGID: 0]
>>>>>> [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.932518] D [MSGID: 0]
>>>>>> [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No
>>>>>> such file or
>>>>>> directory [No such file or directory]
>>>>>> [2016-11-18 16:07:11.932725] D [logging.c:1953:_gf_msg_internal]
>>>>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>>>>>> About to flush least recently used log message to disk
>>>>>> [2016-11-18 16:07:11.932545] D [MSGID: 0]
>>>>>> [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
>>>>>> 0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file
>>>>>> or directory
>>>>>> [No such file or directory]
>>>>>> [2016-11-18 16:07:11.932722] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
>>>>>> MKNOD scheduled
>>>>>> as normal fop
>>>>>> [2016-11-18 16:07:11.933008] D [MSGID: 0]
>>>>>> [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
>>>>>> creating /cdc/kk.txt on home-lab-3-client-3
>>>>>> [2016-11-18 16:07:11.934086] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>>>> event 19 on graph -1418558531
>>>>>> [2016-11-18 16:07:11.934323] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
>>>>>> scheduled as normal fop
>>>>>> [2016-11-18 16:07:11.934922] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
>>>>>> STAT scheduled
>>>>>> as fast fop
>>>>>> [2016-11-18 16:07:11.935497] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
>>>>>> scheduled as normal fop
>>>>>> [2016-11-18 16:07:11.936102] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>>>> event 19 on graph 0
>>>>>> [2016-11-18 16:07:11.936903] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
>>>>>> OPEN scheduled
>>>>>> as fast fop
>>>>>> [2016-11-18 16:07:11.937540] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
>>>>>> WRITE scheduled
>>>>>> as slow fop
>>>>>> [2016-11-18 16:07:11.937682] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
>>>>>> FSTAT scheduled
>>>>>> as fast fop
>>>>>> [2016-11-18 16:07:11.938267] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>>>> event 19 on graph 0
>>>>>> [2016-11-18 16:07:11.938757] D [fuse-bridge.c:52:fuse_invalidate]
>>>>>> 0-fuse: Invalidate inode id 140363155120652.
>>>>>> [2016-11-18 16:07:11.938994] I [io-stats.c:1574:io_stats_dump_fd]
>>>>>> 0-home-lab-3: --- fd stats ---
>>>>>> [2016-11-18 16:07:11.939072] I [io-stats.c:1579:io_stats_dump_fd]
>>>>>> 0-home-lab-3: Filename : /cdc/kk.txt
>>>>>> [2016-11-18 16:07:11.939096] I [io-stats.c:1594:io_stats_dump_fd]
>>>>>> 0-home-lab-3: BytesWritten : 7 bytes
>>>>>> [2016-11-18 16:07:11.939112] I [io-stats.c:1606:io_stats_dump_fd]
>>>>>> 0-home-lab-3: Write 000004b+ : 1
>>>>>> [2016-11-18 16:07:11.939448] D [MSGID: 0]
>>>>>> [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR
>>>>>> scheduled as normal fop
>>>>>> [2016-11-18 16:07:11.940028] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>>>> event 19 on graph 0
>>>>>> [2016-11-18 16:07:11.942130] D [fuse-bridge.c:5264:notify] 0-fuse: got
>>>>>> event 19 on graph 0
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ** TEST DONE as root --> NO PROBLEM
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> [2016-11-18 16:07:28.794150] D
>>>>>> [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
>>>>>> 0-home-lab-3-client-2: returning as transport is already
>>>>>> disconnected OR
>>>>>> there are no frames (0 || 0)
>>>>>> [2016-11-18 16:07:31.794656] D
>>>>>> [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
>>>>>> 0-home-lab-3-client-0: returning as transport is already
>>>>>> disconnected OR
>>>>>> there are no frames (0 || 0)
>>>>>> [2016-11-18 16:07:31.794925] D
>>>>>> [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
>>>>>> 0-home-lab-3-client-1: returning as transport is already
>>>>>> disconnected OR
>>>>>> there are no frames (0 || 0)
>>>>>> [2016-11-18 16:07:31.794984] D
>>>>>> [rpc-clnt-ping.c:295:rpc_clnt_start_ping]
>>>>>> 0-home-lab-3-client-3: returning as transport is already
>>>>>> disconnected OR
>>>>>> there are no frames (0 || 0)
>>>>>>
>>>>>>
>>>>>> ** USER ROOT EXITED at this point
>>>>>>
>>>>>> El 18/11/16 a las 15:16, Miklos Szeredi escribió:
>>>>>>> On Wed, Nov 16, 2016 at 7:19 PM, Omar Walid Llorente
>>>>>>> <[email protected]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> In any case, the kernel we are using (3.13.0-96-generic #143-Ubuntu
>>>>>>>> SMP Mon
>>>>>>>> Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux) has fuse
>>>>>>>> included
>>>>>>>> (not as a module) so if it would be necessary to change
>>>>>>>> the fuse mount
>>>>>>>> options some help would be needed.
>>>>>>> Can you reproduce this on latest (e.g 4.8) kernel?
>>>>>>>
>>>>>>> Also fuse debug logs (with whaterver kernel) could be
>>>>>>> useful; not sure
>>>>>>> how to enable it for glusterfs, usual convention for fuse filesystems
>>>>>>> is "-d".
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Miklos
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>>> the body of a message to [email protected]
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>
>> --
>> ----------------------------------------------------------------
>> Centro de Cálculo Depto. Ingeniería Sistemas Telemáticos
>> E-mail: [email protected] Universidad Politécnica de Madrid
>> Fax:(+34) 913367333 E.T.S. Ing. Telecomunicación
>> Tel:(+34) 915495700-Ext.3005 28040 Madrid (Spain)
>> Tel:(+34) 915495762-Ext.3005
>> Tel:(+34) 913367366-Ext.3005
>> ----------------------------------------------------------------

--
----------------------------------------------------------------
Centro de Cálculo Depto. Ingeniería Sistemas Telemáticos
E-mail: [email protected] Universidad Politécnica de Madrid
Fax:(+34) 913367333 E.T.S. Ing. Telecomunicación
Tel:(+34) 915495700-Ext.3005 28040 Madrid (Spain)
Tel:(+34) 915495762-Ext.3005
Tel:(+34) 913367366-Ext.3005
----------------------------------------------------------------

2016-12-15 20:19:15