Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
To:     linux-rdma@vger.kernel.org,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
From:   Timo Rothenpieler <timo@rothenpieler.org>
Subject: copy_file_range() infinitely hangs on NFSv4.2 over RDMA
Message-ID: <57f67888-160f-891c-6217-69e174d7e42b@rothenpieler.org>
Date:   Sun, 14 Feb 2021 04:31:16 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.7.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk

On our Fileserver, running a few weeks old 5.10, we are running into a 
weird issue with NFS 4.2 Server-Side Copy and RDMA (and ZFS, though I'm 
not sure how relevant that is to the issue).
The servers are connected via InfiniBand, on a Mellanox ConnectX-4 card, 
using the mlx5 driver.

Anything using the copy_file_range() syscall to copy stuff just hangs.
In strace, the syscall never returns.

Simple way to reproduce on the client:
 > xfs_io -fc "pwrite 0 1M" testfile
 > xfs_io -fc "copy_range testfile" testfile.copy

The second call just never exits. It sits in S+ state, with no CPU 
usage, and can easily be killed via Ctrl+C.
I let it sit for a couple hours as well, it does not seem to ever complete.

Some more observations about it:

If I do a fresh reboot of the client, the operation works fine for a 
short while (like, 10~15 minutes). No load is on the system during that 
time, it's effectively idle.

The operation actually does successfully copy all data. The size and 
checksum of the target file is as expected. It just never returns.

This only happens when mounting via RDMA. Mounting the same NFS share 
via plain TCP has the operation work reliably.

Had this issue with Kernel 5.4 already, and had hoped that 5.10 might 
have fixed it, but unfortunately it didn't.

I tried two server and 30 different client machines, they all exhibit 
the exact same behaviour. So I'd carefully rule out a hardware issue.


Any pointers on how to debug or maybe even fix this?


Thanks,
Timo