2020-05-29 18:16:00

by Chuck Lever III

Subject: [PATCH v1] NFS: Fix direct WRITE throughput regression

I measured a 50% throughput regression for large direct writes.

The observed on-the-wire behavior is that the client sends every
NFS WRITE twice: once as an UNSTABLE WRITE plus a COMMIT, and once
as a FILE_SYNC WRITE.

This is because the nfs_write_match_verf() check in
nfs_direct_commit_complete() fails for every WRITE.

Buffered writes use nfs_write_completion(), which sets req->wb_verf
correctly. Direct writes use nfs_direct_write_completion(), which
does not set req->wb_verf at all. This leaves req->wb_verf set to
all zeroes for every direct WRITE, and thus
nfs_direct_commit_completion() always sets NFS_ODIRECT_RESCHED_WRITES.

This fix appears to restore nearly all of the lost performance.

Fixes: 1f28476dcb98 ("NFS: Fix O_DIRECT commit verifier handling")
Signed-off-by: Chuck Lever <[email protected]>
---
fs/nfs/direct.c | 2 ++
1 file changed, 2 insertions(+)

diff --git a/fs/nfs/direct.c b/fs/nfs/direct.c
index a57e7c72c7f4..d49b1d197908 100644
--- a/fs/nfs/direct.c
+++ b/fs/nfs/direct.c
@@ -731,6 +731,8 @@ static void nfs_direct_write_completion(struct nfs_pgio_header *hdr)
nfs_list_remove_request(req);
if (request_commit) {
kref_get(&req->wb_kref);
+ memcpy(&req->wb_verf, &hdr->verf.verifier,
+ sizeof(req->wb_verf));
nfs_mark_request_commit(req, hdr->lseg, &cinfo,
hdr->ds_commit_idx);
}

2020-05-29 21:28:39

by Trond Myklebust

Subject: Re: [PATCH v1] NFS: Fix direct WRITE throughput regression

On Fri, 2020-05-29 at 14:14 -0400, Chuck Lever wrote:
> I measured a 50% throughput regression for large direct writes.
>
> The observed on-the-wire behavior is that the client sends every
> NFS WRITE twice: once as an UNSTABLE WRITE plus a COMMIT, and once
> as a FILE_SYNC WRITE.
>
> This is because the nfs_write_match_verf() check in
> nfs_direct_commit_complete() fails for every WRITE.
>
> Buffered writes use nfs_write_completion(), which sets req->wb_verf
> correctly. Direct writes use nfs_direct_write_completion(), which
> does not set req->wb_verf at all. This leaves req->wb_verf set to
> all zeroes for every direct WRITE, and thus
> nfs_direct_commit_completion() always sets
> NFS_ODIRECT_RESCHED_WRITES.
>
> This fix appears to restore nearly all of the lost performance.
>
> Fixes: 1f28476dcb98 ("NFS: Fix O_DIRECT commit verifier handling")
> Signed-off-by: Chuck Lever <[email protected]>
> ---
> fs/nfs/direct.c | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/fs/nfs/direct.c b/fs/nfs/direct.c
> index a57e7c72c7f4..d49b1d197908 100644
> --- a/fs/nfs/direct.c
> +++ b/fs/nfs/direct.c
> @@ -731,6 +731,8 @@ static void nfs_direct_write_completion(struct
> nfs_pgio_header *hdr)
> nfs_list_remove_request(req);
> if (request_commit) {
> kref_get(&req->wb_kref);
> + memcpy(&req->wb_verf, &hdr->verf.verifier,
> + sizeof(req->wb_verf));
> nfs_mark_request_commit(req, hdr->lseg, &cinfo,
> hdr->ds_commit_idx);
> }
>

Thanks Chuck! That's a no-brainer... Sorry for the screwup.
Anna, are you taking this one?

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]