Received-SPF: pass (google.com: domain of linux-nfs+bounces-3166-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
From: Benjamin Coddington <bcodding@redhat.com>
To: Thomas Glanzmann <thomas@glanzmann.de>
Cc: kvm@vger.kernel.org, linux-nfs@vger.kernel.org
Subject: Re: I/O stalls when merging qcow2 snapshots on nfs
Date: Mon, 06 May 2024 07:25:34 -0400
Message-ID: <CC139243-7C48-4416-BE71-3C7B651F00FC@redhat.com>
In-Reply-To: <ZjdtmFu92mSlaHZ2@glanzmann.de>
References: <ZjdtmFu92mSlaHZ2@glanzmann.de>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On 5 May 2024, at 7:29, Thomas Glanzmann wrote:

> Hello,
> I often take snapshots in order to move kvm VMs from one nfs share to
> another while they're running or to take backups. Sometimes I have very=

> large VMs (1.1 TB) which take a very long time (40 minutes - 2 hours) t=
o
> backup or move. They also write between 20 - 60 GB of data while being
> backed up or moved. Once the backup or move is done the dirty snapshot
> data needs to be merged to the parent disk. While doing this I often
> experience I/O stalls within the VMs in the range of 1 - 20 seconds.
> Sometimes worse. But I have some very latency sensitive VMs which crash=

> or misbehave after 15 seconds I/O stalls. So I would like to know if th=
ere
> is some tuening I can do to make these I/O stalls shorter.
>
> - I already tried to set vm.dirty_expire_centisecs=3D100 which appears =
to
>   make it better, but not under 15 seconds. Perfect would be I/O stalls=

>   no more than 1 second.
>
> This is how you can reproduce the issue:
>
> - NFS Server:
> mkdir /ssd
> apt install -y nfs-kernel-server
> echo '/nfs 0.0.0.0/0.0.0.0(rw,no_root_squash,no_subtree_check,sync)' > =
/etc/exports
> exports -ra
>
> - NFS Client / KVM Host:
> mount server:/ssd /mnt
> # Put a VM on /mnt and start it.
> # Create a snapshot:
> virsh snapshot-create-as --domain testy guest-state1 --diskspec vda,fil=
e=3D/mnt/overlay.qcow2 --disk-only --atomic --no-metadata -no-metadata

What NFS version ends up getting mounted here?  You might eliminate some
head-of-line blocking issues with the "nconnect=3D16" mount option to ope=
n
additional TCP connections.

My view of what could be happening is that the IO from your guest's proce=
ss
is congesting with the IO from your 'virsh blockcommit' process, and we
don't currently have a great way to classify and queue IO from various
sources in various ways.

Ben