Received-SPF: pass (google.com: domain of linux-kernel+bounces-125668-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Date: Sat, 30 Mar 2024 10:17:43 +0000
Message-ID: <87r0fsrpko.wl-maz@kernel.org>
From: Marc Zyngier <maz@kernel.org>
To: Krister Johansen <kjlx@templeofstupid.com>
Cc: Oliver Upton <oliver.upton@linux.dev>,
	James Morse <james.morse@arm.com>,
	Suzuki K Poulose <suzuki.poulose@arm.com>,
	Zenghui Yu <yuzenghui@huawei.com>,
	Catalin Marinas <catalin.marinas@arm.com>,
	Will Deacon <will@kernel.org>,
	Ali Saidi <alisaidi@amazon.com>,
	David Reaver <me@davidreaver.com>,
	linux-arm-kernel@lists.infradead.org,
	kvmarm@lists.linux.dev,
	linux-kernel@vger.kernel.org
Subject: Re: [PATCH] KVM: arm64: Limit stage2_apply_range() batch size to smallest block
In-Reply-To: <20240329191537.GA2051@templeofstupid.com>
References: <cover.1711649501.git.kjlx@templeofstupid.com>
	<ebf0fac84cb1d19bdc6e73576e3cc40a9cab0635.1711649501.git.kjlx@templeofstupid.com>
	<ZgbGtpj5mStTkAkn@linux.dev>
	<20240329191537.GA2051@templeofstupid.com>
User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue)
 FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/28.2
 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO)
Precedence: bulk
MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue")
Content-Type: text/plain; charset=US-ASCII

On Fri, 29 Mar 2024 19:15:37 +0000,
Krister Johansen <kjlx@templeofstupid.com> wrote:
> 
> Hi Oliver,
> Thanks for the response.
> 
> On Fri, Mar 29, 2024 at 06:48:38AM -0700, Oliver Upton wrote:
> > On Thu, Mar 28, 2024 at 12:05:08PM -0700, Krister Johansen wrote:
> > > stage2_apply_range() for unmap operations can interfere with the
> > > performance of IO if the device's interrupts share the CPU where the
> > > unmap operation is occurring.  commit 5994bc9e05c2 ("KVM: arm64: Limit
> > > stage2_apply_range() batch size to largest block") improved this.  Prior
> > > to that commit, workloads that were unfortunate enough to have their IO
> > > interrupts pinned to the same CPU as the unmap operation would observe a
> > > complete stall.  With the switch to using the largest block size, it is
> > > possible for IO to make progress, albeit at a reduced speed.
> > 
> > Can you describe the workload a bit more? I'm having a hard time
> > understanding how you're unmapping that much memory on the fly in
> > your workload. Is guest memory getting swapped? Are VMs being torn
> > down?
> 
> Sorry I wasn't clear here.  Yes, it's the VMs getting torn down that's
> causing the problems.  The container VMs don't have long lifetimes, but
> some may be up to 256Gb in size, depending on the user.  The workloads
> running the VMs aren't especially performance sensitive, but their users
> do notice when network connections time-out.  IOW, if the performance is
> bad enough to temporarily prevent new TCP connections from being
> established or requests / responses being recieved in a timely fashion,
> we'll hear about it.  Users deploy their services a lot, so there's a
> lot of container vm churn.  (Really it's automation redeploying the
> services on behalf of the users in response to new commits to their
> repos...)

I think this advocates for a teardown-specific code path rather than
just relying on the usual S2 unmapping which is really designed for
eviction. There are two things to consider here:

- TLB invalidation: this should only take a single VMALLS12E1, rather
  than iterating over the PTs

- Cache maintenance: this could be elided with FWB, or *optionally*
  elided if userspace buys in a "I don't need to see the memory of the
  guest after teardown" type of behaviour

> > Also, it seems a bit odd to steer interrupts *into* the workload you
> > care about...
> 
> Ah, that was only intentionally done for the purposes of measuring the
> impact.  That's not done on purpose in production.
> 
> Nevertheless, the example we tend to run into is that a box may have 2
> NICs and each NIC has 32 Tx-Rx queues.  This means we've got 64 NIC
> interrupts, each assigned to a different CPU.  Our systems have 64 CPUs.
> What happens in practice is that a VM will get torn down, and that has a
> 1-in-64 chance of impacting the performance of the subset of the flows
> that are mapped via RSS to the interrupt that happens to be assigned to
> the CPU where the VM is being torn down.
> 
> Of course, the obvious next question is why not just bind the VMs flows
> to the CPUs the VM is running on?  We don't have a 1:1 mapping of
> network device to VM, or VM to CPU right now, which frustrates this
> approach.
> 
> > > Further reducing the stage2_apply_range() batch size has substantial
> > > performance improvements for IO that share a CPU performing an unmap
> > > operation.  By switching to a 2mb chunk, IO performance regressions were
> > > no longer observed in this author's tests.  E.g. it was possible to
> > > obtain the advertised device throughput despite an unmap operation
> > > occurring on the CPU where the interrupt was running.  There is a
> > > tradeoff, however.  No changes were observed in per-operation timings
> > > when running the kvm_pagetable_test without an interrupt load.  However,
> > > with a 64gb VM, 1 vcpu, and 4k pages and a IO load, map times increased
> > > by about 15% and unmap times increased by about 58%.  In essence, this
> > > trades slower map/unmap times for improved IO throughput.
> > 
> > There are other users of the range-based operations, like
> > write-protection. Live migration is especially sensitive to the latency
> > of page table updates as it can affect the VMM's ability to converge
> > with the guest.
> 
> To be clear, the reduction in performance was observed when I
> concurrently executed both the kvm_pagetable_test and a networking
> benchmark where the NIC's interrupts were assigned to the same CPU where
> the pagetable test was executing.  I didn't see a slowdown just running
> the pagetable test.

Any chance you could share more details about your HW configuration
(what CPU is that?)  and the type of traffic? This is the sort of
things I'd like to be able to reproduce in order to experiment various
strategies.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.