Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Sun, 25 Aug 2019 11:20:00 +0100
Message-ID: <86wof1mtrj.wl-maz@kernel.org>
From:   Marc Zyngier <maz@kernel.org>
To:     Auger Eric <eric.auger@redhat.com>
Cc:     eric.auger.pro@gmail.com, linux-kernel@vger.kernel.org,
        kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu,
        yuzenghui@huawei.com, zhang.zhanghailiang@huawei.com,
        wanghaibin.wang@huawei.com, james.morse@arm.com,
        qemu-arm@nongnu.org, julien.thierry.kdev@gmail.com,
        suzuki.poulose@arm.com, peter.maydell@linaro.org,
        andre.przywara@arm.com
Subject: Re: [PATCH] KVM: arm/arm64: vgic: Use a single IO device per redistributor
In-Reply-To: <f5b47614-de48-f3cb-0e6f-8a667cb951c0@redhat.com>
References: <20190823173330.23342-1-eric.auger@redhat.com>
        <f5b47614-de48-f3cb-0e6f-8a667cb951c0@redhat.com>
User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue)
 FLIM/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL/10.8 EasyPG/1.0.0 Emacs/26
 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO)
Organization: Approximate
MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue")
Content-Type: text/plain; charset=US-ASCII
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Fri, 23 Aug 2019 18:52:32 +0100,
Auger Eric <eric.auger@redhat.com> wrote:
> 
> Hi Zenghui, Marc,
> 
> On 8/23/19 7:33 PM, Eric Auger wrote:
> > At the moment we use 2 IO devices per GICv3 redistributor: one
> > one for the RD_base frame and one for the SGI_base frame.
> > 
> > Instead we can use a single IO device per redistributor (the 2
> > frames are contiguous). This saves slots on the KVM_MMIO_BUS
> > which is currently limited to NR_IOBUS_DEVS (1000).
> > 
> > This change allows to instantiate up to 512 redistributors and may
> > speed the guest boot with a large number of VCPUs.
> > 
> > Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> I tested this patch with below kernel and QEMU branches:
> kernel: https://github.com/eauger/linux/tree/256fix-v1
> (Marc's patch + this patch)
> https://github.com/eauger/qemu/tree/v4.1.0-256fix-rfc1-rc0
> (header update + kvm_arm_gic_set_irq modification)

A small comment on this: you don't seem to check that
KVM_CAP_ARM_IRQ_LINE_LAYOUT_2 is available before allowing more than
256 vcpus. It'd be worth doing that before allowing a guest to start.

> 
> On a machine with 224 pcpus, I was able to boot a 512 vcpu guest.
> 
> As expected, qemu outputs warnings:
> 
> qemu-system-aarch64: warning: Number of SMP cpus requested (512) exceeds
> the recommended cpus supported by KVM (224)
> qemu-system-aarch64: warning: Number of hotpluggable cpus requested
> (512) exceeds the recommended cpus supported by KVM (224)
> 
> on the guest: getconf _NPROCESSORS_ONLN returns 512
> 
> Then I have no clue about what can be expected of such overcommit config
> and I have not further exercised the guest at the moment.

It will just work, albeit slowly. I often boot 64 vcpu guests on 4 or
8 core systems, just to check that we don't regress too much.

> But at least
> it seems to boot properly. I also tested without overcommit and it seems
> to behave as before (boot, migration).
> 
> I still need to look at the migration of > 256vcpu guest at qemu level.

OK, please let us know how it goes. I'd like some more reviewing on
the userspace ABI change before merging it though. Peter, your input
would be very good to have.

Thanks,

	M.

-- 
Jazz is not dead, it just smells funny.