Received: by 2002:a05:6a10:8c0a:0:0:0:0 with SMTP id go10csp586278pxb; Wed, 27 Jan 2021 15:56:38 -0800 (PST) X-Google-Smtp-Source: ABdhPJw1JYDdzxrc/HGIG7a0SpUmDSNjjdp5uMK+rnw5ljVo7WQylC0tD4bsIWCY1NDJujJi18DR X-Received: by 2002:a17:906:578e:: with SMTP id k14mr8315542ejq.243.1611791798470; Wed, 27 Jan 2021 15:56:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1611791798; cv=none; d=google.com; s=arc-20160816; b=Ewb8TM12AEjvSvf/qCZy0gV+YomeDYhiHjG2gsIi96vFKhz4uWjg2IPCp8lWTV8T6Q BwR+vShAzXcYNKi3S7bSjoQyVf7J9n4uNBsx5OwgVwRcvcHI0kXvnTUDoqc8Nsd9ua5L KbWXItOATRM7P1XwqpQH9R9g31mc1zO7PO/Ehzm0DpadBJfxuMQMSZpC5QhFrp7do7DT bRW+52xy76UgEFIRmxvBXqKbF5COLQItx/ZIq+LkfUHiW5xo4R2aGTzt0Beqt5JBUXVB Zgqhq0L++7Y71no7OVqO1xb11wFW0DmCSS6anifCoKEMfbdmyNARk7qLpX4W18vv/2fX Ve6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from; bh=mdyZZdP+BC/HjYs0v4HsG7/NAgh5Vt55u2BOrm15aXM=; b=BBkm0U8ebknI60MVvxPRdxnFb+/zXlXD2/KquB7C8TK3aWkEckIUr1lUogrgGR1nmH ObJw/DwfnUf3ZNGHW2EL8WyaBCWTl64VkThJxkxsAHNsD6p8t8kW7gIqm7aMLcF82EV6 V85wjTAz5D1TYokXz8KZMbNNWZezJ6ZLM4IfQV4oeSP46K9ztWCLWF22UhhPNdtSUPRE BB8X0/uZk8Hw9x15rEsDmrQ7BI+g327y7u2CvH3GQhUI7elCn1l8Yt1nSdKk8BPUisRU P9YF49jQsgr69lMaVWs8x/QxdFdOVxco8eO1NkD5TKu2oeHGzevr9nWxE9SMmMK4Q28P dktw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id ox4si1462969ejb.116.2021.01.27.15.56.14; Wed, 27 Jan 2021 15:56:38 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236241AbhA0MVa (ORCPT + 99 others); Wed, 27 Jan 2021 07:21:30 -0500 Received: from szxga07-in.huawei.com ([45.249.212.35]:11902 "EHLO szxga07-in.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237709AbhA0MOr (ORCPT ); Wed, 27 Jan 2021 07:14:47 -0500 Received: from DGGEMS404-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga07-in.huawei.com (SkyGuard) with ESMTP id 4DQjD34KLGz7bFy; Wed, 27 Jan 2021 20:12:51 +0800 (CST) Received: from DESKTOP-7FEPK9S.china.huawei.com (10.174.186.182) by DGGEMS404-HUB.china.huawei.com (10.3.19.204) with Microsoft SMTP Server id 14.3.498.0; Wed, 27 Jan 2021 20:13:57 +0800 From: Shenming Lu To: Marc Zyngier , Eric Auger , "Will Deacon" , , , , CC: Alex Williamson , Cornelia Huck , Lorenzo Pieralisi , , , Subject: [PATCH v3 0/4] KVM: arm64: Add VLPI migration support on GICv4.1 Date: Wed, 27 Jan 2021 20:13:33 +0800 Message-ID: <20210127121337.1092-1-lushenming@huawei.com> X-Mailer: git-send-email 2.27.0.windows.1 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-Originating-IP: [10.174.186.182] X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Marc, sorry for the late commit. In GICv4.1, migration has been supported except for (directly-injected) VLPI. And GICv4.1 Spec explicitly gives a way to get the VLPI's pending state (which was crucially missing in GICv4.0). So we make VLPI migration capable on GICv4.1 in this patch set. In order to support VLPI migration, we need to save and restore all required configuration information and pending states of VLPIs. But in fact, the configuration information of VLPIs has already been saved (or will be reallocated on the dst host...) in vgic(kvm) migration. So we only have to migrate the pending states of VLPIs specially. Below is the related workflow in migration. On the save path: In migration completion: pause all vCPUs | call each VM state change handler: pause other devices (just keep from sending interrupts, and such as VFIO migration protocol has already realized it [1]) | flush ITS tables into guest RAM | flush RDIST pending tables (also flush VLPI state here) | ... On the resume path: load each device's state: restore ITS tables (include pending tables) from guest RAM | for other (PCI) devices (paused), if configured to have VLPIs, establish the forwarding paths of their VLPIs (and transfer the pending states from kvm's vgic to VPT here) We have tested this series in VFIO migration, and found some related issues in QEMU [2]. Links: [1] vfio: UAPI for migration interface for device state: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a8a24f3f6e38103b77cf399c38eb54e1219d00d6 [2] vfio: Some fixes and optimizations for VFIO migration: https://patchwork.ozlabs.org/cover/1413263/ History: v2 -> v3 - Add the vgic initialized check to ensure that the allocation and enabling of the doorbells have already been done before unmapping the vPEs. - Check all get_vlpi_state related conditions in save_pending_tables in one place. - Nit fixes. v1 -> v2: - Get the VLPI state from the KVM side. - Nit fixes. Thanks, Shenming Shenming Lu (3): KVM: arm64: GICv4.1: Add function to get VLPI state KVM: arm64: GICv4.1: Try to save hw pending state in save_pending_tables KVM: arm64: GICv4.1: Give a chance to save VLPI's pending state Zenghui Yu (1): KVM: arm64: GICv4.1: Restore VLPI's pending state to physical side .../virt/kvm/devices/arm-vgic-its.rst | 2 +- arch/arm64/kvm/vgic/vgic-its.c | 6 +- arch/arm64/kvm/vgic/vgic-v3.c | 61 +++++++++++++++++-- arch/arm64/kvm/vgic/vgic-v4.c | 33 ++++++++++ arch/arm64/kvm/vgic/vgic.h | 1 + 5 files changed, 93 insertions(+), 10 deletions(-) -- 2.19.1