[All I want for Christmas is... another monster GIC series!]
This rather long series expands the existing GICv4 support to deal with the
new GICv4.1 architecture, which comes with a set of major improvements
compared to v4.0:
- One architectural doorbell per vcpu, instead of one doorbell per VLPI
- Doorbell entirely managed by the HW, with an "at most once" delivery
guarantee per non-residency phase and only when requested by the
hypervisor
- A shared memory scheme between ITSs and redistributors, allowing for an
optimised residency sequence (the use of VMOVP becomes less frequent)
- Support for direct virtual SGI delivery (the injection path still involves
the hypervisor), at the cost of losing the active state on SGIs. It
shouldn't be a big deal, but some guest operating systems might notice
(Linux definitely won't care).
On the other hand, public documentation is not available yet, so that's a
bit annoying...
The series is roughly organised in 5 parts:
(1) Feature detection, errata workaround for TX1
(2) VPE table allocation, new flavours of VMAPP/VMOVP commands
(3) v4.1 doorbell management
(4) Virtual SGI support
(5) Plumbing of virtual SGIs in KVM
Ideally, I'd like to start merging some of this into 5.6.
Notes:
- This series has uncovered a behaviour that looks like a HW bug on
the Cavium ThunderX (aka TX1) platform (see patch #3). I'd very
much welcome some clarification from the Marvell/Cavium folks on
Cc, as well as an official erratum number if this happens to be an
actual bug.
[v3 update]
People have ignored for two months now, and it is fairly obvious
that support for this machine is slowly bit-rotting. Maybe I'll
drop the patch and instead start the process of removing all TX1
support from the kernel (we'd certainly be better off without it).
* From v2 [2]:
- Another bunch of fixes thanks to Zenghui Yu's very careful review
- HW-accelerated SGIs are now optional thanks to new architected
discovery/selection bits exposed by KVM and used by the guest kernel
- Rebased on v5.5-rc2
* From v1 [1]:
- A bunch of minor reworks after Zenghui Yu's review
- A workaround for what looks like a new and unexpected TX1 bug
- A subtle reorder of the series so that some patches can go in early
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
Marc Zyngier (32):
irqchip/gic-v3: Detect GICv4.1 supporting RVPEID
irqchip/gic-v3: Add GICv4.1 VPEID size discovery
irqchip/gic-v3: Workaround Cavium TX1 erratum when reading GICD_TYPER2
irqchip/gic-v3: Use SGIs without active state if offered
irqchip/gic-v4.1: VPE table (aka GICR_VPROPBASER) allocation
irqchip/gic-v4.1: Implement the v4.1 flavour of VMAPP
irqchip/gic-v4.1: Don't use the VPE proxy if RVPEID is set
irqchip/gic-v4.1: Implement the v4.1 flavour of VMOVP
irqchip/gic-v4.1: Plumb skeletal VPE irqchip
irqchip/gic-v4.1: Add mask/unmask doorbell callbacks
irqchip/gic-v4.1: Add VPE residency callback
irqchip/gic-v4.1: Add VPE eviction callback
irqchip/gic-v4.1: Add VPE INVALL callback
irqchip/gic-v4.1: Suppress per-VLPI doorbell
irqchip/gic-v4.1: Allow direct invalidation of VLPIs
irqchip/gic-v4.1: Advertise support v4.1 to KVM
irqchip/gic-v4.1: Map the ITS SGIR register page
irqchip/gic-v4.1: Plumb skeletal VSGI irqchip
irqchip/gic-v4.1: Add initial SGI configuration
irqchip/gic-v4.1: Plumb mask/unmask SGI callbacks
irqchip/gic-v4.1: Plumb get/set_irqchip_state SGI callbacks
irqchip/gic-v4.1: Plumb set_vcpu_affinity SGI callbacks
irqchip/gic-v4.1: Move doorbell management to the GICv4 abstraction
layer
irqchip/gic-v4.1: Add VSGI allocation/teardown
irqchip/gic-v4.1: Add VSGI property setup
irqchip/gic-v4.1: Eagerly vmap vPEs
KVM: arm64: GICv4.1: Let doorbells be auto-enabled
KVM: arm64: GICv4.1: Add direct injection capability to SGI registers
KVM: arm64: GICv4.1: Allow SGIs to switch between HW and SW interrupts
KVM: arm64: GICv4.1: Plumb SGI implementation selection in the
distributor
KVM: arm64: GICv4.1: Reload VLPI configuration on distributor
enable/disable
KVM: arm64: GICv4.1: Expose HW-based SGIs in debugfs
arch/arm/include/asm/arch_gicv3.h | 2 +
arch/arm64/include/asm/arch_gicv3.h | 1 +
arch/arm64/include/asm/kvm_host.h | 1 +
drivers/irqchip/irq-gic-v3-its.c | 996 +++++++++++++++++++++++--
drivers/irqchip/irq-gic-v3.c | 57 +-
drivers/irqchip/irq-gic-v4.c | 134 +++-
include/kvm/arm_vgic.h | 4 +
include/linux/irqchip/arm-gic-common.h | 2 +
include/linux/irqchip/arm-gic-v3.h | 76 +-
include/linux/irqchip/arm-gic-v4.h | 43 +-
virt/kvm/arm/arm.c | 8 +
virt/kvm/arm/vgic/vgic-debug.c | 14 +-
virt/kvm/arm/vgic/vgic-mmio-v3.c | 68 +-
virt/kvm/arm/vgic/vgic-mmio.c | 88 ++-
virt/kvm/arm/vgic/vgic-v3.c | 3 +
virt/kvm/arm/vgic/vgic-v4.c | 139 +++-
virt/kvm/arm/vgic/vgic.h | 1 +
17 files changed, 1522 insertions(+), 115 deletions(-)
--
2.20.1
Despite the architecture spec being extremely clear about the fact
that reserved registers in the GIC distributor memory map are RES0
(and thus are not allowed to generate an exception), the Cavium
ThunderX (aka TX1) SoC explodes as such:
[ 0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[ 0.000000] GICv3: 128 SPIs implemented
[ 0.000000] GICv3: 0 Extended SPIs implemented
[ 0.000000] Internal error: synchronous external abort: 96000210 [#1] SMP
[ 0.000000] Modules linked in:
[ 0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.4.0-rc4-00035-g3cf6a3d5725f #7956
[ 0.000000] Hardware name: cavium,thunder-88xx (DT)
[ 0.000000] pstate: 60000085 (nZCv daIf -PAN -UAO)
[ 0.000000] pc : __raw_readl+0x0/0x8
[ 0.000000] lr : gic_init_bases+0x110/0x560
[ 0.000000] sp : ffff800011243d90
[ 0.000000] x29: ffff800011243d90 x28: 0000000000000000
[ 0.000000] x27: 0000000000000018 x26: 0000000000000002
[ 0.000000] x25: ffff8000116f0000 x24: ffff000fbe6a2c80
[ 0.000000] x23: 0000000000000000 x22: ffff010fdc322b68
[ 0.000000] x21: ffff800010a7a208 x20: 00000000009b0404
[ 0.000000] x19: ffff80001124dad0 x18: 0000000000000010
[ 0.000000] x17: 000000004d8d492b x16: 00000000f67eb9af
[ 0.000000] x15: ffffffffffffffff x14: ffff800011249908
[ 0.000000] x13: ffff800091243ae7 x12: ffff800011243af4
[ 0.000000] x11: ffff80001126e000 x10: ffff800011243a70
[ 0.000000] x9 : 00000000ffffffd0 x8 : ffff80001069c828
[ 0.000000] x7 : 0000000000000059 x6 : ffff8000113fb4d1
[ 0.000000] x5 : 0000000000000001 x4 : 0000000000000000
[ 0.000000] x3 : 0000000000000000 x2 : 0000000000000000
[ 0.000000] x1 : 0000000000000000 x0 : ffff8000116f000c
[ 0.000000] Call trace:
[ 0.000000] __raw_readl+0x0/0x8
[ 0.000000] gic_of_init+0x188/0x224
[ 0.000000] of_irq_init+0x200/0x3cc
[ 0.000000] irqchip_init+0x1c/0x40
[ 0.000000] init_IRQ+0x160/0x1d0
[ 0.000000] start_kernel+0x2ec/0x4b8
[ 0.000000] Code: a8c47bfd d65f03c0 d538d080 d65f03c0 (b9400000)
when reading the GICv4.1 GICD_TYPER2 register, which is unexpected...
Work around it by adding a new quirk flagging all the A1 revisions
of the distributor, but it remains unknown whether this could affect
other revisions of this SoC (or even other SoCs from the same silicon
vendor).
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3.c | 23 ++++++++++++++++++++++-
1 file changed, 22 insertions(+), 1 deletion(-)
diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index 286f98222878..640d4db65b78 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -34,6 +34,7 @@
#define GICD_INT_NMI_PRI (GICD_INT_DEF_PRI & ~0x80)
#define FLAGS_WORKAROUND_GICR_WAKER_MSM8996 (1ULL << 0)
+#define FLAGS_WORKAROUND_GICD_TYPER2_TX1 (1ULL << 1)
struct redist_region {
void __iomem *redist_base;
@@ -1464,6 +1465,15 @@ static bool gic_enable_quirk_msm8996(void *data)
return true;
}
+static bool gic_enable_quirk_tx1(void *data)
+{
+ struct gic_chip_data *d = data;
+
+ d->flags |= FLAGS_WORKAROUND_GICD_TYPER2_TX1;
+
+ return true;
+}
+
static bool gic_enable_quirk_hip06_07(void *data)
{
struct gic_chip_data *d = data;
@@ -1502,6 +1512,12 @@ static const struct gic_quirk gic_quirks[] = {
.mask = 0xffffffff,
.init = gic_enable_quirk_hip06_07,
},
+ {
+ .desc = "GICv3: Cavium TX1 GICD_TYPER2 erratum",
+ .iidr = 0xa100034c,
+ .mask = 0xfff00fff,
+ .init = gic_enable_quirk_tx1,
+ },
{
}
};
@@ -1577,7 +1593,12 @@ static int __init gic_init_bases(void __iomem *dist_base,
pr_info("%d SPIs implemented\n", GIC_LINE_NR - 32);
pr_info("%d Extended SPIs implemented\n", GIC_ESPI_NR);
- gic_data.rdists.gicd_typer2 = readl_relaxed(gic_data.dist_base + GICD_TYPER2);
+ /*
+ * ThunderX1 explodes on reading GICD_TYPER2, in total violation
+ * of the spec (which says that reserved addresses are RES0).
+ */
+ if (!(gic_data.flags & FLAGS_WORKAROUND_GICD_TYPER2_TX1))
+ gic_data.rdists.gicd_typer2 = readl_relaxed(gic_data.dist_base + GICD_TYPER2);
gic_data.domain = irq_domain_create_tree(handle, &gic_irq_domain_ops,
&gic_data);
--
2.20.1
GICv4.1 supports the RVPEID ("Residency per vPE ID"), which allows for
a much efficient way of making virtual CPUs resident (to allow direct
injection of interrupts).
The functionnality needs to be discovered on each and every redistributor
in the system, and disabled if the settings are inconsistent.
Reviewed-by: Zenghui Yu <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3.c | 21 ++++++++++++++++++---
include/linux/irqchip/arm-gic-v3.h | 2 ++
2 files changed, 20 insertions(+), 3 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index d6218012097b..ffcb018395ed 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -858,8 +858,21 @@ static int __gic_update_rdist_properties(struct redist_region *region,
void __iomem *ptr)
{
u64 typer = gic_read_typer(ptr + GICR_TYPER);
+
gic_data.rdists.has_vlpis &= !!(typer & GICR_TYPER_VLPIS);
- gic_data.rdists.has_direct_lpi &= !!(typer & GICR_TYPER_DirectLPIS);
+
+ /* RVPEID implies some form of DirectLPI, no matter what the doc says... :-/ */
+ gic_data.rdists.has_rvpeid &= !!(typer & GICR_TYPER_RVPEID);
+ gic_data.rdists.has_direct_lpi &= (!!(typer & GICR_TYPER_DirectLPIS) |
+ gic_data.rdists.has_rvpeid);
+
+ /* Detect non-sensical configurations */
+ if (WARN_ON_ONCE(gic_data.rdists.has_rvpeid && !gic_data.rdists.has_vlpis)) {
+ gic_data.rdists.has_direct_lpi = false;
+ gic_data.rdists.has_vlpis = false;
+ gic_data.rdists.has_rvpeid = false;
+ }
+
gic_data.ppi_nr = min(GICR_TYPER_NR_PPIS(typer), gic_data.ppi_nr);
return 1;
@@ -872,9 +885,10 @@ static void gic_update_rdist_properties(void)
if (WARN_ON(gic_data.ppi_nr == UINT_MAX))
gic_data.ppi_nr = 0;
pr_info("%d PPIs implemented\n", gic_data.ppi_nr);
- pr_info("%sVLPI support, %sdirect LPI support\n",
+ pr_info("%sVLPI support, %sdirect LPI support, %sRVPEID support\n",
!gic_data.rdists.has_vlpis ? "no " : "",
- !gic_data.rdists.has_direct_lpi ? "no " : "");
+ !gic_data.rdists.has_direct_lpi ? "no " : "",
+ !gic_data.rdists.has_rvpeid ? "no " : "");
}
/* Check whether it's single security state view */
@@ -1566,6 +1580,7 @@ static int __init gic_init_bases(void __iomem *dist_base,
&gic_data);
irq_domain_update_bus_token(gic_data.domain, DOMAIN_BUS_WIRED);
gic_data.rdists.rdist = alloc_percpu(typeof(*gic_data.rdists.rdist));
+ gic_data.rdists.has_rvpeid = true;
gic_data.rdists.has_vlpis = true;
gic_data.rdists.has_direct_lpi = true;
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index de991d6633a5..9a5f85d30701 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -234,6 +234,7 @@
#define GICR_TYPER_VLPIS (1U << 1)
#define GICR_TYPER_DirectLPIS (1U << 3)
#define GICR_TYPER_LAST (1U << 4)
+#define GICR_TYPER_RVPEID (1U << 7)
#define GIC_V3_REDIST_SIZE 0x20000
@@ -615,6 +616,7 @@ struct rdists {
u64 flags;
u32 gicd_typer;
bool has_vlpis;
+ bool has_rvpeid;
bool has_direct_lpi;
};
--
2.20.1
Just like for GICv4.0, each VPE has its own doorbell interrupt, and
thus an irqchip that manages them. Since the doorbell management is
quite different on GICv4.1, let's introduce an almost empty irqchip
the will get populated over the next new patches.
Reviewed-by: Zenghui Yu <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3-its.c | 32 +++++++++++++++++++++++++++++++-
1 file changed, 31 insertions(+), 1 deletion(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index fd9d3b6bb465..157f51398850 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -3554,6 +3554,32 @@ static struct irq_chip its_vpe_irq_chip = {
.irq_set_vcpu_affinity = its_vpe_set_vcpu_affinity,
};
+static int its_vpe_4_1_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
+{
+ struct its_cmd_info *info = vcpu_info;
+
+ switch (info->cmd_type) {
+ case SCHEDULE_VPE:
+ return 0;
+
+ case DESCHEDULE_VPE:
+ return 0;
+
+ case INVALL_VPE:
+ return 0;
+
+ default:
+ return -EINVAL;
+ }
+}
+
+static struct irq_chip its_vpe_4_1_irq_chip = {
+ .name = "GICv4.1-vpe",
+ .irq_eoi = irq_chip_eoi_parent,
+ .irq_set_affinity = its_vpe_set_affinity,
+ .irq_set_vcpu_affinity = its_vpe_4_1_set_vcpu_affinity,
+};
+
static int its_vpe_id_alloc(void)
{
return ida_simple_get(&its_vpeid_ida, 0, ITS_MAX_VPEID, GFP_KERNEL);
@@ -3634,6 +3660,7 @@ static void its_vpe_irq_domain_free(struct irq_domain *domain,
static int its_vpe_irq_domain_alloc(struct irq_domain *domain, unsigned int virq,
unsigned int nr_irqs, void *args)
{
+ struct irq_chip *irqchip = &its_vpe_irq_chip;
struct its_vm *vm = args;
unsigned long *bitmap;
struct page *vprop_page;
@@ -3661,6 +3688,9 @@ static int its_vpe_irq_domain_alloc(struct irq_domain *domain, unsigned int virq
vm->nr_db_lpis = nr_ids;
vm->vprop_page = vprop_page;
+ if (gic_rdists->has_rvpeid)
+ irqchip = &its_vpe_4_1_irq_chip;
+
for (i = 0; i < nr_irqs; i++) {
vm->vpes[i]->vpe_db_lpi = base + i;
err = its_vpe_init(vm->vpes[i]);
@@ -3671,7 +3701,7 @@ static int its_vpe_irq_domain_alloc(struct irq_domain *domain, unsigned int virq
if (err)
break;
irq_domain_set_hwirq_and_chip(domain, virq + i, i,
- &its_vpe_irq_chip, vm->vpes[i]);
+ irqchip, vm->vpes[i]);
set_bit(i, bitmap);
}
--
2.20.1
The infamous VPE proxy device isn't used with GICv4.1 because:
- we can invalidate any LPI from the DirectLPI MMIO interface
- the ITS and redistributors understand the life cycle of
the doorbell, so we don't need to enable/disable it all
the time
So let's escape early from the proxy related functions.
Reviewed-by: Zenghui Yu <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3-its.c | 25 ++++++++++++++++++++++++-
1 file changed, 24 insertions(+), 1 deletion(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index e3c7d9ae0bdb..86c69b5cc156 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -3213,7 +3213,7 @@ static const struct irq_domain_ops its_domain_ops = {
/*
* This is insane.
*
- * If a GICv4 doesn't implement Direct LPIs (which is extremely
+ * If a GICv4.0 doesn't implement Direct LPIs (which is extremely
* likely), the only way to perform an invalidate is to use a fake
* device to issue an INV command, implying that the LPI has first
* been mapped to some event on that device. Since this is not exactly
@@ -3221,9 +3221,20 @@ static const struct irq_domain_ops its_domain_ops = {
* only issue an UNMAP if we're short on available slots.
*
* Broken by design(tm).
+ *
+ * GICv4.1, on the other hand, mandates that we're able to invalidate
+ * by writing to a MMIO register. It doesn't implement the whole of
+ * DirectLPI, but that's good enough. And most of the time, we don't
+ * even have to invalidate anything, as the redistributor can be told
+ * whether to generate a doorbell or not (we thus leave it enabled,
+ * always).
*/
static void its_vpe_db_proxy_unmap_locked(struct its_vpe *vpe)
{
+ /* GICv4.1 doesn't use a proxy, so nothing to do here */
+ if (gic_rdists->has_rvpeid)
+ return;
+
/* Already unmapped? */
if (vpe->vpe_proxy_event == -1)
return;
@@ -3246,6 +3257,10 @@ static void its_vpe_db_proxy_unmap_locked(struct its_vpe *vpe)
static void its_vpe_db_proxy_unmap(struct its_vpe *vpe)
{
+ /* GICv4.1 doesn't use a proxy, so nothing to do here */
+ if (gic_rdists->has_rvpeid)
+ return;
+
if (!gic_rdists->has_direct_lpi) {
unsigned long flags;
@@ -3257,6 +3272,10 @@ static void its_vpe_db_proxy_unmap(struct its_vpe *vpe)
static void its_vpe_db_proxy_map_locked(struct its_vpe *vpe)
{
+ /* GICv4.1 doesn't use a proxy, so nothing to do here */
+ if (gic_rdists->has_rvpeid)
+ return;
+
/* Already mapped? */
if (vpe->vpe_proxy_event != -1)
return;
@@ -3279,6 +3298,10 @@ static void its_vpe_db_proxy_move(struct its_vpe *vpe, int from, int to)
unsigned long flags;
struct its_collection *target_col;
+ /* GICv4.1 doesn't use a proxy, so nothing to do here */
+ if (gic_rdists->has_rvpeid)
+ return;
+
if (gic_rdists->has_direct_lpi) {
void __iomem *rdbase;
--
2.20.1
The ITS VMAPP command gains some new fields with GICv4.1:
- a default doorbell, which allows a single doorbell to be used for
all the VLPIs routed to a given VPE
- a pointer to the configuration table (instead of having it in a register
that gets context switched)
- a flag indicating whether this is the first map or the last unmap for
this particular VPE
- a flag indicating whether the pending table is known to be zeroed, or not
Plumb in the new fields in the VMAPP builder, and add the map/unmap
refcounting so that the ITS can do the right thing.
Reviewed-by: Zenghui Yu <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3-its.c | 60 +++++++++++++++++++++++++++---
include/linux/irqchip/arm-gic-v4.h | 18 +++++++--
2 files changed, 69 insertions(+), 9 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 15dea531c34f..e3c7d9ae0bdb 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -450,6 +450,27 @@ static void its_encode_vpt_size(struct its_cmd_block *cmd, u8 vpt_size)
its_mask_encode(&cmd->raw_cmd[3], vpt_size, 4, 0);
}
+static void its_encode_vconf_addr(struct its_cmd_block *cmd, u64 vconf_pa)
+{
+ its_mask_encode(&cmd->raw_cmd[0], vconf_pa >> 16, 51, 16);
+}
+
+static void its_encode_alloc(struct its_cmd_block *cmd, bool alloc)
+{
+ its_mask_encode(&cmd->raw_cmd[0], alloc, 8, 8);
+}
+
+static void its_encode_ptz(struct its_cmd_block *cmd, bool ptz)
+{
+ its_mask_encode(&cmd->raw_cmd[0], ptz, 9, 9);
+}
+
+static void its_encode_vmapp_default_db(struct its_cmd_block *cmd,
+ u32 vpe_db_lpi)
+{
+ its_mask_encode(&cmd->raw_cmd[1], vpe_db_lpi, 31, 0);
+}
+
static inline void its_fixup_cmd(struct its_cmd_block *cmd)
{
/* Let's fixup BE commands */
@@ -633,19 +654,45 @@ static struct its_vpe *its_build_vmapp_cmd(struct its_node *its,
struct its_cmd_block *cmd,
struct its_cmd_desc *desc)
{
- unsigned long vpt_addr;
+ unsigned long vpt_addr, vconf_addr;
u64 target;
-
- vpt_addr = virt_to_phys(page_address(desc->its_vmapp_cmd.vpe->vpt_page));
- target = desc->its_vmapp_cmd.col->target_address + its->vlpi_redist_offset;
+ bool alloc;
its_encode_cmd(cmd, GITS_CMD_VMAPP);
its_encode_vpeid(cmd, desc->its_vmapp_cmd.vpe->vpe_id);
its_encode_valid(cmd, desc->its_vmapp_cmd.valid);
+
+ if (!desc->its_vmapp_cmd.valid) {
+ if (is_v4_1(its)) {
+ alloc = !atomic_dec_return(&desc->its_vmapp_cmd.vpe->vmapp_count);
+ its_encode_alloc(cmd, alloc);
+ }
+
+ goto out;
+ }
+
+ vpt_addr = virt_to_phys(page_address(desc->its_vmapp_cmd.vpe->vpt_page));
+ target = desc->its_vmapp_cmd.col->target_address + its->vlpi_redist_offset;
+
its_encode_target(cmd, target);
its_encode_vpt_addr(cmd, vpt_addr);
its_encode_vpt_size(cmd, LPI_NRBITS - 1);
+ if (!is_v4_1(its))
+ goto out;
+
+ vconf_addr = virt_to_phys(page_address(desc->its_vmapp_cmd.vpe->its_vm->vprop_page));
+
+ alloc = !atomic_fetch_inc(&desc->its_vmapp_cmd.vpe->vmapp_count);
+
+ its_encode_alloc(cmd, alloc);
+
+ /* We can only signal PTZ when alloc==1. Why do we have two bits? */
+ its_encode_ptz(cmd, alloc);
+ its_encode_vconf_addr(cmd, vconf_addr);
+ its_encode_vmapp_default_db(cmd, desc->its_vmapp_cmd.vpe->vpe_db_lpi);
+
+out:
its_fixup_cmd(cmd);
return valid_vpe(its, desc->its_vmapp_cmd.vpe);
@@ -3493,7 +3540,10 @@ static int its_vpe_init(struct its_vpe *vpe)
vpe->vpe_id = vpe_id;
vpe->vpt_page = vpt_page;
- vpe->vpe_proxy_event = -1;
+ if (gic_rdists->has_rvpeid)
+ atomic_set(&vpe->vmapp_count, 0);
+ else
+ vpe->vpe_proxy_event = -1;
return 0;
}
diff --git a/include/linux/irqchip/arm-gic-v4.h b/include/linux/irqchip/arm-gic-v4.h
index 5dbcfc65f21e..498e523085a7 100644
--- a/include/linux/irqchip/arm-gic-v4.h
+++ b/include/linux/irqchip/arm-gic-v4.h
@@ -39,8 +39,20 @@ struct its_vpe {
irq_hw_number_t vpe_db_lpi;
/* VPE resident */
bool resident;
- /* VPE proxy mapping */
- int vpe_proxy_event;
+ union {
+ /* GICv4.0 implementations */
+ struct {
+ /* VPE proxy mapping */
+ int vpe_proxy_event;
+ /* Implementation Defined Area Invalid */
+ bool idai;
+ };
+ /* GICv4.1 implementations */
+ struct {
+ atomic_t vmapp_count;
+ };
+ };
+
/*
* This collection ID is used to indirect the target
* redistributor for this VPE. The ID itself isn't involved in
@@ -49,8 +61,6 @@ struct its_vpe {
u16 col_idx;
/* Unique (system-wide) VPE identifier */
u16 vpe_id;
- /* Implementation Defined Area Invalid */
- bool idai;
/* Pending VLPIs on schedule out? */
bool pending_last;
};
--
2.20.1
If running under control of a hypervisor that implements GICv4.1
SGIs, allow the hypervisor to use them at the expense of loosing
the Active state (which we don't care about for SGIs).
This is trivially done by checking for GICD_TYPER2.nASSGIcap, and
setting GICD_CTLR.nASSGIreq when enabling Group-1 interrupts.
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3.c | 10 ++++++++--
include/linux/irqchip/arm-gic-v3.h | 2 ++
2 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index 640d4db65b78..624f351c0362 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -724,6 +724,7 @@ static void __init gic_dist_init(void)
unsigned int i;
u64 affinity;
void __iomem *base = gic_data.dist_base;
+ u32 val;
/* Disable the distributor */
writel_relaxed(0, base + GICD_CTLR);
@@ -756,9 +757,14 @@ static void __init gic_dist_init(void)
/* Now do the common stuff, and wait for the distributor to drain */
gic_dist_config(base, GIC_LINE_NR, gic_dist_wait_for_rwp);
+ val = GICD_CTLR_ARE_NS | GICD_CTLR_ENABLE_G1A | GICD_CTLR_ENABLE_G1;
+ if (gic_data.rdists.gicd_typer2 & GICD_TYPER2_nASSGIcap) {
+ pr_info("Enabling SGIs without active state\n");
+ val |= GICD_CTLR_nASSGIreq;
+ }
+
/* Enable distributor with ARE, Group1 */
- writel_relaxed(GICD_CTLR_ARE_NS | GICD_CTLR_ENABLE_G1A | GICD_CTLR_ENABLE_G1,
- base + GICD_CTLR);
+ writel_relaxed(val, base + GICD_CTLR);
/*
* Set all global interrupts to the boot CPU only. ARE must be
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index 9dfe64189d99..72b69f4e6c7b 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -57,6 +57,7 @@
#define GICD_SPENDSGIR 0x0F20
#define GICD_CTLR_RWP (1U << 31)
+#define GICD_CTLR_nASSGIreq (1U << 8)
#define GICD_CTLR_DS (1U << 6)
#define GICD_CTLR_ARE_NS (1U << 4)
#define GICD_CTLR_ENABLE_G1A (1U << 1)
@@ -90,6 +91,7 @@
#define GICD_TYPER_ESPIS(typer) \
(((typer) & GICD_TYPER_ESPI) ? GICD_TYPER_SPIS((typer) >> 27) : 0)
+#define GICD_TYPER2_nASSGIcap (1U << 8)
#define GICD_TYPER2_VIL (1U << 7)
#define GICD_TYPER2_VID GENMASK(4, 0)
--
2.20.1
With GICv4.1, VMOVP is extended to allow a default doorbell to be
specified, as well as a validity bit for this doorbell. As an added
bonus, VMOVP isn't required anymore of moving a VPE between
redistributors that share the same affinity.
Let's add this support to the VMOVP builder, and make sure we don't
issue the command if we don't really need to.
Reviewed-by: Zenghui Yu <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3-its.c | 40 ++++++++++++++++++++++++++------
1 file changed, 33 insertions(+), 7 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 86c69b5cc156..fd9d3b6bb465 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -471,6 +471,17 @@ static void its_encode_vmapp_default_db(struct its_cmd_block *cmd,
its_mask_encode(&cmd->raw_cmd[1], vpe_db_lpi, 31, 0);
}
+static void its_encode_vmovp_default_db(struct its_cmd_block *cmd,
+ u32 vpe_db_lpi)
+{
+ its_mask_encode(&cmd->raw_cmd[3], vpe_db_lpi, 31, 0);
+}
+
+static void its_encode_db(struct its_cmd_block *cmd, bool db)
+{
+ its_mask_encode(&cmd->raw_cmd[2], db, 63, 63);
+}
+
static inline void its_fixup_cmd(struct its_cmd_block *cmd)
{
/* Let's fixup BE commands */
@@ -757,6 +768,11 @@ static struct its_vpe *its_build_vmovp_cmd(struct its_node *its,
its_encode_vpeid(cmd, desc->its_vmovp_cmd.vpe->vpe_id);
its_encode_target(cmd, target);
+ if (is_v4_1(its)) {
+ its_encode_db(cmd, true);
+ its_encode_vmovp_default_db(cmd, desc->its_vmovp_cmd.vpe->vpe_db_lpi);
+ }
+
its_fixup_cmd(cmd);
return valid_vpe(its, desc->its_vmovp_cmd.vpe);
@@ -3328,7 +3344,7 @@ static int its_vpe_set_affinity(struct irq_data *d,
bool force)
{
struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
- int cpu = cpumask_first(mask_val);
+ int from, cpu = cpumask_first(mask_val);
/*
* Changing affinity is mega expensive, so let's be as lazy as
@@ -3336,14 +3352,24 @@ static int its_vpe_set_affinity(struct irq_data *d,
* into the proxy device, we need to move the doorbell
* interrupt to its new location.
*/
- if (vpe->col_idx != cpu) {
- int from = vpe->col_idx;
+ if (vpe->col_idx == cpu)
+ goto out;
- vpe->col_idx = cpu;
- its_send_vmovp(vpe);
- its_vpe_db_proxy_move(vpe, from, cpu);
- }
+ from = vpe->col_idx;
+ vpe->col_idx = cpu;
+
+ /*
+ * GICv4.1 allows us to skip VMOVP if moving to a cpu whose RD
+ * is sharing its VPE table with the current one.
+ */
+ if (gic_data_rdist_cpu(cpu)->vpe_table_mask &&
+ cpumask_test_cpu(from, gic_data_rdist_cpu(cpu)->vpe_table_mask))
+ goto out;
+ its_send_vmovp(vpe);
+ its_vpe_db_proxy_move(vpe, from, cpu);
+
+out:
irq_data_update_effective_affinity(d, cpumask_of(cpu));
return IRQ_SET_MASK_OK_DONE;
--
2.20.1
GICv4.1 defines a new VPE table that is potentially shared between
both the ITSs and the redistributors, following complicated affinity
rules.
To make things more confusing, the programming of this table at
the redistributor level is reusing the GICv4.0 GICR_VPROPBASER register
for something completely different.
The code flow is somewhat complexified by the need to respect the
affinities required by the HW, meaning that tables can either be
inherited from a previously discovered ITS or redistributor.
Signed-off-by: Marc Zyngier <[email protected]>
---
arch/arm/include/asm/arch_gicv3.h | 2 +
arch/arm64/include/asm/arch_gicv3.h | 1 +
drivers/irqchip/irq-gic-v3-its.c | 312 +++++++++++++++++++++++++++-
include/linux/irqchip/arm-gic-v3.h | 33 ++-
4 files changed, 341 insertions(+), 7 deletions(-)
diff --git a/arch/arm/include/asm/arch_gicv3.h b/arch/arm/include/asm/arch_gicv3.h
index fa50bb04f580..b5752f0e8936 100644
--- a/arch/arm/include/asm/arch_gicv3.h
+++ b/arch/arm/include/asm/arch_gicv3.h
@@ -10,6 +10,7 @@
#ifndef __ASSEMBLY__
#include <linux/io.h>
+#include <linux/io-64-nonatomic-lo-hi.h>
#include <asm/barrier.h>
#include <asm/cacheflush.h>
#include <asm/cp15.h>
@@ -327,6 +328,7 @@ static inline u64 __gic_readq_nonatomic(const volatile void __iomem *addr)
/*
* GITS_VPROPBASER - hi and lo bits may be accessed independently.
*/
+#define gits_read_vpropbaser(c) __gic_readq_nonatomic(c)
#define gits_write_vpropbaser(v, c) __gic_writeq_nonatomic(v, c)
/*
diff --git a/arch/arm64/include/asm/arch_gicv3.h b/arch/arm64/include/asm/arch_gicv3.h
index 89e4c8b79349..4750fc8030c3 100644
--- a/arch/arm64/include/asm/arch_gicv3.h
+++ b/arch/arm64/include/asm/arch_gicv3.h
@@ -141,6 +141,7 @@ static inline u32 gic_read_rpr(void)
#define gicr_read_pendbaser(c) readq_relaxed(c)
#define gits_write_vpropbaser(v, c) writeq_relaxed(v, c)
+#define gits_read_vpropbaser(c) readq_relaxed(c)
#define gits_write_vpendbaser(v, c) writeq_relaxed(v, c)
#define gits_read_vpendbaser(c) readq_relaxed(c)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index d9d256471dc0..15dea531c34f 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -45,6 +45,7 @@
#define RDIST_FLAGS_PROPBASE_NEEDS_FLUSHING (1 << 0)
#define RDIST_FLAGS_RD_TABLES_PREALLOCATED (1 << 1)
+#define RDIST_FLAGS_VPE_INDIRECT (1 << 2)
static u32 lpi_id_bits;
@@ -106,6 +107,7 @@ struct its_node {
u64 typer;
u64 cbaser_save;
u32 ctlr_save;
+ u32 mpidr;
struct list_head its_device_list;
u64 flags;
unsigned long list_nr;
@@ -116,6 +118,7 @@ struct its_node {
};
#define is_v4(its) (!!((its)->typer & GITS_TYPER_VLPIS))
+#define is_v4_1(its) (!!((its)->typer & GITS_TYPER_VMAPP))
#define device_ids(its) (FIELD_GET(GITS_TYPER_DEVBITS, (its)->typer) + 1)
#define ITS_ITT_ALIGN SZ_256
@@ -2099,6 +2102,65 @@ static bool its_parse_indirect_baser(struct its_node *its,
return indirect;
}
+static u32 compute_common_aff(u64 val)
+{
+ u32 aff, clpiaff;
+
+ aff = FIELD_GET(GICR_TYPER_AFFINITY, val);
+ clpiaff = FIELD_GET(GICR_TYPER_COMMON_LPI_AFF, val);
+
+ return aff & ~(GENMASK(31, 0) >> (clpiaff * 8));
+}
+
+static u32 compute_its_aff(struct its_node *its)
+{
+ u64 val;
+ u32 svpet;
+
+ /*
+ * Reencode the ITS SVPET and MPIDR as a GICR_TYPER, and compute
+ * the resulting affinity. We then use that to see if this match
+ * our own affinity.
+ */
+ svpet = FIELD_GET(GITS_TYPER_SVPET, its->typer);
+ val = FIELD_PREP(GICR_TYPER_COMMON_LPI_AFF, svpet);
+ val |= FIELD_PREP(GICR_TYPER_AFFINITY, its->mpidr);
+ return compute_common_aff(val);
+}
+
+static struct its_node *find_sibling_its(struct its_node *cur_its)
+{
+ struct its_node *its;
+ u32 aff;
+
+ if (!FIELD_GET(GITS_TYPER_SVPET, cur_its->typer))
+ return NULL;
+
+ aff = compute_its_aff(cur_its);
+
+ list_for_each_entry(its, &its_nodes, entry) {
+ u64 baser;
+
+ if (!is_v4_1(its) || its == cur_its)
+ continue;
+
+ if (!FIELD_GET(GITS_TYPER_SVPET, its->typer))
+ continue;
+
+ if (aff != compute_its_aff(its))
+ continue;
+
+ /* GICv4.1 guarantees that the vPE table is GITS_BASER2 */
+ baser = its->tables[2].val;
+ if (!(baser & GITS_BASER_VALID))
+ continue;
+
+ return its;
+ }
+
+ return NULL;
+}
+
static void its_free_tables(struct its_node *its)
{
int i;
@@ -2141,6 +2203,17 @@ static int its_alloc_tables(struct its_node *its)
break;
case GITS_BASER_TYPE_VCPU:
+ if (is_v4_1(its)) {
+ struct its_node *sibling;
+
+ WARN_ON(i != 2);
+ if ((sibling = find_sibling_its(its))) {
+ *baser = sibling->tables[2];
+ its_write_baser(its, baser, baser->val);
+ continue;
+ }
+ }
+
indirect = its_parse_indirect_baser(its, baser,
psz, &order,
ITS_MAX_VPEID_BITS);
@@ -2162,6 +2235,221 @@ static int its_alloc_tables(struct its_node *its)
return 0;
}
+static u64 inherit_vpe_l1_table_from_its(void)
+{
+ struct its_node *its;
+ u64 val;
+ u32 aff;
+
+ val = gic_read_typer(gic_data_rdist_rd_base() + GICR_TYPER);
+ aff = compute_common_aff(val);
+
+ list_for_each_entry(its, &its_nodes, entry) {
+ u64 baser, addr;
+
+ if (!is_v4_1(its))
+ continue;
+
+ if (!FIELD_GET(GITS_TYPER_SVPET, its->typer))
+ continue;
+
+ if (aff != compute_its_aff(its))
+ continue;
+
+ /* GICv4.1 guarantees that the vPE table is GITS_BASER2 */
+ baser = its->tables[2].val;
+ if (!(baser & GITS_BASER_VALID))
+ continue;
+
+ /* We have a winner! */
+ val = GICR_VPROPBASER_4_1_VALID;
+ if (baser & GITS_BASER_INDIRECT)
+ val |= GICR_VPROPBASER_4_1_INDIRECT;
+ val |= FIELD_PREP(GICR_VPROPBASER_4_1_PAGE_SIZE,
+ FIELD_GET(GITS_BASER_PAGE_SIZE_MASK, baser));
+ switch (FIELD_GET(GITS_BASER_PAGE_SIZE_MASK, baser)) {
+ case GIC_PAGE_SIZE_64K:
+ addr = GITS_BASER_ADDR_48_to_52(baser);
+ break;
+ default:
+ addr = baser & GENMASK_ULL(47, 12);
+ break;
+ }
+ val |= FIELD_PREP(GICR_VPROPBASER_4_1_ADDR, addr >> 12);
+ val |= FIELD_PREP(GICR_VPROPBASER_SHAREABILITY_MASK,
+ FIELD_GET(GITS_BASER_SHAREABILITY_MASK, baser));
+ val |= FIELD_PREP(GICR_VPROPBASER_INNER_CACHEABILITY_MASK,
+ FIELD_GET(GITS_BASER_INNER_CACHEABILITY_MASK, baser));
+ val |= FIELD_PREP(GICR_VPROPBASER_4_1_SIZE, GITS_BASER_NR_PAGES(baser) - 1);
+
+ return val;
+ }
+
+ return 0;
+}
+
+static u64 inherit_vpe_l1_table_from_rd(cpumask_t **mask)
+{
+ u32 aff;
+ u64 val;
+ int cpu;
+
+ val = gic_read_typer(gic_data_rdist_rd_base() + GICR_TYPER);
+ aff = compute_common_aff(val);
+
+ for_each_possible_cpu(cpu) {
+ void __iomem *base = gic_data_rdist_cpu(cpu)->rd_base;
+ u32 tmp;
+
+ if (!base || cpu == smp_processor_id())
+ continue;
+
+ val = gic_read_typer(base + GICR_TYPER);
+ tmp = compute_common_aff(val);
+ if (tmp != aff)
+ continue;
+
+ /*
+ * At this point, we have a victim. This particular CPU
+ * has already booted, and has an affinity that matches
+ * ours wrt CommonLPIAff. Let's use its own VPROPBASER.
+ * Make sure we don't write the Z bit in that case.
+ */
+ val = gits_read_vpropbaser(base + SZ_128K + GICR_VPROPBASER);
+ val &= ~GICR_VPROPBASER_4_1_Z;
+
+ *mask = gic_data_rdist_cpu(cpu)->vpe_table_mask;
+
+ return val;
+ }
+
+ return 0;
+}
+
+static int allocate_vpe_l1_table(void)
+{
+ void __iomem *vlpi_base = gic_data_rdist_vlpi_base();
+ u64 val, gpsz, npg, pa;
+ unsigned int psz = SZ_64K;
+ unsigned int np, epp, esz;
+ struct page *page;
+
+ if (!gic_rdists->has_rvpeid)
+ return 0;
+
+ /*
+ * if VPENDBASER.Valid is set, disable any previously programmed
+ * VPE by setting PendingLast while clearing Valid. This has the
+ * effect of making sure no doorbell will be generated and we can
+ * then safely clear VPROPBASER.Valid.
+ */
+ if (gits_read_vpendbaser(vlpi_base + GICR_VPENDBASER) & GICR_VPENDBASER_Valid)
+ gits_write_vpendbaser(GICR_VPENDBASER_PendingLast,
+ vlpi_base + GICR_VPENDBASER);
+
+ /*
+ * If we can inherit the configuration from another RD, let's do
+ * so. Otherwise, we have to go through the allocation process. We
+ * assume that all RDs have the exact same requirements, as
+ * nothing will work otherwise.
+ */
+ val = inherit_vpe_l1_table_from_rd(&gic_data_rdist()->vpe_table_mask);
+ if (val & GICR_VPROPBASER_4_1_VALID)
+ goto out;
+
+ gic_data_rdist()->vpe_table_mask = kzalloc(sizeof(cpumask_t), GFP_KERNEL);
+ if (!gic_data_rdist()->vpe_table_mask)
+ return -ENOMEM;
+
+ val = inherit_vpe_l1_table_from_its();
+ if (val & GICR_VPROPBASER_4_1_VALID)
+ goto out;
+
+ /* First probe the page size */
+ val = FIELD_PREP(GICR_VPROPBASER_4_1_PAGE_SIZE, GIC_PAGE_SIZE_64K);
+ gits_write_vpropbaser(val, vlpi_base + GICR_VPROPBASER);
+ val = gits_read_vpropbaser(vlpi_base + GICR_VPROPBASER);
+ gpsz = FIELD_GET(GICR_VPROPBASER_4_1_PAGE_SIZE, val);
+ esz = FIELD_GET(GICR_VPROPBASER_4_1_ENTRY_SIZE, val);
+
+ switch (gpsz) {
+ default:
+ gpsz = GIC_PAGE_SIZE_4K;
+ /* fall through */
+ case GIC_PAGE_SIZE_4K:
+ psz = SZ_4K;
+ break;
+ case GIC_PAGE_SIZE_16K:
+ psz = SZ_16K;
+ break;
+ case GIC_PAGE_SIZE_64K:
+ psz = SZ_64K;
+ break;
+ }
+
+ /*
+ * Start populating the register from scratch, including RO fields
+ * (which we want to print in debug cases...)
+ */
+ val = 0;
+ val |= FIELD_PREP(GICR_VPROPBASER_4_1_PAGE_SIZE, gpsz);
+ val |= FIELD_PREP(GICR_VPROPBASER_4_1_ENTRY_SIZE, esz);
+
+ /* How many entries per GIC page? */
+ esz++;
+ epp = psz / (esz * SZ_8);
+
+ /*
+ * If we need more than just a single L1 page, flag the table
+ * as indirect and compute the number of required L1 pages.
+ */
+ if (epp < ITS_MAX_VPEID) {
+ int nl2;
+
+ gic_rdists->flags |= RDIST_FLAGS_VPE_INDIRECT;
+ val |= GICR_VPROPBASER_4_1_INDIRECT;
+
+ /* Number of L2 pages required to cover the VPEID space */
+ nl2 = DIV_ROUND_UP(ITS_MAX_VPEID, epp);
+
+ /* Number of L1 pages to point to the L2 pages */
+ npg = DIV_ROUND_UP(nl2 * SZ_8, psz);
+ } else {
+ npg = 1;
+ }
+
+ val |= FIELD_PREP(GICR_VPROPBASER_4_1_SIZE, npg);
+
+ /* Right, that's the number of CPU pages we need for L1 */
+ np = DIV_ROUND_UP(npg * psz, PAGE_SIZE);
+
+ pr_debug("np = %d, npg = %lld, psz = %d, epp = %d, esz = %d\n",
+ np, npg, psz, epp, esz);
+ page = alloc_pages(GFP_KERNEL | __GFP_ZERO, get_order(np * PAGE_SIZE));
+ if (!page)
+ return -ENOMEM;
+
+ gic_data_rdist()->vpe_l1_page = page;
+ pa = virt_to_phys(page_address(page));
+ WARN_ON(!IS_ALIGNED(pa, psz));
+
+ val |= FIELD_PREP(GICR_VPROPBASER_4_1_ADDR, pa >> 12);
+ val |= GICR_VPROPBASER_RaWb;
+ val |= GICR_VPROPBASER_InnerShareable;
+ val |= GICR_VPROPBASER_4_1_Z;
+ val |= GICR_VPROPBASER_4_1_VALID;
+
+out:
+ gits_write_vpropbaser(val, vlpi_base + GICR_VPROPBASER);
+ cpumask_set_cpu(smp_processor_id(), gic_data_rdist()->vpe_table_mask);
+
+ pr_debug("CPU%d: VPROPBASER = %llx %*pbl\n",
+ smp_processor_id(), val,
+ cpumask_pr_args(gic_data_rdist()->vpe_table_mask));
+
+ return 0;
+}
+
static int its_alloc_collections(struct its_node *its)
{
int i;
@@ -2361,7 +2649,7 @@ static void its_cpu_init_lpis(void)
val |= GICR_CTLR_ENABLE_LPIS;
writel_relaxed(val, rbase + GICR_CTLR);
- if (gic_rdists->has_vlpis) {
+ if (gic_rdists->has_vlpis && !gic_rdists->has_rvpeid) {
void __iomem *vlpi_base = gic_data_rdist_vlpi_base();
/*
@@ -2385,6 +2673,16 @@ static void its_cpu_init_lpis(void)
WARN_ON(val & GICR_VPENDBASER_Dirty);
}
+ if (allocate_vpe_l1_table()) {
+ /*
+ * If the allocation has failed, we're in massive trouble.
+ * Disable direct injection, and pray that no VM was
+ * already running...
+ */
+ gic_rdists->has_rvpeid = false;
+ gic_rdists->has_vlpis = false;
+ }
+
/* Make sure the GIC has seen the above */
dsb(sy);
out:
@@ -3787,6 +4085,14 @@ static int __init its_probe_one(struct resource *res,
} else {
pr_info("ITS@%pa: Single VMOVP capable\n", &res->start);
}
+
+ if (is_v4_1(its)) {
+ u32 svpet = FIELD_GET(GITS_TYPER_SVPET, typer);
+ its->mpidr = readl_relaxed(its_base + GITS_MPIDR);
+
+ pr_info("ITS@%pa: Using GICv4.1 mode %08x %08x\n",
+ &res->start, its->mpidr, svpet);
+ }
}
its->numa_node = numa_node;
@@ -4147,6 +4453,8 @@ int __init its_init(struct fwnode_handle *handle, struct rdists *rdists,
bool has_v4 = false;
int err;
+ gic_rdists = rdists;
+
its_parent = parent_domain;
of_node = to_of_node(handle);
if (of_node)
@@ -4159,8 +4467,6 @@ int __init its_init(struct fwnode_handle *handle, struct rdists *rdists,
return -ENXIO;
}
- gic_rdists = rdists;
-
err = allocate_lpi_tables();
if (err)
return err;
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index 72b69f4e6c7b..df3b7cb50956 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -104,6 +104,11 @@
#define GIC_V3_DIST_SIZE 0x10000
+#define GIC_PAGE_SIZE_4K 0ULL
+#define GIC_PAGE_SIZE_16K 1ULL
+#define GIC_PAGE_SIZE_64K 2ULL
+#define GIC_PAGE_SIZE_MASK 3ULL
+
/*
* Re-Distributor registers, offsets from RD_base
*/
@@ -241,6 +246,8 @@
#define GICR_TYPER_DirectLPIS (1U << 3)
#define GICR_TYPER_LAST (1U << 4)
#define GICR_TYPER_RVPEID (1U << 7)
+#define GICR_TYPER_COMMON_LPI_AFF GENMASK_ULL(25, 24)
+#define GICR_TYPER_AFFINITY GENMASK_ULL(63, 32)
#define GIC_V3_REDIST_SIZE 0x20000
@@ -279,6 +286,18 @@
#define GICR_VPROPBASER_RaWaWt GIC_BASER_CACHEABILITY(GICR_VPROPBASER, INNER, RaWaWt)
#define GICR_VPROPBASER_RaWaWb GIC_BASER_CACHEABILITY(GICR_VPROPBASER, INNER, RaWaWb)
+/*
+ * GICv4.1 VPROPBASER reinvention. A subtle mix between the old
+ * VPROPBASER and ITS_BASER. Just not quite any of the two.
+ */
+#define GICR_VPROPBASER_4_1_VALID (1ULL << 63)
+#define GICR_VPROPBASER_4_1_ENTRY_SIZE GENMASK_ULL(61, 59)
+#define GICR_VPROPBASER_4_1_INDIRECT (1ULL << 55)
+#define GICR_VPROPBASER_4_1_PAGE_SIZE GENMASK_ULL(54, 53)
+#define GICR_VPROPBASER_4_1_Z (1ULL << 52)
+#define GICR_VPROPBASER_4_1_ADDR GENMASK_ULL(51, 12)
+#define GICR_VPROPBASER_4_1_SIZE GENMASK_ULL(6, 0)
+
#define GICR_VPENDBASER 0x0078
#define GICR_VPENDBASER_SHAREABILITY_SHIFT (10)
@@ -316,6 +335,7 @@
#define GITS_CTLR 0x0000
#define GITS_IIDR 0x0004
#define GITS_TYPER 0x0008
+#define GITS_MPIDR 0x0018
#define GITS_CBASER 0x0080
#define GITS_CWRITER 0x0088
#define GITS_CREADR 0x0090
@@ -349,6 +369,8 @@
#define GITS_TYPER_HCC_SHIFT 24
#define GITS_TYPER_HCC(r) (((r) >> GITS_TYPER_HCC_SHIFT) & 0xff)
#define GITS_TYPER_VMOVP (1ULL << 37)
+#define GITS_TYPER_VMAPP (1ULL << 40)
+#define GITS_TYPER_SVPET GENMASK_ULL(42, 41)
#define GITS_IIDR_REV_SHIFT 12
#define GITS_IIDR_REV_MASK (0xf << GITS_IIDR_REV_SHIFT)
@@ -419,10 +441,11 @@
#define GITS_BASER_InnerShareable \
GIC_BASER_SHAREABILITY(GITS_BASER, InnerShareable)
#define GITS_BASER_PAGE_SIZE_SHIFT (8)
-#define GITS_BASER_PAGE_SIZE_4K (0ULL << GITS_BASER_PAGE_SIZE_SHIFT)
-#define GITS_BASER_PAGE_SIZE_16K (1ULL << GITS_BASER_PAGE_SIZE_SHIFT)
-#define GITS_BASER_PAGE_SIZE_64K (2ULL << GITS_BASER_PAGE_SIZE_SHIFT)
-#define GITS_BASER_PAGE_SIZE_MASK (3ULL << GITS_BASER_PAGE_SIZE_SHIFT)
+#define __GITS_BASER_PSZ(sz) (GIC_PAGE_SIZE_ ## sz << GITS_BASER_PAGE_SIZE_SHIFT)
+#define GITS_BASER_PAGE_SIZE_4K __GITS_BASER_PSZ(4K)
+#define GITS_BASER_PAGE_SIZE_16K __GITS_BASER_PSZ(16K)
+#define GITS_BASER_PAGE_SIZE_64K __GITS_BASER_PSZ(64K)
+#define GITS_BASER_PAGE_SIZE_MASK __GITS_BASER_PSZ(MASK)
#define GITS_BASER_PAGES_MAX 256
#define GITS_BASER_PAGES_SHIFT (0)
#define GITS_BASER_NR_PAGES(r) (((r) & 0xff) + 1)
@@ -614,8 +637,10 @@ struct rdists {
struct {
void __iomem *rd_base;
struct page *pend_page;
+ struct page *vpe_l1_page;
phys_addr_t phys_base;
bool lpi_enabled;
+ cpumask_t *vpe_table_mask;
} __percpu *rdist;
phys_addr_t prop_table_pa;
void *prop_table_va;
--
2.20.1
As GICv4.1 understands the life cycle of doorbells (instead of
just randomly firing them at the most inconvenient time), just
enable them at irq_request time, and be done with it.
Signed-off-by: Marc Zyngier <[email protected]>
---
virt/kvm/arm/vgic/vgic-v4.c | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/virt/kvm/arm/vgic/vgic-v4.c b/virt/kvm/arm/vgic/vgic-v4.c
index 1eb0f8c76219..c2fcde104ea2 100644
--- a/virt/kvm/arm/vgic/vgic-v4.c
+++ b/virt/kvm/arm/vgic/vgic-v4.c
@@ -141,6 +141,7 @@ int vgic_v4_init(struct kvm *kvm)
kvm_for_each_vcpu(i, vcpu, kvm) {
int irq = dist->its_vm.vpes[i]->irq;
+ unsigned long irq_flags = DB_IRQ_FLAGS;
/*
* Don't automatically enable the doorbell, as we're
@@ -148,8 +149,14 @@ int vgic_v4_init(struct kvm *kvm)
* blocked. Also disable the lazy disabling, as the
* doorbell could kick us out of the guest too
* early...
+ *
+ * On GICv4.1, the doorbell is managed in HW and must
+ * be left enabled.
*/
- irq_set_status_flags(irq, DB_IRQ_FLAGS);
+ if (kvm_vgic_global_state.has_gicv4_1)
+ irq_flags &= ~IRQ_NOAUTOEN;
+ irq_set_status_flags(irq, irq_flags);
+
ret = request_irq(irq, vgic_v4_doorbell_handler,
0, "vcpu", vcpu);
if (ret) {
--
2.20.1
Now that we have HW-accelerated SGIs being delivered to VPEs, it
becomes required to map the VPEs on all ITSs instead of relying
on the lazy approach that we would use when using the ITS-list
mechanism.
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3-its.c | 39 +++++++++++++++++++++++++-------
1 file changed, 31 insertions(+), 8 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 5126bdcfe079..3234bb9fbdbe 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -1554,12 +1554,31 @@ static int its_irq_set_irqchip_state(struct irq_data *d,
return 0;
}
+/*
+ * Two favourable cases:
+ *
+ * (a) Either we have a GICv4.1, and all vPEs have to be mapped at all times
+ * for vSGI delivery
+ *
+ * (b) Or the ITSs do not use a list map, meaning that VMOVP is cheap enough
+ * and we're better off mapping all VPEs always
+ *
+ * If neither (a) nor (b) is true, then we map vPEs on demand.
+ *
+ */
+static bool gic_requires_eager_mapping(void)
+{
+ if (!its_list_map || gic_rdists->has_rvpeid)
+ return true;
+
+ return false;
+}
+
static void its_map_vm(struct its_node *its, struct its_vm *vm)
{
unsigned long flags;
- /* Not using the ITS list? Everything is always mapped. */
- if (!its_list_map)
+ if (gic_requires_eager_mapping())
return;
raw_spin_lock_irqsave(&vmovp_lock, flags);
@@ -1593,7 +1612,7 @@ static void its_unmap_vm(struct its_node *its, struct its_vm *vm)
unsigned long flags;
/* Not using the ITS list? Everything is always mapped. */
- if (!its_list_map)
+ if (gic_requires_eager_mapping())
return;
raw_spin_lock_irqsave(&vmovp_lock, flags);
@@ -4109,8 +4128,12 @@ static int its_vpe_irq_domain_activate(struct irq_domain *domain,
struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
struct its_node *its;
- /* If we use the list map, we issue VMAPP on demand... */
- if (its_list_map)
+ /*
+ * If we use the list map, we issue VMAPP on demand... Unless
+ * we're on a GICv4.1 and we eagerly map the VPE on all ITSs
+ * so that VSGIs can work.
+ */
+ if (!gic_requires_eager_mapping())
return 0;
/* Map the VPE to the first possible CPU */
@@ -4136,10 +4159,10 @@ static void its_vpe_irq_domain_deactivate(struct irq_domain *domain,
struct its_node *its;
/*
- * If we use the list map, we unmap the VPE once no VLPIs are
- * associated with the VM.
+ * If we use the list map on GICv4.0, we unmap the VPE once no
+ * VLPIs are associated with the VM.
*/
- if (its_list_map)
+ if (!gic_requires_eager_mapping())
return;
list_for_each_entry(its, &its_nodes, entry) {
--
2.20.1
Allocate per-VPE SGIs when initializing the GIC-specific part of the
VPE data structure.
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v4.c | 68 +++++++++++++++++++++++++++++-
include/linux/irqchip/arm-gic-v4.h | 2 +
2 files changed, 69 insertions(+), 1 deletion(-)
diff --git a/drivers/irqchip/irq-gic-v4.c b/drivers/irqchip/irq-gic-v4.c
index 117ba6db023d..99b33f60ac63 100644
--- a/drivers/irqchip/irq-gic-v4.c
+++ b/drivers/irqchip/irq-gic-v4.c
@@ -92,6 +92,47 @@ static bool has_v4_1(void)
return !!sgi_domain_ops;
}
+static int its_alloc_vcpu_sgis(struct its_vpe *vpe, int idx)
+{
+ char *name;
+ int sgi_base;
+
+ if (!has_v4_1())
+ return 0;
+
+ name = kasprintf(GFP_KERNEL, "GICv4-sgi-%d", task_pid_nr(current));
+ if (!name)
+ goto err;
+
+ vpe->fwnode = irq_domain_alloc_named_id_fwnode(name, idx);
+ if (!vpe->fwnode)
+ goto err;
+
+ kfree(name);
+ name = NULL;
+
+ vpe->sgi_domain = irq_domain_create_linear(vpe->fwnode, 16,
+ sgi_domain_ops, vpe);
+ if (!vpe->sgi_domain)
+ goto err;
+
+ sgi_base = __irq_domain_alloc_irqs(vpe->sgi_domain, -1, 16,
+ NUMA_NO_NODE, vpe,
+ false, NULL);
+ if (sgi_base <= 0)
+ goto err;
+
+ return 0;
+
+err:
+ if (vpe->sgi_domain)
+ irq_domain_remove(vpe->sgi_domain);
+ if (vpe->fwnode)
+ irq_domain_free_fwnode(vpe->fwnode);
+ kfree(name);
+ return -ENOMEM;
+}
+
int its_alloc_vcpu_irqs(struct its_vm *vm)
{
int vpe_base_irq, i;
@@ -118,8 +159,13 @@ int its_alloc_vcpu_irqs(struct its_vm *vm)
if (vpe_base_irq <= 0)
goto err;
- for (i = 0; i < vm->nr_vpes; i++)
+ for (i = 0; i < vm->nr_vpes; i++) {
+ int ret;
vm->vpes[i]->irq = vpe_base_irq + i;
+ ret = its_alloc_vcpu_sgis(vm->vpes[i], i);
+ if (ret)
+ goto err;
+ }
return 0;
@@ -132,8 +178,28 @@ int its_alloc_vcpu_irqs(struct its_vm *vm)
return -ENOMEM;
}
+static void its_free_sgi_irqs(struct its_vm *vm)
+{
+ int i;
+
+ if (!has_v4_1())
+ return;
+
+ for (i = 0; i < vm->nr_vpes; i++) {
+ unsigned int irq = irq_find_mapping(vm->vpes[i]->sgi_domain, 0);
+
+ if (WARN_ON(!irq))
+ continue;
+
+ irq_domain_free_irqs(irq, 16);
+ irq_domain_remove(vm->vpes[i]->sgi_domain);
+ irq_domain_free_fwnode(vm->vpes[i]->fwnode);
+ }
+}
+
void its_free_vcpu_irqs(struct its_vm *vm)
{
+ its_free_sgi_irqs(vm);
irq_domain_free_irqs(vm->vpes[0]->irq, vm->nr_vpes);
irq_domain_remove(vm->domain);
irq_domain_free_fwnode(vm->fwnode);
diff --git a/include/linux/irqchip/arm-gic-v4.h b/include/linux/irqchip/arm-gic-v4.h
index cca4198fa1d5..9fbd0418f569 100644
--- a/include/linux/irqchip/arm-gic-v4.h
+++ b/include/linux/irqchip/arm-gic-v4.h
@@ -49,6 +49,8 @@ struct its_vpe {
};
/* GICv4.1 implementations */
struct {
+ struct fwnode_handle *fwnode;
+ struct irq_domain *sgi_domain;
struct {
u8 priority;
bool enabled;
--
2.20.1
In order to hide some of the differences between v4.0 and v4.1, move
the doorbell management out of the KVM code, and into the GICv4-specific
layer. This allows the calling code to ask for the doorbell when blocking,
and otherwise to leave the doorbell permanently disabled.
This matches the v4.1 code perfectly, and only results in a minor
refactoring of the v4.0 code.
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v4.c | 45 +++++++++++++++++++++++++++---
include/kvm/arm_vgic.h | 1 +
include/linux/irqchip/arm-gic-v4.h | 3 +-
virt/kvm/arm/vgic/vgic-v3.c | 1 +
virt/kvm/arm/vgic/vgic-v4.c | 34 ++++++++++------------
5 files changed, 59 insertions(+), 25 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v4.c b/drivers/irqchip/irq-gic-v4.c
index c01910d53f9e..117ba6db023d 100644
--- a/drivers/irqchip/irq-gic-v4.c
+++ b/drivers/irqchip/irq-gic-v4.c
@@ -87,6 +87,11 @@ static struct irq_domain *gic_domain;
static const struct irq_domain_ops *vpe_domain_ops;
static const struct irq_domain_ops *sgi_domain_ops;
+static bool has_v4_1(void)
+{
+ return !!sgi_domain_ops;
+}
+
int its_alloc_vcpu_irqs(struct its_vm *vm)
{
int vpe_base_irq, i;
@@ -139,18 +144,50 @@ static int its_send_vpe_cmd(struct its_vpe *vpe, struct its_cmd_info *info)
return irq_set_vcpu_affinity(vpe->irq, info);
}
-int its_schedule_vpe(struct its_vpe *vpe, bool on)
+int its_make_vpe_non_resident(struct its_vpe *vpe, bool db)
{
- struct its_cmd_info info;
+ struct irq_desc *desc = irq_to_desc(vpe->irq);
+ struct its_cmd_info info = { };
int ret;
WARN_ON(preemptible());
- info.cmd_type = on ? SCHEDULE_VPE : DESCHEDULE_VPE;
+ info.cmd_type = DESCHEDULE_VPE;
+ if (has_v4_1()) {
+ /* GICv4.1 can directly deal with doorbells */
+ info.req_db = db;
+ } else {
+ /* Undo the nested disable_irq() calls... */
+ while (db && irqd_irq_disabled(&desc->irq_data))
+ enable_irq(vpe->irq);
+ }
+
+ ret = its_send_vpe_cmd(vpe, &info);
+ if (!ret)
+ vpe->resident = false;
+
+ return ret;
+}
+
+int its_make_vpe_resident(struct its_vpe *vpe, bool g0en, bool g1en)
+{
+ struct its_cmd_info info = { };
+ int ret;
+
+ WARN_ON(preemptible());
+
+ info.cmd_type = SCHEDULE_VPE;
+ if (has_v4_1()) {
+ info.g0en = g0en;
+ info.g1en = g1en;
+ } else {
+ /* Disabled the doorbell, as we're about to enter the guest */
+ disable_irq_nosync(vpe->irq);
+ }
ret = its_send_vpe_cmd(vpe, &info);
if (!ret)
- vpe->resident = on;
+ vpe->resident = true;
return ret;
}
diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
index 9d53f545a3d5..63457908c9c4 100644
--- a/include/kvm/arm_vgic.h
+++ b/include/kvm/arm_vgic.h
@@ -70,6 +70,7 @@ struct vgic_global {
/* Hardware has GICv4? */
bool has_gicv4;
+ bool has_gicv4_1;
/* GIC system register CPU interface */
struct static_key_false gicv3_cpuif;
diff --git a/include/linux/irqchip/arm-gic-v4.h b/include/linux/irqchip/arm-gic-v4.h
index a1a9d40266f5..cca4198fa1d5 100644
--- a/include/linux/irqchip/arm-gic-v4.h
+++ b/include/linux/irqchip/arm-gic-v4.h
@@ -120,7 +120,8 @@ struct its_cmd_info {
int its_alloc_vcpu_irqs(struct its_vm *vm);
void its_free_vcpu_irqs(struct its_vm *vm);
-int its_schedule_vpe(struct its_vpe *vpe, bool on);
+int its_make_vpe_resident(struct its_vpe *vpe, bool g0en, bool g1en);
+int its_make_vpe_non_resident(struct its_vpe *vpe, bool db);
int its_invall_vpe(struct its_vpe *vpe);
int its_map_vlpi(int irq, struct its_vlpi_map *map);
int its_get_vlpi(int irq, struct its_vlpi_map *map);
diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
index f45635a6f0ec..c2fdea201747 100644
--- a/virt/kvm/arm/vgic/vgic-v3.c
+++ b/virt/kvm/arm/vgic/vgic-v3.c
@@ -595,6 +595,7 @@ int vgic_v3_probe(const struct gic_kvm_info *info)
/* GICv4 support? */
if (info->has_v4) {
kvm_vgic_global_state.has_gicv4 = gicv4_enable;
+ kvm_vgic_global_state.has_gicv4_1 = info->has_v4_1;
kvm_info("GICv4 support %sabled\n",
gicv4_enable ? "en" : "dis");
}
diff --git a/virt/kvm/arm/vgic/vgic-v4.c b/virt/kvm/arm/vgic/vgic-v4.c
index 46f875589c47..1eb0f8c76219 100644
--- a/virt/kvm/arm/vgic/vgic-v4.c
+++ b/virt/kvm/arm/vgic/vgic-v4.c
@@ -67,10 +67,10 @@
* it. And if we've migrated our vcpu from one CPU to another, we must
* tell the ITS (so that the messages reach the right redistributor).
* This is done in two steps: first issue a irq_set_affinity() on the
- * irq corresponding to the vcpu, then call its_schedule_vpe(). You
- * must be in a non-preemptible context. On exit, another call to
- * its_schedule_vpe() tells the redistributor that we're done with the
- * vcpu.
+ * irq corresponding to the vcpu, then call its_make_vpe_resident().
+ * You must be in a non-preemptible context. On exit, a call to
+ * its_make_vpe_non_resident() tells the redistributor that we're done
+ * with the vcpu.
*
* Finally, the doorbell handling: Each vcpu is allocated an interrupt
* which will fire each time a VLPI is made pending whilst the vcpu is
@@ -86,7 +86,8 @@ static irqreturn_t vgic_v4_doorbell_handler(int irq, void *info)
struct kvm_vcpu *vcpu = info;
/* We got the message, no need to fire again */
- if (!irqd_irq_disabled(&irq_to_desc(irq)->irq_data))
+ if (!kvm_vgic_global_state.has_gicv4_1 &&
+ !irqd_irq_disabled(&irq_to_desc(irq)->irq_data))
disable_irq_nosync(irq);
vcpu->arch.vgic_cpu.vgic_v3.its_vpe.pending_last = true;
@@ -199,19 +200,11 @@ void vgic_v4_teardown(struct kvm *kvm)
int vgic_v4_put(struct kvm_vcpu *vcpu, bool need_db)
{
struct its_vpe *vpe = &vcpu->arch.vgic_cpu.vgic_v3.its_vpe;
- struct irq_desc *desc = irq_to_desc(vpe->irq);
if (!vgic_supports_direct_msis(vcpu->kvm) || !vpe->resident)
return 0;
- /*
- * If blocking, a doorbell is required. Undo the nested
- * disable_irq() calls...
- */
- while (need_db && irqd_irq_disabled(&desc->irq_data))
- enable_irq(vpe->irq);
-
- return its_schedule_vpe(vpe, false);
+ return its_make_vpe_non_resident(vpe, need_db);
}
int vgic_v4_load(struct kvm_vcpu *vcpu)
@@ -232,18 +225,19 @@ int vgic_v4_load(struct kvm_vcpu *vcpu)
if (err)
return err;
- /* Disabled the doorbell, as we're about to enter the guest */
- disable_irq_nosync(vpe->irq);
-
- err = its_schedule_vpe(vpe, true);
+ err = its_make_vpe_resident(vpe, false, vcpu->kvm->arch.vgic.enabled);
if (err)
return err;
/*
* Now that the VPE is resident, let's get rid of a potential
- * doorbell interrupt that would still be pending.
+ * doorbell interrupt that would still be pending. This is a
+ * GICv4.0 only "feature"...
*/
- return irq_set_irqchip_state(vpe->irq, IRQCHIP_STATE_PENDING, false);
+ if (!kvm_vgic_global_state.has_gicv4_1)
+ err = irq_set_irqchip_state(vpe->irq, IRQCHIP_STATE_PENDING, false);
+
+ return err;
}
static struct vgic_its *vgic_get_its(struct kvm *kvm,
--
2.20.1
Just like for INVALL, GICv4.1 has grown a VPE-aware INVLPI register.
Let's plumb it in and make use of the DirectLPI code in that case.
Reviewed-by: Zenghui Yu <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3-its.c | 53 ++++++++++++++++++++----------
include/linux/irqchip/arm-gic-v3.h | 1 +
2 files changed, 36 insertions(+), 18 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index f00c8ddd3798..360ca2c1533f 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -228,11 +228,26 @@ static struct its_vlpi_map *dev_event_to_vlpi_map(struct its_device *its_dev,
return &its_dev->event_map.vlpi_maps[event];
}
-static struct its_collection *irq_to_col(struct irq_data *d)
+static struct its_vlpi_map *get_vlpi_map(struct irq_data *d)
{
struct its_device *its_dev = irq_data_get_irq_chip_data(d);
+ u32 event = its_get_event_id(d);
+
+ if (!irqd_is_forwarded_to_vcpu(d))
+ return NULL;
- return dev_event_to_col(its_dev, its_get_event_id(d));
+ return dev_event_to_vlpi_map(its_dev, event);
+}
+
+static int irq_to_cpuid(struct irq_data *d)
+{
+ struct its_device *its_dev = irq_data_get_irq_chip_data(d);
+ struct its_vlpi_map *map = get_vlpi_map(d);
+
+ if (map)
+ return map->vpe->col_idx;
+
+ return its_dev->event_map.col_map[its_get_event_id(d)];
}
static struct its_collection *valid_col(struct its_collection *col)
@@ -1270,17 +1285,6 @@ static void its_send_invdb(struct its_node *its, struct its_vpe *vpe)
/*
* irqchip functions - assumes MSI, mostly.
*/
-static struct its_vlpi_map *get_vlpi_map(struct irq_data *d)
-{
- struct its_device *its_dev = irq_data_get_irq_chip_data(d);
- u32 event = its_get_event_id(d);
-
- if (!irqd_is_forwarded_to_vcpu(d))
- return NULL;
-
- return dev_event_to_vlpi_map(its_dev, event);
-}
-
static void lpi_write_config(struct irq_data *d, u8 clr, u8 set)
{
struct its_vlpi_map *map = get_vlpi_map(d);
@@ -1323,13 +1327,25 @@ static void wait_for_syncr(void __iomem *rdbase)
static void direct_lpi_inv(struct irq_data *d)
{
- struct its_collection *col;
+ struct its_vlpi_map *map = get_vlpi_map(d);
void __iomem *rdbase;
+ u64 val;
+
+ if (map) {
+ struct its_device *its_dev = irq_data_get_irq_chip_data(d);
+
+ WARN_ON(!is_v4_1(its_dev->its));
+
+ val = GICR_INVLPIR_V;
+ val |= FIELD_PREP(GICR_INVLPIR_VPEID, map->vpe->vpe_id);
+ val |= FIELD_PREP(GICR_INVLPIR_INTID, map->vintid);
+ } else {
+ val = d->hwirq;
+ }
/* Target the redistributor this LPI is currently routed to */
- col = irq_to_col(d);
- rdbase = per_cpu_ptr(gic_rdists->rdist, col->col_id)->rd_base;
- gic_write_lpir(d->hwirq, rdbase + GICR_INVLPIR);
+ rdbase = per_cpu_ptr(gic_rdists->rdist, irq_to_cpuid(d))->rd_base;
+ gic_write_lpir(val, rdbase + GICR_INVLPIR);
wait_for_syncr(rdbase);
}
@@ -1339,7 +1355,8 @@ static void lpi_update_config(struct irq_data *d, u8 clr, u8 set)
struct its_device *its_dev = irq_data_get_irq_chip_data(d);
lpi_write_config(d, clr, set);
- if (gic_rdists->has_direct_lpi && !irqd_is_forwarded_to_vcpu(d))
+ if (gic_rdists->has_direct_lpi &&
+ (is_v4_1(its_dev->its) || !irqd_is_forwarded_to_vcpu(d)))
direct_lpi_inv(d);
else if (!irqd_is_forwarded_to_vcpu(d))
its_send_inv(its_dev, its_get_event_id(d));
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index 37b14fb82773..a30ea5294784 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -249,6 +249,7 @@
#define GICR_TYPER_COMMON_LPI_AFF GENMASK_ULL(25, 24)
#define GICR_TYPER_AFFINITY GENMASK_ULL(63, 32)
+#define GICR_INVLPIR_INTID GENMASK_ULL(31, 0)
#define GICR_INVLPIR_VPEID GENMASK_ULL(47, 32)
#define GICR_INVLPIR_V GENMASK_ULL(63, 63)
--
2.20.1
The GICv4.1 ITS has yet another new command (VSGI) which allows
a VPE-targeted SGI to be configured (or have its pending state
cleared). Add support for this command and plumb it into the
activate irqdomain callback so that it is ready to be used.
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3-its.c | 76 +++++++++++++++++++++++++++++-
include/linux/irqchip/arm-gic-v3.h | 3 +-
2 files changed, 77 insertions(+), 2 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index b38823b9c27f..9ca2ad6000d8 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -354,6 +354,15 @@ struct its_cmd_desc {
struct {
struct its_vpe *vpe;
} its_invdb_cmd;
+
+ struct {
+ struct its_vpe *vpe;
+ u8 sgi;
+ u8 priority;
+ bool enable;
+ bool group;
+ bool clear;
+ } its_vsgi_cmd;
};
};
@@ -502,6 +511,31 @@ static void its_encode_db(struct its_cmd_block *cmd, bool db)
its_mask_encode(&cmd->raw_cmd[2], db, 63, 63);
}
+static void its_encode_sgi_intid(struct its_cmd_block *cmd, u8 sgi)
+{
+ its_mask_encode(&cmd->raw_cmd[0], sgi, 35, 32);
+}
+
+static void its_encode_sgi_priority(struct its_cmd_block *cmd, u8 prio)
+{
+ its_mask_encode(&cmd->raw_cmd[0], prio >> 4, 23, 20);
+}
+
+static void its_encode_sgi_group(struct its_cmd_block *cmd, bool grp)
+{
+ its_mask_encode(&cmd->raw_cmd[0], grp, 10, 10);
+}
+
+static void its_encode_sgi_clear(struct its_cmd_block *cmd, bool clr)
+{
+ its_mask_encode(&cmd->raw_cmd[0], clr, 9, 9);
+}
+
+static void its_encode_sgi_enable(struct its_cmd_block *cmd, bool en)
+{
+ its_mask_encode(&cmd->raw_cmd[0], en, 8, 8);
+}
+
static inline void its_fixup_cmd(struct its_cmd_block *cmd)
{
/* Let's fixup BE commands */
@@ -867,6 +901,26 @@ static struct its_vpe *its_build_invdb_cmd(struct its_node *its,
return valid_vpe(its, desc->its_invdb_cmd.vpe);
}
+static struct its_vpe *its_build_vsgi_cmd(struct its_node *its,
+ struct its_cmd_block *cmd,
+ struct its_cmd_desc *desc)
+{
+ if (WARN_ON(!is_v4_1(its)))
+ return NULL;
+
+ its_encode_cmd(cmd, GITS_CMD_VSGI);
+ its_encode_vpeid(cmd, desc->its_vsgi_cmd.vpe->vpe_id);
+ its_encode_sgi_intid(cmd, desc->its_vsgi_cmd.sgi);
+ its_encode_sgi_priority(cmd, desc->its_vsgi_cmd.priority);
+ its_encode_sgi_group(cmd, desc->its_vsgi_cmd.group);
+ its_encode_sgi_clear(cmd, desc->its_vsgi_cmd.clear);
+ its_encode_sgi_enable(cmd, desc->its_vsgi_cmd.enable);
+
+ its_fixup_cmd(cmd);
+
+ return valid_vpe(its, desc->its_vsgi_cmd.vpe);
+}
+
static u64 its_cmd_ptr_to_offset(struct its_node *its,
struct its_cmd_block *ptr)
{
@@ -3740,6 +3794,21 @@ static struct irq_chip its_vpe_4_1_irq_chip = {
.irq_set_vcpu_affinity = its_vpe_4_1_set_vcpu_affinity,
};
+static void its_configure_sgi(struct irq_data *d, bool clear)
+{
+ struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
+ struct its_cmd_desc desc;
+
+ desc.its_vsgi_cmd.vpe = vpe;
+ desc.its_vsgi_cmd.sgi = d->hwirq;
+ desc.its_vsgi_cmd.priority = vpe->sgi_config[d->hwirq].priority;
+ desc.its_vsgi_cmd.enable = vpe->sgi_config[d->hwirq].enabled;
+ desc.its_vsgi_cmd.group = vpe->sgi_config[d->hwirq].group;
+ desc.its_vsgi_cmd.clear = clear;
+
+ its_send_single_vcommand(find_4_1_its(), its_build_vsgi_cmd, &desc);
+}
+
static int its_sgi_set_affinity(struct irq_data *d,
const struct cpumask *mask_val,
bool force)
@@ -3785,13 +3854,18 @@ static void its_sgi_irq_domain_free(struct irq_domain *domain,
static int its_sgi_irq_domain_activate(struct irq_domain *domain,
struct irq_data *d, bool reserve)
{
+ /* Write out the initial SGI configuration */
+ its_configure_sgi(d, false);
return 0;
}
static void its_sgi_irq_domain_deactivate(struct irq_domain *domain,
struct irq_data *d)
{
- /* Nothing to do */
+ struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
+
+ vpe->sgi_config[d->hwirq].enabled = false;
+ its_configure_sgi(d, true);
}
static struct irq_domain_ops its_sgi_domain_ops = {
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index a30ea5294784..64818ed4c238 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -502,8 +502,9 @@
#define GITS_CMD_VMAPTI GITS_CMD_GICv4(GITS_CMD_MAPTI)
#define GITS_CMD_VMOVI GITS_CMD_GICv4(GITS_CMD_MOVI)
#define GITS_CMD_VSYNC GITS_CMD_GICv4(GITS_CMD_SYNC)
-/* VMOVP and INVDB are the odd ones, as they dont have a physical counterpart */
+/* VMOVP, VSGI and INVDB are the odd ones, as they dont have a physical counterpart */
#define GITS_CMD_VMOVP GITS_CMD_GICv4(2)
+#define GITS_CMD_VSGI GITS_CMD_GICv4(3)
#define GITS_CMD_INVDB GITS_CMD_GICv4(0xe)
/*
--
2.20.1
To implement the get/set_irqchip_state callbacks (limited to the
PENDING state), we have to use a particular set of hacks:
- Reading the pending state is done by using a pair of new redistributor
registers (GICR_VSGIR, GICR_VSGIPENDR), which allow the 16 interrupts
state to be retrieved.
- Setting the pending state is done by generating it as we'd otherwise do
for a guest (writing to GITS_SGIR)
- Clearing the pending state is done by emiting a VSGI command with the
"clear" bit set.
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3-its.c | 56 ++++++++++++++++++++++++++++++
include/linux/irqchip/arm-gic-v3.h | 14 ++++++++
2 files changed, 70 insertions(+)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index c13c889553c5..47d63a5b43ff 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -3832,11 +3832,67 @@ static int its_sgi_set_affinity(struct irq_data *d,
return -EINVAL;
}
+static int its_sgi_set_irqchip_state(struct irq_data *d,
+ enum irqchip_irq_state which,
+ bool state)
+{
+ if (which != IRQCHIP_STATE_PENDING)
+ return -EINVAL;
+
+ if (state) {
+ struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
+ struct its_node *its = find_4_1_its();
+ u64 val;
+
+ val = FIELD_PREP(GITS_SGIR_VPEID, vpe->vpe_id);
+ val |= FIELD_PREP(GITS_SGIR_VINTID, d->hwirq);
+ writeq_relaxed(val, its->sgir_base + GITS_SGIR - SZ_128K);
+ } else {
+ its_configure_sgi(d, true);
+ }
+
+ return 0;
+}
+
+static int its_sgi_get_irqchip_state(struct irq_data *d,
+ enum irqchip_irq_state which, bool *val)
+{
+ struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
+ void __iomem *base = gic_data_rdist_cpu(vpe->col_idx)->rd_base + SZ_128K;
+ u32 count = 1000000; /* 1s! */
+ u32 status;
+
+ if (which != IRQCHIP_STATE_PENDING)
+ return -EINVAL;
+
+ writel_relaxed(vpe->vpe_id, base + GICR_VSGIR);
+ do {
+ status = readl_relaxed(base + GICR_VSGIPENDR);
+ if (!(status & GICR_VSGIPENDR_BUSY))
+ goto out;
+
+ count--;
+ if (!count) {
+ pr_err_ratelimited("Unable to get SGI status\n");
+ goto out;
+ }
+ cpu_relax();
+ udelay(1);
+ } while(count);
+
+out:
+ *val = !!(status & (1 << d->hwirq));
+
+ return 0;
+}
+
static struct irq_chip its_sgi_irq_chip = {
.name = "GICv4.1-sgi",
.irq_mask = its_sgi_mask_irq,
.irq_unmask = its_sgi_unmask_irq,
.irq_set_affinity = its_sgi_set_affinity,
+ .irq_set_irqchip_state = its_sgi_set_irqchip_state,
+ .irq_get_irqchip_state = its_sgi_get_irqchip_state,
};
static int its_sgi_irq_domain_alloc(struct irq_domain *domain,
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index 64818ed4c238..a0566bd5516e 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -345,6 +345,15 @@
#define GICR_VPENDBASER_4_1_VGRP1EN (1ULL << 58)
#define GICR_VPENDBASER_4_1_VPEID GENMASK_ULL(15, 0)
+#define GICR_VSGIR 0x0080
+
+#define GICR_VSGIR_VPEID GENMASK(15, 0)
+
+#define GICR_VSGIPENDR 0x0088
+
+#define GICR_VSGIPENDR_BUSY (1U << 31)
+#define GICR_VSGIPENDR_PENDING GENMASK(15, 0)
+
/*
* ITS registers, offsets from ITS_base
*/
@@ -368,6 +377,11 @@
#define GITS_TRANSLATER 0x10040
+#define GITS_SGIR 0x20020
+
+#define GITS_SGIR_VPEID GENMASK_ULL(47, 32)
+#define GITS_SGIR_VINTID GENMASK_ULL(7, 0)
+
#define GITS_CTLR_ENABLE (1U << 0)
#define GITS_CTLR_ImDe (1U << 1)
#define GITS_CTLR_ITS_NUMBER_SHIFT 4
--
2.20.1
When descheduling a VPE, special care must be taken to tell the GIC
about whether we want to receive a doorbell or not. This is a
major improvement on GICv4.0, where the doorbell had to be separately
enabled/disabled.
Reviewed-by: Zenghui Yu <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3-its.c | 53 +++++++++++++++++++++++++-------
1 file changed, 42 insertions(+), 11 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 49850297fe56..a5567d099a3e 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -2631,7 +2631,7 @@ static int __init allocate_lpi_tables(void)
return 0;
}
-static u64 its_clear_vpend_valid(void __iomem *vlpi_base)
+static u64 its_clear_vpend_valid(void __iomem *vlpi_base, u64 clr, u64 set)
{
u32 count = 1000000; /* 1s! */
bool clean;
@@ -2639,6 +2639,8 @@ static u64 its_clear_vpend_valid(void __iomem *vlpi_base)
val = gits_read_vpendbaser(vlpi_base + GICR_VPENDBASER);
val &= ~GICR_VPENDBASER_Valid;
+ val &= ~clr;
+ val |= set;
gits_write_vpendbaser(val, vlpi_base + GICR_VPENDBASER);
do {
@@ -2651,6 +2653,11 @@ static u64 its_clear_vpend_valid(void __iomem *vlpi_base)
}
} while (!clean && count);
+ if (unlikely(val & GICR_VPENDBASER_Dirty)) {
+ pr_err_ratelimited("ITS virtual pending table not cleaning\n");
+ val |= GICR_VPENDBASER_PendingLast;
+ }
+
return val;
}
@@ -2759,7 +2766,7 @@ static void its_cpu_init_lpis(void)
* ancient programming gets left in and has possibility of
* corrupting memory.
*/
- val = its_clear_vpend_valid(vlpi_base);
+ val = its_clear_vpend_valid(vlpi_base, 0, 0);
WARN_ON(val & GICR_VPENDBASER_Dirty);
}
@@ -3439,16 +3446,10 @@ static void its_vpe_deschedule(struct its_vpe *vpe)
void __iomem *vlpi_base = gic_data_rdist_vlpi_base();
u64 val;
- val = its_clear_vpend_valid(vlpi_base);
+ val = its_clear_vpend_valid(vlpi_base, 0, 0);
- if (unlikely(val & GICR_VPENDBASER_Dirty)) {
- pr_err_ratelimited("ITS virtual pending table not cleaning\n");
- vpe->idai = false;
- vpe->pending_last = true;
- } else {
- vpe->idai = !!(val & GICR_VPENDBASER_IDAI);
- vpe->pending_last = !!(val & GICR_VPENDBASER_PendingLast);
- }
+ vpe->idai = !!(val & GICR_VPENDBASER_IDAI);
+ vpe->pending_last = !!(val & GICR_VPENDBASER_PendingLast);
}
static void its_vpe_invall(struct its_vpe *vpe)
@@ -3640,6 +3641,35 @@ static void its_vpe_4_1_schedule(struct its_vpe *vpe,
gits_write_vpendbaser(val, vlpi_base + GICR_VPENDBASER);
}
+static void its_vpe_4_1_deschedule(struct its_vpe *vpe,
+ struct its_cmd_info *info)
+{
+ void __iomem *vlpi_base = gic_data_rdist_vlpi_base();
+ u64 val;
+
+ if (info->req_db) {
+ /*
+ * vPE is going to block: make the vPE non-resident with
+ * PendingLast clear and DB set. The GIC guarantees that if
+ * we read-back PendingLast clear, then a doorbell will be
+ * delivered when an interrupt comes.
+ */
+ val = its_clear_vpend_valid(vlpi_base,
+ GICR_VPENDBASER_PendingLast,
+ GICR_VPENDBASER_4_1_DB);
+ vpe->pending_last = !!(val & GICR_VPENDBASER_PendingLast);
+ } else {
+ /*
+ * We're not blocking, so just make the vPE non-resident
+ * with PendingLast set, indicating that we'll be back.
+ */
+ val = its_clear_vpend_valid(vlpi_base,
+ 0,
+ GICR_VPENDBASER_PendingLast);
+ vpe->pending_last = true;
+ }
+}
+
static int its_vpe_4_1_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
{
struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
@@ -3651,6 +3681,7 @@ static int its_vpe_4_1_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
return 0;
case DESCHEDULE_VPE:
+ its_vpe_4_1_deschedule(vpe, info);
return 0;
case INVALL_VPE:
--
2.20.1
The GICv4.1 architecture gives the hypervisor the option to let
the guest choose whether it wants the good old SGIs with an
active state, or the new, HW-based ones that do not have one.
For this, plumb the configuration of SGIs into the GICv3 MMIO
handling, present the GICD_TYPER2.nASSGIcap to the guest,
and handle the GICD_CTLR.nASSGIreq setting.
Signed-off-by: Marc Zyngier <[email protected]>
---
virt/kvm/arm/vgic/vgic-mmio-v3.c | 48 ++++++++++++++++++++++++++++++--
1 file changed, 46 insertions(+), 2 deletions(-)
diff --git a/virt/kvm/arm/vgic/vgic-mmio-v3.c b/virt/kvm/arm/vgic/vgic-mmio-v3.c
index d73f4ffd7d36..31ff8c7e09b0 100644
--- a/virt/kvm/arm/vgic/vgic-mmio-v3.c
+++ b/virt/kvm/arm/vgic/vgic-mmio-v3.c
@@ -3,6 +3,7 @@
* VGICv3 MMIO handling functions
*/
+#include <linux/bitfield.h>
#include <linux/irqchip/arm-gic-v3.h>
#include <linux/kvm.h>
#include <linux/kvm_host.h>
@@ -70,6 +71,8 @@ static unsigned long vgic_mmio_read_v3_misc(struct kvm_vcpu *vcpu,
if (vgic->enabled)
value |= GICD_CTLR_ENABLE_SS_G1;
value |= GICD_CTLR_ARE_NS | GICD_CTLR_DS;
+ if (kvm_vgic_global_state.has_gicv4_1 && vgic->nassgireq)
+ value |= GICD_CTLR_nASSGIreq;
break;
case GICD_TYPER:
value = vgic->nr_spis + VGIC_NR_PRIVATE_IRQS;
@@ -81,6 +84,10 @@ static unsigned long vgic_mmio_read_v3_misc(struct kvm_vcpu *vcpu,
value |= (INTERRUPT_ID_BITS_SPIS - 1) << 19;
}
break;
+ case GICD_TYPER2:
+ if (kvm_vgic_global_state.has_gicv4_1)
+ value = GICD_TYPER2_nASSGIcap;
+ break;
case GICD_IIDR:
value = (PRODUCT_ID_KVM << GICD_IIDR_PRODUCT_ID_SHIFT) |
(vgic->implementation_rev << GICD_IIDR_REVISION_SHIFT) |
@@ -98,17 +105,43 @@ static void vgic_mmio_write_v3_misc(struct kvm_vcpu *vcpu,
unsigned long val)
{
struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
- bool was_enabled = dist->enabled;
switch (addr & 0x0c) {
- case GICD_CTLR:
+ case GICD_CTLR: {
+ bool was_enabled, is_hwsgi;
+
+ mutex_lock(&vcpu->kvm->lock);
+
+ was_enabled = dist->enabled;
+ is_hwsgi = dist->nassgireq;
+
dist->enabled = val & GICD_CTLR_ENABLE_SS_G1;
+ /* Not a GICv4.1? No HW SGIs */
+ if (!kvm_vgic_global_state.has_gicv4_1)
+ val &= ~GICD_CTLR_nASSGIreq;
+
+ /* Dist stays enabled? nASSGIreq is RO */
+ if (was_enabled && dist->enabled) {
+ val &= ~GICD_CTLR_nASSGIreq;
+ val |= FIELD_PREP(GICD_CTLR_nASSGIreq, is_hwsgi);
+ }
+
+ /* Switching HW SGIs? */
+ dist->nassgireq = val & GICD_CTLR_nASSGIreq;
+ if (is_hwsgi != dist->nassgireq)
+ vgic_v4_configure_vsgis(vcpu->kvm);
+
if (!was_enabled && dist->enabled)
vgic_kick_vcpus(vcpu->kvm);
+
+ mutex_unlock(&vcpu->kvm->lock);
break;
+ }
case GICD_TYPER:
+ case GICD_TYPER2:
case GICD_IIDR:
+ /* This is at best for documentation purposes... */
return;
}
}
@@ -117,10 +150,21 @@ static int vgic_mmio_uaccess_write_v3_misc(struct kvm_vcpu *vcpu,
gpa_t addr, unsigned int len,
unsigned long val)
{
+ struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+
switch (addr & 0x0c) {
case GICD_IIDR:
if (val != vgic_mmio_read_v3_misc(vcpu, addr, len))
return -EINVAL;
+ return 0;
+ case GICD_CTLR:
+ /* Not a GICv4.1? No HW SGIs */
+ if (!kvm_vgic_global_state.has_gicv4_1)
+ val &= ~GICD_CTLR_nASSGIreq;
+
+ dist->enabled = val & GICD_CTLR_ENABLE_SS_G1;
+ dist->nassgireq = val & GICD_CTLR_nASSGIreq;
+ return 0;
}
vgic_mmio_write_v3_misc(vcpu, addr, len, val);
--
2.20.1
Implement mask/unmask for virtual SGIs by calling into the
configuration helper.
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3-its.c | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 9ca2ad6000d8..c13c889553c5 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -3809,6 +3809,22 @@ static void its_configure_sgi(struct irq_data *d, bool clear)
its_send_single_vcommand(find_4_1_its(), its_build_vsgi_cmd, &desc);
}
+static void its_sgi_mask_irq(struct irq_data *d)
+{
+ struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
+
+ vpe->sgi_config[d->hwirq].enabled = false;
+ its_configure_sgi(d, false);
+}
+
+static void its_sgi_unmask_irq(struct irq_data *d)
+{
+ struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
+
+ vpe->sgi_config[d->hwirq].enabled = true;
+ its_configure_sgi(d, false);
+}
+
static int its_sgi_set_affinity(struct irq_data *d,
const struct cpumask *mask_val,
bool force)
@@ -3818,6 +3834,8 @@ static int its_sgi_set_affinity(struct irq_data *d,
static struct irq_chip its_sgi_irq_chip = {
.name = "GICv4.1-sgi",
+ .irq_mask = its_sgi_mask_irq,
+ .irq_unmask = its_sgi_unmask_irq,
.irq_set_affinity = its_sgi_set_affinity,
};
--
2.20.1
Since GICv4.1 has the capability to inject 16 SGIs into each VPE,
and that I'm keen not to invent too many specific interfaces to
manupulate these interrupts, let's pretend that each of these SGIs
is an actual Linux interrupt.
For that matter, let's introduce a minimal irqchip and irqdomain
setup that will get fleshed up in the following patches.
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3-its.c | 68 +++++++++++++++++++++++++++++-
drivers/irqchip/irq-gic-v4.c | 8 +++-
include/linux/irqchip/arm-gic-v4.h | 9 +++-
3 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 2f94fae67dfd..b38823b9c27f 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -3740,6 +3740,67 @@ static struct irq_chip its_vpe_4_1_irq_chip = {
.irq_set_vcpu_affinity = its_vpe_4_1_set_vcpu_affinity,
};
+static int its_sgi_set_affinity(struct irq_data *d,
+ const struct cpumask *mask_val,
+ bool force)
+{
+ return -EINVAL;
+}
+
+static struct irq_chip its_sgi_irq_chip = {
+ .name = "GICv4.1-sgi",
+ .irq_set_affinity = its_sgi_set_affinity,
+};
+
+static int its_sgi_irq_domain_alloc(struct irq_domain *domain,
+ unsigned int virq, unsigned int nr_irqs,
+ void *args)
+{
+ struct its_vpe *vpe = args;
+ int i;
+
+ /* Yes, we do want 16 SGIs */
+ WARN_ON(nr_irqs != 16);
+
+ for (i = 0; i < 16; i++) {
+ vpe->sgi_config[i].priority = 0;
+ vpe->sgi_config[i].enabled = false;
+ vpe->sgi_config[i].group = false;
+
+ irq_domain_set_hwirq_and_chip(domain, virq + i, i,
+ &its_sgi_irq_chip, vpe);
+ irq_set_status_flags(virq + i, IRQ_DISABLE_UNLAZY);
+ }
+
+ return 0;
+}
+
+static void its_sgi_irq_domain_free(struct irq_domain *domain,
+ unsigned int virq,
+ unsigned int nr_irqs)
+{
+ /* Nothing to do */
+}
+
+static int its_sgi_irq_domain_activate(struct irq_domain *domain,
+ struct irq_data *d, bool reserve)
+{
+ return 0;
+}
+
+static void its_sgi_irq_domain_deactivate(struct irq_domain *domain,
+ struct irq_data *d)
+{
+ /* Nothing to do */
+}
+
+static struct irq_domain_ops its_sgi_domain_ops = {
+ .alloc = its_sgi_irq_domain_alloc,
+ .free = its_sgi_irq_domain_free,
+ .activate = its_sgi_irq_domain_activate,
+ .deactivate = its_sgi_irq_domain_deactivate,
+};
+
static int its_vpe_id_alloc(void)
{
return ida_simple_get(&its_vpeid_ida, 0, ITS_MAX_VPEID, GFP_KERNEL);
@@ -4781,8 +4842,13 @@ int __init its_init(struct fwnode_handle *handle, struct rdists *rdists,
rdists->has_rvpeid = false;
if (has_v4 & rdists->has_vlpis) {
+ struct irq_domain_ops *sgi_ops = NULL;
+
+ if (has_v4_1)
+ sgi_ops = &its_sgi_domain_ops;
+
if (its_init_vpe_domain() ||
- its_init_v4(parent_domain, &its_vpe_domain_ops)) {
+ its_init_v4(parent_domain, &its_vpe_domain_ops, sgi_ops)) {
rdists->has_vlpis = false;
pr_err("ITS: Disabling GICv4 support\n");
}
diff --git a/drivers/irqchip/irq-gic-v4.c b/drivers/irqchip/irq-gic-v4.c
index 45969927cc81..c01910d53f9e 100644
--- a/drivers/irqchip/irq-gic-v4.c
+++ b/drivers/irqchip/irq-gic-v4.c
@@ -85,6 +85,7 @@
static struct irq_domain *gic_domain;
static const struct irq_domain_ops *vpe_domain_ops;
+static const struct irq_domain_ops *sgi_domain_ops;
int its_alloc_vcpu_irqs(struct its_vm *vm)
{
@@ -216,12 +217,15 @@ int its_prop_update_vlpi(int irq, u8 config, bool inv)
return irq_set_vcpu_affinity(irq, &info);
}
-int its_init_v4(struct irq_domain *domain, const struct irq_domain_ops *ops)
+int its_init_v4(struct irq_domain *domain,
+ const struct irq_domain_ops *vpe_ops,
+ const struct irq_domain_ops *sgi_ops)
{
if (domain) {
pr_info("ITS: Enabling GICv4 support\n");
gic_domain = domain;
- vpe_domain_ops = ops;
+ vpe_domain_ops = vpe_ops;
+ sgi_domain_ops = sgi_ops;
return 0;
}
diff --git a/include/linux/irqchip/arm-gic-v4.h b/include/linux/irqchip/arm-gic-v4.h
index d9c34968467a..30b4855bf766 100644
--- a/include/linux/irqchip/arm-gic-v4.h
+++ b/include/linux/irqchip/arm-gic-v4.h
@@ -49,6 +49,11 @@ struct its_vpe {
};
/* GICv4.1 implementations */
struct {
+ struct {
+ u8 priority;
+ bool enabled;
+ bool group;
+ } sgi_config[16];
atomic_t vmapp_count;
};
};
@@ -118,6 +123,8 @@ int its_unmap_vlpi(int irq);
int its_prop_update_vlpi(int irq, u8 config, bool inv);
struct irq_domain_ops;
-int its_init_v4(struct irq_domain *domain, const struct irq_domain_ops *ops);
+int its_init_v4(struct irq_domain *domain,
+ const struct irq_domain_ops *vpe_ops,
+ const struct irq_domain_ops *sgi_ops);
#endif
--
2.20.1
In order to let a guest buy in the new, active-less SGIs, we
need to be able to switch between the two modes.
Handle this by stopping all guest activity, transfer the state
from one mode to the other, and resume the guest.
Signed-off-by: Marc Zyngier <[email protected]>
---
include/kvm/arm_vgic.h | 3 ++
virt/kvm/arm/vgic/vgic-v3.c | 2 +
virt/kvm/arm/vgic/vgic-v4.c | 96 +++++++++++++++++++++++++++++++++++++
virt/kvm/arm/vgic/vgic.h | 1 +
4 files changed, 102 insertions(+)
diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
index 63457908c9c4..69f4164d6477 100644
--- a/include/kvm/arm_vgic.h
+++ b/include/kvm/arm_vgic.h
@@ -231,6 +231,9 @@ struct vgic_dist {
/* distributor enabled */
bool enabled;
+ /* Wants SGIs without active state */
+ bool nassgireq;
+
struct vgic_irq *spis;
struct vgic_io_device dist_iodev;
diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
index c2fdea201747..c79a251c4974 100644
--- a/virt/kvm/arm/vgic/vgic-v3.c
+++ b/virt/kvm/arm/vgic/vgic-v3.c
@@ -540,6 +540,8 @@ int vgic_v3_map_resources(struct kvm *kvm)
goto out;
}
+ if (kvm_vgic_global_state.has_gicv4_1)
+ vgic_v4_configure_vsgis(kvm);
dist->ready = true;
out:
diff --git a/virt/kvm/arm/vgic/vgic-v4.c b/virt/kvm/arm/vgic/vgic-v4.c
index c2fcde104ea2..063785fd2dc7 100644
--- a/virt/kvm/arm/vgic/vgic-v4.c
+++ b/virt/kvm/arm/vgic/vgic-v4.c
@@ -97,6 +97,102 @@ static irqreturn_t vgic_v4_doorbell_handler(int irq, void *info)
return IRQ_HANDLED;
}
+static void vgic_v4_sync_sgi_config(struct its_vpe *vpe, struct vgic_irq *irq)
+{
+ vpe->sgi_config[irq->intid].enabled = irq->enabled;
+ vpe->sgi_config[irq->intid].group = irq->group;
+ vpe->sgi_config[irq->intid].priority = irq->priority;
+}
+
+static void vgic_v4_enable_vsgis(struct kvm_vcpu *vcpu)
+{
+ struct its_vpe *vpe = &vcpu->arch.vgic_cpu.vgic_v3.its_vpe;
+ int i;
+
+ /*
+ * With GICv4.1, every virtual SGI can be directly injected. So
+ * let's pretend that they are HW interrupts, tied to a host
+ * IRQ. The SGI code will do its magic.
+ */
+ for (i = 0; i < VGIC_NR_SGIS; i++) {
+ struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, i);
+ struct irq_desc *desc;
+ int ret;
+
+ if (irq->hw) {
+ vgic_put_irq(vcpu->kvm, irq);
+ continue;
+ }
+
+ irq->hw = true;
+ irq->host_irq = irq_find_mapping(vpe->sgi_domain, i);
+ vgic_v4_sync_sgi_config(vpe, irq);
+ /*
+ * SGIs are initialised as disabled. Enable them if
+ * required by the rest of the VGIC init code.
+ */
+ desc = irq_to_desc(irq->host_irq);
+ ret = irq_domain_activate_irq(irq_desc_get_irq_data(desc),
+ false);
+ if (!WARN_ON(ret)) {
+ /* Transfer pending state */
+ ret = irq_set_irqchip_state(irq->host_irq,
+ IRQCHIP_STATE_PENDING,
+ irq->pending_latch);
+ WARN_ON(ret);
+ irq->pending_latch = false;
+ }
+
+ vgic_put_irq(vcpu->kvm, irq);
+ }
+}
+
+static void vgic_v4_disable_vsgis(struct kvm_vcpu *vcpu)
+{
+ int i;
+
+ for (i = 0; i < VGIC_NR_SGIS; i++) {
+ struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, i);
+ struct irq_desc *desc;
+ int ret;
+
+ if (!irq->hw) {
+ vgic_put_irq(vcpu->kvm, irq);
+ continue;
+ }
+
+ irq->hw = false;
+ ret = irq_get_irqchip_state(irq->host_irq,
+ IRQCHIP_STATE_PENDING,
+ &irq->pending_latch);
+ WARN_ON(ret);
+
+ desc = irq_to_desc(irq->host_irq);
+ irq_domain_deactivate_irq(irq_desc_get_irq_data(desc));
+
+ vgic_put_irq(vcpu->kvm, irq);
+ }
+}
+
+/* Must be called with the kvm lock held */
+void vgic_v4_configure_vsgis(struct kvm *kvm)
+{
+ struct vgic_dist *dist = &kvm->arch.vgic;
+ struct kvm_vcpu *vcpu;
+ int i;
+
+ kvm_arm_halt_guest(kvm);
+
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ if (dist->nassgireq)
+ vgic_v4_enable_vsgis(vcpu);
+ else
+ vgic_v4_disable_vsgis(vcpu);
+ }
+
+ kvm_arm_resume_guest(kvm);
+}
+
/**
* vgic_v4_init - Initialize the GICv4 data structures
* @kvm: Pointer to the VM being initialized
diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
index c7fefd6b1c80..769e4802645e 100644
--- a/virt/kvm/arm/vgic/vgic.h
+++ b/virt/kvm/arm/vgic/vgic.h
@@ -316,5 +316,6 @@ void vgic_its_invalidate_cache(struct kvm *kvm);
bool vgic_supports_direct_msis(struct kvm *kvm);
int vgic_v4_init(struct kvm *kvm);
void vgic_v4_teardown(struct kvm *kvm);
+void vgic_v4_configure_vsgis(struct kvm *kvm);
#endif
--
2.20.1
Making a VPE resident on GICv4.1 is pretty simple, as it is just a
single write to the local redistributor. We just need extra information
about which groups to enable, which the KVM code will have to provide.
Reviewed-by: Zenghui Yu <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3-its.c | 17 +++++++++++++++++
include/linux/irqchip/arm-gic-v3.h | 9 +++++++++
include/linux/irqchip/arm-gic-v4.h | 5 +++++
3 files changed, 31 insertions(+)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 0da6baa48153..49850297fe56 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -3625,12 +3625,29 @@ static void its_vpe_4_1_unmask_irq(struct irq_data *d)
its_vpe_4_1_send_inv(d);
}
+static void its_vpe_4_1_schedule(struct its_vpe *vpe,
+ struct its_cmd_info *info)
+{
+ void __iomem *vlpi_base = gic_data_rdist_vlpi_base();
+ u64 val = 0;
+
+ /* Schedule the VPE */
+ val |= GICR_VPENDBASER_Valid;
+ val |= info->g0en ? GICR_VPENDBASER_4_1_VGRP0EN : 0;
+ val |= info->g1en ? GICR_VPENDBASER_4_1_VGRP1EN : 0;
+ val |= FIELD_PREP(GICR_VPENDBASER_4_1_VPEID, vpe->vpe_id);
+
+ gits_write_vpendbaser(val, vlpi_base + GICR_VPENDBASER);
+}
+
static int its_vpe_4_1_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
{
+ struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
struct its_cmd_info *info = vcpu_info;
switch (info->cmd_type) {
case SCHEDULE_VPE:
+ its_vpe_4_1_schedule(vpe, info);
return 0;
case DESCHEDULE_VPE:
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index 33519b7c96cf..1c91dede7663 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -329,6 +329,15 @@
#define GICR_VPENDBASER_IDAI (1ULL << 62)
#define GICR_VPENDBASER_Valid (1ULL << 63)
+/*
+ * GICv4.1 VPENDBASER, used for VPE residency. On top of these fields,
+ * also use the above Valid, PendingLast and Dirty.
+ */
+#define GICR_VPENDBASER_4_1_DB (1ULL << 62)
+#define GICR_VPENDBASER_4_1_VGRP0EN (1ULL << 59)
+#define GICR_VPENDBASER_4_1_VGRP1EN (1ULL << 58)
+#define GICR_VPENDBASER_4_1_VPEID GENMASK_ULL(15, 0)
+
/*
* ITS registers, offsets from ITS_base
*/
diff --git a/include/linux/irqchip/arm-gic-v4.h b/include/linux/irqchip/arm-gic-v4.h
index 498e523085a7..d9c34968467a 100644
--- a/include/linux/irqchip/arm-gic-v4.h
+++ b/include/linux/irqchip/arm-gic-v4.h
@@ -100,6 +100,11 @@ struct its_cmd_info {
union {
struct its_vlpi_map *map;
u8 config;
+ bool req_db;
+ struct {
+ bool g0en;
+ bool g1en;
+ };
};
};
--
2.20.1
Tell KVM that we support v4.1. Nothing uses this information so far.
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3-its.c | 9 ++++++++-
drivers/irqchip/irq-gic-v3.c | 2 ++
include/linux/irqchip/arm-gic-common.h | 2 ++
3 files changed, 12 insertions(+), 1 deletion(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 360ca2c1533f..48d7ee443544 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -4739,6 +4739,7 @@ int __init its_init(struct fwnode_handle *handle, struct rdists *rdists,
struct device_node *of_node;
struct its_node *its;
bool has_v4 = false;
+ bool has_v4_1 = false;
int err;
gic_rdists = rdists;
@@ -4759,8 +4760,14 @@ int __init its_init(struct fwnode_handle *handle, struct rdists *rdists,
if (err)
return err;
- list_for_each_entry(its, &its_nodes, entry)
+ list_for_each_entry(its, &its_nodes, entry) {
has_v4 |= is_v4(its);
+ has_v4_1 |= is_v4_1(its);
+ }
+
+ /* Don't bother with inconsistent systems */
+ if (WARN_ON(!has_v4_1 && rdists->has_rvpeid))
+ rdists->has_rvpeid = false;
if (has_v4 & rdists->has_vlpis) {
if (its_init_vpe_domain() ||
diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index 624f351c0362..fcbd220c92b3 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -1784,6 +1784,7 @@ static void __init gic_of_setup_kvm_info(struct device_node *node)
gic_v3_kvm_info.vcpu = r;
gic_v3_kvm_info.has_v4 = gic_data.rdists.has_vlpis;
+ gic_v3_kvm_info.has_v4_1 = gic_data.rdists.has_rvpeid;
gic_set_kvm_info(&gic_v3_kvm_info);
}
@@ -2094,6 +2095,7 @@ static void __init gic_acpi_setup_kvm_info(void)
}
gic_v3_kvm_info.has_v4 = gic_data.rdists.has_vlpis;
+ gic_v3_kvm_info.has_v4_1 = gic_data.rdists.has_rvpeid;
gic_set_kvm_info(&gic_v3_kvm_info);
}
diff --git a/include/linux/irqchip/arm-gic-common.h b/include/linux/irqchip/arm-gic-common.h
index b9850f5f1906..fa8c0455c352 100644
--- a/include/linux/irqchip/arm-gic-common.h
+++ b/include/linux/irqchip/arm-gic-common.h
@@ -32,6 +32,8 @@ struct gic_kvm_info {
struct resource vctrl;
/* vlpi support */
bool has_v4;
+ /* rvpeid support */
+ bool has_v4_1;
};
const struct gic_kvm_info *gic_get_kvm_info(void);
--
2.20.1
GICv4.1 redistributors have a VPE-aware INVALL register. Progress!
We can now emulate a guest-requested INVALL without emiting a
VINVALL command.
Reviewed-by: Zenghui Yu <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3-its.c | 14 ++++++++++++++
include/linux/irqchip/arm-gic-v3.h | 6 ++++++
2 files changed, 20 insertions(+)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index a5567d099a3e..403f5753e1ed 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -3670,6 +3670,19 @@ static void its_vpe_4_1_deschedule(struct its_vpe *vpe,
}
}
+static void its_vpe_4_1_invall(struct its_vpe *vpe)
+{
+ void __iomem *rdbase;
+ u64 val;
+
+ val = GICR_INVALLR_V;
+ val |= FIELD_PREP(GICR_INVALLR_VPEID, vpe->vpe_id);
+
+ /* Target the redistributor this vPE is currently known on */
+ rdbase = per_cpu_ptr(gic_rdists->rdist, vpe->col_idx)->rd_base;
+ gic_write_lpir(val, rdbase + GICR_INVALLR);
+}
+
static int its_vpe_4_1_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
{
struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
@@ -3685,6 +3698,7 @@ static int its_vpe_4_1_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
return 0;
case INVALL_VPE:
+ its_vpe_4_1_invall(vpe);
return 0;
default:
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index 1c91dede7663..37b14fb82773 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -249,6 +249,12 @@
#define GICR_TYPER_COMMON_LPI_AFF GENMASK_ULL(25, 24)
#define GICR_TYPER_AFFINITY GENMASK_ULL(63, 32)
+#define GICR_INVLPIR_VPEID GENMASK_ULL(47, 32)
+#define GICR_INVLPIR_V GENMASK_ULL(63, 63)
+
+#define GICR_INVALLR_VPEID GICR_INVLPIR_VPEID
+#define GICR_INVALLR_V GICR_INVLPIR_V
+
#define GIC_V3_REDIST_SIZE 0x20000
#define LPI_PROP_GROUP1 (1 << 1)
--
2.20.1
One of the new features of GICv4.1 is to allow virtual SGIs to be
directly signaled to a VPE. For that, the ITS has grown a new
64kB page containing only a single register that is used to
signal a SGI to a given VPE.
Add a second mapping covering this new 64kB range, and take this
opportunity to limit the original mapping to 64kB, which is enough
to cover the span of the ITS registers.
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3-its.c | 15 +++++++++++++--
1 file changed, 13 insertions(+), 2 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 48d7ee443544..2f94fae67dfd 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -97,6 +97,7 @@ struct its_node {
struct mutex dev_alloc_lock;
struct list_head entry;
void __iomem *base;
+ void __iomem *sgir_base;
phys_addr_t phys_base;
struct its_cmd_block *cmd_base;
struct its_cmd_block *cmd_write;
@@ -4325,7 +4326,7 @@ static int __init its_probe_one(struct resource *res,
struct page *page;
int err;
- its_base = ioremap(res->start, resource_size(res));
+ its_base = ioremap(res->start, SZ_64K);
if (!its_base) {
pr_warn("ITS@%pa: Unable to map ITS registers\n", &res->start);
return -ENOMEM;
@@ -4376,6 +4377,13 @@ static int __init its_probe_one(struct resource *res,
if (is_v4_1(its)) {
u32 svpet = FIELD_GET(GITS_TYPER_SVPET, typer);
+
+ its->sgir_base = ioremap(res->start + SZ_128K, SZ_64K);
+ if (!its->sgir_base) {
+ err = -ENOMEM;
+ goto out_free_its;
+ }
+
its->mpidr = readl_relaxed(its_base + GITS_MPIDR);
pr_info("ITS@%pa: Using GICv4.1 mode %08x %08x\n",
@@ -4389,7 +4397,7 @@ static int __init its_probe_one(struct resource *res,
get_order(ITS_CMD_QUEUE_SZ));
if (!page) {
err = -ENOMEM;
- goto out_free_its;
+ goto out_unmap_sgir;
}
its->cmd_base = (void *)page_address(page);
its->cmd_write = its->cmd_base;
@@ -4456,6 +4464,9 @@ static int __init its_probe_one(struct resource *res,
its_free_tables(its);
out_free_cmd:
free_pages((unsigned long)its->cmd_base, get_order(ITS_CMD_QUEUE_SZ));
+out_unmap_sgir:
+ if (its->sgir_base)
+ iounmap(its->sgir_base);
out_free_its:
kfree(its);
out_unmap:
--
2.20.1
As for VLPIs, there is a number of configuration bits that cannot
be directly communicated through the normal irqchip API, and we
have to use our good old friend set_vcpu_affinity.
This is used to configure group and priority for a given vSGI.
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3-its.c | 18 ++++++++++++++++++
include/linux/irqchip/arm-gic-v4.h | 5 +++++
2 files changed, 23 insertions(+)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 47d63a5b43ff..5126bdcfe079 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -3886,6 +3886,23 @@ static int its_sgi_get_irqchip_state(struct irq_data *d,
return 0;
}
+static int its_sgi_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
+{
+ struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
+ struct its_cmd_info *info = vcpu_info;
+
+ switch (info->cmd_type) {
+ case PROP_UPDATE_SGI:
+ vpe->sgi_config[d->hwirq].priority = info->priority;
+ vpe->sgi_config[d->hwirq].group = info->group;
+ its_configure_sgi(d, false);
+ return 0;
+
+ default:
+ return -EINVAL;
+ }
+}
+
static struct irq_chip its_sgi_irq_chip = {
.name = "GICv4.1-sgi",
.irq_mask = its_sgi_mask_irq,
@@ -3893,6 +3910,7 @@ static struct irq_chip its_sgi_irq_chip = {
.irq_set_affinity = its_sgi_set_affinity,
.irq_set_irqchip_state = its_sgi_set_irqchip_state,
.irq_get_irqchip_state = its_sgi_get_irqchip_state,
+ .irq_set_vcpu_affinity = its_sgi_set_vcpu_affinity,
};
static int its_sgi_irq_domain_alloc(struct irq_domain *domain,
diff --git a/include/linux/irqchip/arm-gic-v4.h b/include/linux/irqchip/arm-gic-v4.h
index 30b4855bf766..a1a9d40266f5 100644
--- a/include/linux/irqchip/arm-gic-v4.h
+++ b/include/linux/irqchip/arm-gic-v4.h
@@ -98,6 +98,7 @@ enum its_vcpu_info_cmd_type {
SCHEDULE_VPE,
DESCHEDULE_VPE,
INVALL_VPE,
+ PROP_UPDATE_SGI,
};
struct its_cmd_info {
@@ -110,6 +111,10 @@ struct its_cmd_info {
bool g0en;
bool g1en;
};
+ struct {
+ u8 priority;
+ bool group;
+ };
};
};
--
2.20.1
Most of the GICv3 emulation code that deals with SGIs now has to be
aware of the v4.1 capabilities in order to benefit from it.
Add such support, keyed on the interrupt having the hw flag set and
being a SGI.
Signed-off-by: Marc Zyngier <[email protected]>
---
virt/kvm/arm/vgic/vgic-mmio-v3.c | 15 +++++-
virt/kvm/arm/vgic/vgic-mmio.c | 88 ++++++++++++++++++++++++++++++--
2 files changed, 96 insertions(+), 7 deletions(-)
diff --git a/virt/kvm/arm/vgic/vgic-mmio-v3.c b/virt/kvm/arm/vgic/vgic-mmio-v3.c
index 7dfd15dbb308..d73f4ffd7d36 100644
--- a/virt/kvm/arm/vgic/vgic-mmio-v3.c
+++ b/virt/kvm/arm/vgic/vgic-mmio-v3.c
@@ -6,6 +6,7 @@
#include <linux/irqchip/arm-gic-v3.h>
#include <linux/kvm.h>
#include <linux/kvm_host.h>
+#include <linux/interrupt.h>
#include <kvm/iodev.h>
#include <kvm/arm_vgic.h>
@@ -939,8 +940,18 @@ void vgic_v3_dispatch_sgi(struct kvm_vcpu *vcpu, u64 reg, bool allow_group1)
* generate interrupts of either group.
*/
if (!irq->group || allow_group1) {
- irq->pending_latch = true;
- vgic_queue_irq_unlock(vcpu->kvm, irq, flags);
+ if (!irq->hw) {
+ irq->pending_latch = true;
+ vgic_queue_irq_unlock(vcpu->kvm, irq, flags);
+ } else {
+ /* HW SGI? Ask the GIC to inject it */
+ int err;
+ err = irq_set_irqchip_state(irq->host_irq,
+ IRQCHIP_STATE_PENDING,
+ true);
+ WARN_RATELIMIT(err, "IRQ %d", irq->host_irq);
+ raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
+ }
} else {
raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
}
diff --git a/virt/kvm/arm/vgic/vgic-mmio.c b/virt/kvm/arm/vgic/vgic-mmio.c
index 0d090482720d..6ebf747a7806 100644
--- a/virt/kvm/arm/vgic/vgic-mmio.c
+++ b/virt/kvm/arm/vgic/vgic-mmio.c
@@ -5,6 +5,8 @@
#include <linux/bitops.h>
#include <linux/bsearch.h>
+#include <linux/interrupt.h>
+#include <linux/irq.h>
#include <linux/kvm.h>
#include <linux/kvm_host.h>
#include <kvm/iodev.h>
@@ -59,6 +61,11 @@ unsigned long vgic_mmio_read_group(struct kvm_vcpu *vcpu,
return value;
}
+static void vgic_update_vsgi(struct vgic_irq *irq)
+{
+ WARN_ON(its_prop_update_vsgi(irq->host_irq, irq->priority, irq->group));
+}
+
void vgic_mmio_write_group(struct kvm_vcpu *vcpu, gpa_t addr,
unsigned int len, unsigned long val)
{
@@ -71,7 +78,12 @@ void vgic_mmio_write_group(struct kvm_vcpu *vcpu, gpa_t addr,
raw_spin_lock_irqsave(&irq->irq_lock, flags);
irq->group = !!(val & BIT(i));
- vgic_queue_irq_unlock(vcpu->kvm, irq, flags);
+ if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
+ vgic_update_vsgi(irq);
+ raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
+ } else {
+ vgic_queue_irq_unlock(vcpu->kvm, irq, flags);
+ }
vgic_put_irq(vcpu->kvm, irq);
}
@@ -113,7 +125,21 @@ void vgic_mmio_write_senable(struct kvm_vcpu *vcpu,
struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, intid + i);
raw_spin_lock_irqsave(&irq->irq_lock, flags);
- if (vgic_irq_is_mapped_level(irq)) {
+ if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
+ if (!irq->enabled) {
+ struct irq_data *data;
+
+ irq->enabled = true;
+ data = &irq_to_desc(irq->host_irq)->irq_data;
+ while (irqd_irq_disabled(data))
+ enable_irq(irq->host_irq);
+ }
+
+ raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
+ vgic_put_irq(vcpu->kvm, irq);
+
+ continue;
+ } else if (vgic_irq_is_mapped_level(irq)) {
bool was_high = irq->line_level;
/*
@@ -148,6 +174,8 @@ void vgic_mmio_write_cenable(struct kvm_vcpu *vcpu,
struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, intid + i);
raw_spin_lock_irqsave(&irq->irq_lock, flags);
+ if (irq->hw && vgic_irq_is_sgi(irq->intid) && irq->enabled)
+ disable_irq_nosync(irq->host_irq);
irq->enabled = false;
@@ -167,10 +195,22 @@ unsigned long vgic_mmio_read_pending(struct kvm_vcpu *vcpu,
for (i = 0; i < len * 8; i++) {
struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, intid + i);
unsigned long flags;
+ bool val;
raw_spin_lock_irqsave(&irq->irq_lock, flags);
- if (irq_is_pending(irq))
- value |= (1U << i);
+ if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
+ int err;
+
+ val = false;
+ err = irq_get_irqchip_state(irq->host_irq,
+ IRQCHIP_STATE_PENDING,
+ &val);
+ WARN_RATELIMIT(err, "IRQ %d", irq->host_irq);
+ } else {
+ val = irq_is_pending(irq);
+ }
+
+ value |= ((u32)val << i);
raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
vgic_put_irq(vcpu->kvm, irq);
@@ -236,6 +276,21 @@ void vgic_mmio_write_spending(struct kvm_vcpu *vcpu,
}
raw_spin_lock_irqsave(&irq->irq_lock, flags);
+
+ if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
+ /* HW SGI? Ask the GIC to inject it */
+ int err;
+ err = irq_set_irqchip_state(irq->host_irq,
+ IRQCHIP_STATE_PENDING,
+ true);
+ WARN_RATELIMIT(err, "IRQ %d", irq->host_irq);
+
+ raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
+ vgic_put_irq(vcpu->kvm, irq);
+
+ continue;
+ }
+
if (irq->hw)
vgic_hw_irq_spending(vcpu, irq, is_uaccess);
else
@@ -290,6 +345,20 @@ void vgic_mmio_write_cpending(struct kvm_vcpu *vcpu,
raw_spin_lock_irqsave(&irq->irq_lock, flags);
+ if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
+ /* HW SGI? Ask the GIC to inject it */
+ int err;
+ err = irq_set_irqchip_state(irq->host_irq,
+ IRQCHIP_STATE_PENDING,
+ false);
+ WARN_RATELIMIT(err, "IRQ %d", irq->host_irq);
+
+ raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
+ vgic_put_irq(vcpu->kvm, irq);
+
+ continue;
+ }
+
if (irq->hw)
vgic_hw_irq_cpending(vcpu, irq, is_uaccess);
else
@@ -339,8 +408,15 @@ static void vgic_mmio_change_active(struct kvm_vcpu *vcpu, struct vgic_irq *irq,
raw_spin_lock_irqsave(&irq->irq_lock, flags);
- if (irq->hw) {
+ if (irq->hw && !vgic_irq_is_sgi(irq->intid)) {
vgic_hw_irq_change_active(vcpu, irq, active, !requester_vcpu);
+ } else if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
+ /*
+ * GICv4.1 VSGI feature doesn't track an active state,
+ * so let's not kid ourselves, there is nothing we can
+ * do here.
+ */
+ irq->active = false;
} else {
u32 model = vcpu->kvm->arch.vgic.vgic_model;
u8 active_source;
@@ -514,6 +590,8 @@ void vgic_mmio_write_priority(struct kvm_vcpu *vcpu,
raw_spin_lock_irqsave(&irq->irq_lock, flags);
/* Narrow the priority range to what we actually support */
irq->priority = (val >> (i * 8)) & GENMASK(7, 8 - VGIC_PRI_BITS);
+ if (irq->hw && vgic_irq_is_sgi(irq->intid))
+ vgic_update_vsgi(irq);
raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
vgic_put_irq(vcpu->kvm, irq);
--
2.20.1
Since GICv4.1 gives us a per-VPE doorbell, avoid programming anything
else on VMOVI/VMAPI/VMAPTI and on any other action that would have
otherwise resulted in a per-VLPI doorbell to be programmed.
Reviewed-by: Zenghui Yu <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3-its.c | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 403f5753e1ed..f00c8ddd3798 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -719,7 +719,7 @@ static struct its_vpe *its_build_vmapti_cmd(struct its_node *its,
{
u32 db;
- if (desc->its_vmapti_cmd.db_enabled)
+ if (!is_v4_1(its) && desc->its_vmapti_cmd.db_enabled)
db = desc->its_vmapti_cmd.vpe->vpe_db_lpi;
else
db = 1023;
@@ -742,7 +742,7 @@ static struct its_vpe *its_build_vmovi_cmd(struct its_node *its,
{
u32 db;
- if (desc->its_vmovi_cmd.db_enabled)
+ if (!is_v4_1(its) && desc->its_vmovi_cmd.db_enabled)
db = desc->its_vmovi_cmd.vpe->vpe_db_lpi;
else
db = 1023;
@@ -1353,6 +1353,13 @@ static void its_vlpi_set_doorbell(struct irq_data *d, bool enable)
u32 event = its_get_event_id(d);
struct its_vlpi_map *map;
+ /*
+ * GICv4.1 does away with the per-LPI nonsense, nothing to do
+ * here.
+ */
+ if (is_v4_1(its_dev->its))
+ return;
+
map = dev_event_to_vlpi_map(its_dev, event);
if (map->db_enabled == enable)
--
2.20.1
Add the SGI configuration entry point for KVM to use.
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v4.c | 13 +++++++++++++
include/linux/irqchip/arm-gic-v4.h | 1 +
2 files changed, 14 insertions(+)
diff --git a/drivers/irqchip/irq-gic-v4.c b/drivers/irqchip/irq-gic-v4.c
index 99b33f60ac63..f3f06c5c7e54 100644
--- a/drivers/irqchip/irq-gic-v4.c
+++ b/drivers/irqchip/irq-gic-v4.c
@@ -320,6 +320,19 @@ int its_prop_update_vlpi(int irq, u8 config, bool inv)
return irq_set_vcpu_affinity(irq, &info);
}
+int its_prop_update_vsgi(int irq, u8 priority, bool group)
+{
+ struct its_cmd_info info = {
+ .cmd_type = PROP_UPDATE_SGI,
+ {
+ .priority = priority,
+ .group = group,
+ },
+ };
+
+ return irq_set_vcpu_affinity(irq, &info);
+}
+
int its_init_v4(struct irq_domain *domain,
const struct irq_domain_ops *vpe_ops,
const struct irq_domain_ops *sgi_ops)
diff --git a/include/linux/irqchip/arm-gic-v4.h b/include/linux/irqchip/arm-gic-v4.h
index 9fbd0418f569..46c167a6349f 100644
--- a/include/linux/irqchip/arm-gic-v4.h
+++ b/include/linux/irqchip/arm-gic-v4.h
@@ -129,6 +129,7 @@ int its_map_vlpi(int irq, struct its_vlpi_map *map);
int its_get_vlpi(int irq, struct its_vlpi_map *map);
int its_unmap_vlpi(int irq);
int its_prop_update_vlpi(int irq, u8 config, bool inv);
+int its_prop_update_vsgi(int irq, u8 priority, bool group);
struct irq_domain_ops;
int its_init_v4(struct irq_domain *domain,
--
2.20.1
Each time a Group-enable bit gets flipped, the state of these bits
needs to be forwarded to the hardware. This is a pretty heavy
handed operation, requiring all vcpus to reload their GICv4
configuration. It is thus implemented as a new request type.
Of course, we only support Group-1 for now...
Signed-off-by: Marc Zyngier <[email protected]>
---
arch/arm64/include/asm/kvm_host.h | 1 +
virt/kvm/arm/arm.c | 8 ++++++++
virt/kvm/arm/vgic/vgic-mmio-v3.c | 5 ++++-
3 files changed, 13 insertions(+), 1 deletion(-)
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index c61260cf63c5..4aeff0cc4ecf 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -45,6 +45,7 @@
#define KVM_REQ_IRQ_PENDING KVM_ARCH_REQ(1)
#define KVM_REQ_VCPU_RESET KVM_ARCH_REQ(2)
#define KVM_REQ_RECORD_STEAL KVM_ARCH_REQ(3)
+#define KVM_REQ_RELOAD_GICv4 KVM_ARCH_REQ(4)
DECLARE_STATIC_KEY_FALSE(userspace_irqchip_in_use);
diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c
index 12e0280291ce..a398dd449880 100644
--- a/virt/kvm/arm/arm.c
+++ b/virt/kvm/arm/arm.c
@@ -682,6 +682,14 @@ static void check_vcpu_requests(struct kvm_vcpu *vcpu)
if (kvm_check_request(KVM_REQ_RECORD_STEAL, vcpu))
kvm_update_stolen_time(vcpu);
+
+ if (kvm_check_request(KVM_REQ_RELOAD_GICv4, vcpu)) {
+ /* The distributor enable bits were changed */
+ preempt_disable();
+ vgic_v4_put(vcpu, false);
+ vgic_v4_load(vcpu);
+ preempt_enable();
+ }
}
}
diff --git a/virt/kvm/arm/vgic/vgic-mmio-v3.c b/virt/kvm/arm/vgic/vgic-mmio-v3.c
index 31ff8c7e09b0..4e72fc08a973 100644
--- a/virt/kvm/arm/vgic/vgic-mmio-v3.c
+++ b/virt/kvm/arm/vgic/vgic-mmio-v3.c
@@ -132,7 +132,10 @@ static void vgic_mmio_write_v3_misc(struct kvm_vcpu *vcpu,
if (is_hwsgi != dist->nassgireq)
vgic_v4_configure_vsgis(vcpu->kvm);
- if (!was_enabled && dist->enabled)
+ if (kvm_vgic_global_state.has_gicv4_1 &&
+ was_enabled != dist->enabled)
+ kvm_make_all_cpus_request(vcpu->kvm, KVM_REQ_RELOAD_GICv4);
+ else if (!was_enabled && dist->enabled)
vgic_kick_vcpus(vcpu->kvm);
mutex_unlock(&vcpu->kvm->lock);
--
2.20.1
The vgic-state debugfs file could do with showing the pending state
of the HW-backed SGIs. Plug it into the low-level code.
Signed-off-by: Marc Zyngier <[email protected]>
---
virt/kvm/arm/vgic/vgic-debug.c | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/virt/kvm/arm/vgic/vgic-debug.c b/virt/kvm/arm/vgic/vgic-debug.c
index cc12fe9b2df3..b13a9e3f99dd 100644
--- a/virt/kvm/arm/vgic/vgic-debug.c
+++ b/virt/kvm/arm/vgic/vgic-debug.c
@@ -178,6 +178,8 @@ static void print_irq_state(struct seq_file *s, struct vgic_irq *irq,
struct kvm_vcpu *vcpu)
{
char *type;
+ bool pending;
+
if (irq->intid < VGIC_NR_SGIS)
type = "SGI";
else if (irq->intid < VGIC_NR_PRIVATE_IRQS)
@@ -190,6 +192,16 @@ static void print_irq_state(struct seq_file *s, struct vgic_irq *irq,
if (irq->intid ==0 || irq->intid == VGIC_NR_PRIVATE_IRQS)
print_header(s, irq, vcpu);
+ pending = irq->pending_latch;
+ if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
+ int err;
+
+ err = irq_get_irqchip_state(irq->host_irq,
+ IRQCHIP_STATE_PENDING,
+ &pending);
+ WARN_ON_ONCE(err);
+ }
+
seq_printf(s, " %s %4d "
" %2d "
"%d%d%d%d%d%d%d "
@@ -201,7 +213,7 @@ static void print_irq_state(struct seq_file *s, struct vgic_irq *irq,
"\n",
type, irq->intid,
(irq->target_vcpu) ? irq->target_vcpu->vcpu_id : -1,
- irq->pending_latch,
+ pending,
irq->line_level,
irq->active,
irq->enabled,
--
2.20.1
masking/unmasking doorbells on GICv4.1 relies on a new INVDB command,
which broadcasts the invalidation to all RDs.
Implement the new command as well as the masking callbacks, and plug
the whole thing into the v4.1 VPE irqchip.
Reviewed-by: Zenghui Yu <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3-its.c | 73 ++++++++++++++++++++++++++++++
include/linux/irqchip/arm-gic-v3.h | 3 +-
2 files changed, 75 insertions(+), 1 deletion(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 157f51398850..0da6baa48153 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -334,6 +334,10 @@ struct its_cmd_desc {
u16 seq_num;
u16 its_list;
} its_vmovp_cmd;
+
+ struct {
+ struct its_vpe *vpe;
+ } its_invdb_cmd;
};
};
@@ -832,6 +836,21 @@ static struct its_vpe *its_build_vclear_cmd(struct its_node *its,
return valid_vpe(its, map->vpe);
}
+static struct its_vpe *its_build_invdb_cmd(struct its_node *its,
+ struct its_cmd_block *cmd,
+ struct its_cmd_desc *desc)
+{
+ if (WARN_ON(!is_v4_1(its)))
+ return NULL;
+
+ its_encode_cmd(cmd, GITS_CMD_INVDB);
+ its_encode_vpeid(cmd, desc->its_invdb_cmd.vpe->vpe_id);
+
+ its_fixup_cmd(cmd);
+
+ return valid_vpe(its, desc->its_invdb_cmd.vpe);
+}
+
static u64 its_cmd_ptr_to_offset(struct its_node *its,
struct its_cmd_block *ptr)
{
@@ -1240,6 +1259,14 @@ static void its_send_vclear(struct its_device *dev, u32 event_id)
its_send_single_vcommand(dev->its, its_build_vclear_cmd, &desc);
}
+static void its_send_invdb(struct its_node *its, struct its_vpe *vpe)
+{
+ struct its_cmd_desc desc;
+
+ desc.its_invdb_cmd.vpe = vpe;
+ its_send_single_vcommand(its, its_build_invdb_cmd, &desc);
+}
+
/*
* irqchip functions - assumes MSI, mostly.
*/
@@ -3554,6 +3581,50 @@ static struct irq_chip its_vpe_irq_chip = {
.irq_set_vcpu_affinity = its_vpe_set_vcpu_affinity,
};
+static struct its_node *find_4_1_its(void)
+{
+ static struct its_node *its = NULL;
+
+ if (!its) {
+ list_for_each_entry(its, &its_nodes, entry) {
+ if (is_v4_1(its))
+ return its;
+ }
+
+ /* Oops? */
+ its = NULL;
+ }
+
+ return its;
+}
+
+static void its_vpe_4_1_send_inv(struct irq_data *d)
+{
+ struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
+ struct its_node *its;
+
+ /*
+ * GICv4.1 wants doorbells to be invalidated using the
+ * INVDB command in order to be broadcast to all RDs. Send
+ * it to the first valid ITS, and let the HW do its magic.
+ */
+ its = find_4_1_its();
+ if (its)
+ its_send_invdb(its, vpe);
+}
+
+static void its_vpe_4_1_mask_irq(struct irq_data *d)
+{
+ lpi_write_config(d->parent_data, LPI_PROP_ENABLED, 0);
+ its_vpe_4_1_send_inv(d);
+}
+
+static void its_vpe_4_1_unmask_irq(struct irq_data *d)
+{
+ lpi_write_config(d->parent_data, 0, LPI_PROP_ENABLED);
+ its_vpe_4_1_send_inv(d);
+}
+
static int its_vpe_4_1_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
{
struct its_cmd_info *info = vcpu_info;
@@ -3575,6 +3646,8 @@ static int its_vpe_4_1_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
static struct irq_chip its_vpe_4_1_irq_chip = {
.name = "GICv4.1-vpe",
+ .irq_mask = its_vpe_4_1_mask_irq,
+ .irq_unmask = its_vpe_4_1_unmask_irq,
.irq_eoi = irq_chip_eoi_parent,
.irq_set_affinity = its_vpe_set_affinity,
.irq_set_vcpu_affinity = its_vpe_4_1_set_vcpu_affinity,
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index df3b7cb50956..33519b7c96cf 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -486,8 +486,9 @@
#define GITS_CMD_VMAPTI GITS_CMD_GICv4(GITS_CMD_MAPTI)
#define GITS_CMD_VMOVI GITS_CMD_GICv4(GITS_CMD_MOVI)
#define GITS_CMD_VSYNC GITS_CMD_GICv4(GITS_CMD_SYNC)
-/* VMOVP is the odd one, as it doesn't have a physical counterpart */
+/* VMOVP and INVDB are the odd ones, as they dont have a physical counterpart */
#define GITS_CMD_VMOVP GITS_CMD_GICv4(2)
+#define GITS_CMD_INVDB GITS_CMD_GICv4(0xe)
/*
* ITS error numbers
--
2.20.1
Hi Marc,
On 2019/12/24 19:10, Marc Zyngier wrote:
> If running under control of a hypervisor that implements GICv4.1
> SGIs, allow the hypervisor to use them at the expense of loosing
> the Active state (which we don't care about for SGIs).
>
> This is trivially done by checking for GICD_TYPER2.nASSGIcap, and
> setting GICD_CTLR.nASSGIreq when enabling Group-1 interrupts.
>
> Signed-off-by: Marc Zyngier <[email protected]>
> ---
> drivers/irqchip/irq-gic-v3.c | 10 ++++++++--
> include/linux/irqchip/arm-gic-v3.h | 2 ++
> 2 files changed, 10 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
> index 640d4db65b78..624f351c0362 100644
> --- a/drivers/irqchip/irq-gic-v3.c
> +++ b/drivers/irqchip/irq-gic-v3.c
> @@ -724,6 +724,7 @@ static void __init gic_dist_init(void)
> unsigned int i;
> u64 affinity;
> void __iomem *base = gic_data.dist_base;
> + u32 val;
>
> /* Disable the distributor */
> writel_relaxed(0, base + GICD_CTLR);
> @@ -756,9 +757,14 @@ static void __init gic_dist_init(void)
> /* Now do the common stuff, and wait for the distributor to drain */
> gic_dist_config(base, GIC_LINE_NR, gic_dist_wait_for_rwp);
>
> + val = GICD_CTLR_ARE_NS | GICD_CTLR_ENABLE_G1A | GICD_CTLR_ENABLE_G1;
> + if (gic_data.rdists.gicd_typer2 & GICD_TYPER2_nASSGIcap) {
> + pr_info("Enabling SGIs without active state\n");
> + val |= GICD_CTLR_nASSGIreq;
> + }
> +
> /* Enable distributor with ARE, Group1 */
> - writel_relaxed(GICD_CTLR_ARE_NS | GICD_CTLR_ENABLE_G1A | GICD_CTLR_ENABLE_G1,
> - base + GICD_CTLR);
> + writel_relaxed(val, base + GICD_CTLR);
>
> /*
> * Set all global interrupts to the boot CPU only. ARE must be
> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> index 9dfe64189d99..72b69f4e6c7b 100644
> --- a/include/linux/irqchip/arm-gic-v3.h
> +++ b/include/linux/irqchip/arm-gic-v3.h
> +#define GICD_CTLR_nASSGIreq (1U << 8)
> +#define GICD_TYPER2_nASSGIcap (1U << 8)
I thought these two bits are newly added in the specification, which is
not available yet... until I've reached patch 29 and 30.
So they are actually some "kvm-implemented" bits and can only be used by
the KVM guests. I have two questions now:
1) As per the latest GIC specification, these two bits are Reserved
(RAZ/WI) from host's perspective, which is good for now. But will
they be (unexpectedly) used one day by the future architecture?
2) Only Linux guest will check and make use of these bits now. What if
some non-Linux guests wants to run with KVM and use the GICv4.1 based
vSGIs? Their developers might have no interests reading at the KVM
code... so what about plumbing some descriptions about these bits
into somewhere in the documentation (or code)? Like below, mostly
copied from your commit messages:
---8<---
// Roughly writing, for the ease of reviewing the later patches.
No-Active-State SGIs (?) Related Field Descriptions
--From Guest's Perspective
With GICv4.1, KVM exposes two single bit (in GICD_TYPER2 and GICD_CTLR
respectively) for guests, which can be used to probe the GICv4.1 based
SGIs support on hypervisor and choose whether guests want the good old
SGIs with an active state, or the new, HW-based ones that do not have
one.
GICD_TYPER2.nASSGIcap, bit [8]
Indicates whether guests are running under control of a hypervisor
that implements GICv4.1 SGIs, allow the hypervisor to use them at
the expense of loosing the Active state.
0b0 GICv4.1 SGIs capability is not offered by hypervisor.
0b1 GICv4.1 SGIs capability is offered by hypervisor.
This field is RO.
GICD_CTLR.nASSGIreq, bit [8]
Indicates whether guests wants to use HW-based SGIs without Active
state if the GICv4.1 SGI capability is offered by hypervisor.
Hypervisor will then try to satisfy the request and switch between
HW/SW delivered SGIs.
0b0 If read, indicated that guest is using the old SW-emulated
SGIs.
If write, indicates that guest requests to use the old
SW-emulated SGIs.
0b1 If read, indicates that guest is using the new HW-based SGIs.
If write, indicates that guest requests to use the new
HW-based SGIs. If GICD_TYPER2.nASSGIcap is 0, the write has
no effect.
Changing this bit is UNPREDICTABLE if the Distributor is enabled.
KVM may just treat this bit as RO when Distributor stays enabled.
See gic_dist_init() in drivers/irqchip/irq-gic-v3.c for an example that
how Linux guest (since 5.6?) makes use of these bits and benefits from
the GICv4.1 based vSGIs.
These two bits are Reserved (RAZ/WI) from host's perspective, which is
good for now.
---8<---
Thanks,
Zenghui
Hi Marc,
On 2019/12/24 19:10, Marc Zyngier wrote:
> Most of the GICv3 emulation code that deals with SGIs now has to be
> aware of the v4.1 capabilities in order to benefit from it.
>
> Add such support, keyed on the interrupt having the hw flag set and
> being a SGI.
>
> Signed-off-by: Marc Zyngier <[email protected]>
> ---
> diff --git a/virt/kvm/arm/vgic/vgic-mmio.c b/virt/kvm/arm/vgic/vgic-mmio.c
> index 0d090482720d..6ebf747a7806 100644
> --- a/virt/kvm/arm/vgic/vgic-mmio.c
> +++ b/virt/kvm/arm/vgic/vgic-mmio.c
> @@ -290,6 +345,20 @@ void vgic_mmio_write_cpending(struct kvm_vcpu *vcpu,
>
> raw_spin_lock_irqsave(&irq->irq_lock, flags);
>
> + if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
> + /* HW SGI? Ask the GIC to inject it */
Shouldn't this be "Ask the GIC to clear its pending state"?
Otherwise looks good!
Thanks,
Zenghui
> + int err;
> + err = irq_set_irqchip_state(irq->host_irq,
> + IRQCHIP_STATE_PENDING,
> + false);
> + WARN_RATELIMIT(err, "IRQ %d", irq->host_irq);
> +
> + raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
> + vgic_put_irq(vcpu->kvm, irq);
> +
> + continue;
> + }
> +
> if (irq->hw)
> vgic_hw_irq_cpending(vcpu, irq, is_uaccess);
> else
On Sat, 28 Dec 2019 17:19:36 +0800
Zenghui Yu <[email protected]> wrote:
> Hi Marc,
>
> On 2019/12/24 19:10, Marc Zyngier wrote:
> > Most of the GICv3 emulation code that deals with SGIs now has to be
> > aware of the v4.1 capabilities in order to benefit from it.
> >
> > Add such support, keyed on the interrupt having the hw flag set and
> > being a SGI.
> >
> > Signed-off-by: Marc Zyngier <[email protected]>
> > ---
>
> > diff --git a/virt/kvm/arm/vgic/vgic-mmio.c b/virt/kvm/arm/vgic/vgic-mmio.c
> > index 0d090482720d..6ebf747a7806 100644
> > --- a/virt/kvm/arm/vgic/vgic-mmio.c
> > +++ b/virt/kvm/arm/vgic/vgic-mmio.c
> > @@ -290,6 +345,20 @@ void vgic_mmio_write_cpending(struct kvm_vcpu *vcpu,
> > > raw_spin_lock_irqsave(&irq->irq_lock, flags);
> > > + if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
> > + /* HW SGI? Ask the GIC to inject it */
>
> Shouldn't this be "Ask the GIC to clear its pending state"?
yeah, a silly copy-paste mistake, thanks for spotting this!
> Otherwise looks good!
Thanks for taking the time to review this work!
M.
--
Jazz is not dead. It just smells funny...
On Sat, 28 Dec 2019 16:56:54 +0800
Zenghui Yu <[email protected]> wrote:
Hi Zenghui,
> Hi Marc,
>
> On 2019/12/24 19:10, Marc Zyngier wrote:
> > If running under control of a hypervisor that implements GICv4.1
> > SGIs, allow the hypervisor to use them at the expense of loosing
> > the Active state (which we don't care about for SGIs).
> >
> > This is trivially done by checking for GICD_TYPER2.nASSGIcap, and
> > setting GICD_CTLR.nASSGIreq when enabling Group-1 interrupts.
> >
> > Signed-off-by: Marc Zyngier <[email protected]>
> > ---
> > drivers/irqchip/irq-gic-v3.c | 10 ++++++++--
> > include/linux/irqchip/arm-gic-v3.h | 2 ++
> > 2 files changed, 10 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
> > index 640d4db65b78..624f351c0362 100644
> > --- a/drivers/irqchip/irq-gic-v3.c
> > +++ b/drivers/irqchip/irq-gic-v3.c
> > @@ -724,6 +724,7 @@ static void __init gic_dist_init(void)
> > unsigned int i;
> > u64 affinity;
> > void __iomem *base = gic_data.dist_base;
> > + u32 val;
> > > /* Disable the distributor */
> > writel_relaxed(0, base + GICD_CTLR);
> > @@ -756,9 +757,14 @@ static void __init gic_dist_init(void)
> > /* Now do the common stuff, and wait for the distributor to drain */
> > gic_dist_config(base, GIC_LINE_NR, gic_dist_wait_for_rwp);
> > > + val = GICD_CTLR_ARE_NS | GICD_CTLR_ENABLE_G1A | GICD_CTLR_ENABLE_G1;
> > + if (gic_data.rdists.gicd_typer2 & GICD_TYPER2_nASSGIcap) {
> > + pr_info("Enabling SGIs without active state\n");
> > + val |= GICD_CTLR_nASSGIreq;
> > + }
> > +
> > /* Enable distributor with ARE, Group1 */
> > - writel_relaxed(GICD_CTLR_ARE_NS | GICD_CTLR_ENABLE_G1A | GICD_CTLR_ENABLE_G1,
> > - base + GICD_CTLR);
> > + writel_relaxed(val, base + GICD_CTLR);
> > > /*
> > * Set all global interrupts to the boot CPU only. ARE must be
> > diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> > index 9dfe64189d99..72b69f4e6c7b 100644
> > --- a/include/linux/irqchip/arm-gic-v3.h
> > +++ b/include/linux/irqchip/arm-gic-v3.h
> > +#define GICD_CTLR_nASSGIreq (1U << 8)
>
> > +#define GICD_TYPER2_nASSGIcap (1U << 8)
>
> I thought these two bits are newly added in the specification, which is
> not available yet... until I've reached patch 29 and 30.
>
> So they are actually some "kvm-implemented" bits and can only be used by
> the KVM guests. I have two questions now:
>
> 1) As per the latest GIC specification, these two bits are Reserved
> (RAZ/WI) from host's perspective, which is good for now. But will
> they be (unexpectedly) used one day by the future architecture?
These bits are now assigned by the latest revision of the GICv4.1 spec,
and are roughly defined as you reversed engineered them below.
The idea is that unmodified SW expects SGIs to have an active state,
because that's the way it has always been so far. Any architecturally
compliant hypervisor must implement an active state for SGIs.
Now, there are obvious benefits in having HW-accelerated SGI delivery,
which for GICv4.1 comes at the expense of loosing the active state.
So we need a way to expose the fact that the hypervisor *can* support
non-active-state SGIs (hence the GICD_TYPER2.nASSGIcap bit), and a way
for the guest to say "I'm OK with losing the active state for SGIs",
which corresponds to the GICD_CTLR.nASSGIreq bit.
Note that this isn't a property that ever needs to be exposed by the
HW, but one that has been architected for the sole benefit of
hypervisors and guests (a HW implementation would keep them as RAZ/WI).
> 2) Only Linux guest will check and make use of these bits now. What if
> some non-Linux guests wants to run with KVM and use the GICv4.1 based
> vSGIs? Their developers might have no interests reading at the KVM
> code... so what about plumbing some descriptions about these bits
> into somewhere in the documentation (or code)? Like below, mostly
> copied from your commit messages:
>
> ---8<---
>
> // Roughly writing, for the ease of reviewing the later patches.
>
> No-Active-State SGIs (?) Related Field Descriptions
> --From Guest's Perspective
>
> With GICv4.1, KVM exposes two single bit (in GICD_TYPER2 and GICD_CTLR
> respectively) for guests, which can be used to probe the GICv4.1 based
> SGIs support on hypervisor and choose whether guests want the good old
> SGIs with an active state, or the new, HW-based ones that do not have
> one.
>
> GICD_TYPER2.nASSGIcap, bit [8]
>
> Indicates whether guests are running under control of a hypervisor
> that implements GICv4.1 SGIs, allow the hypervisor to use them at
> the expense of loosing the Active state.
>
> 0b0 GICv4.1 SGIs capability is not offered by hypervisor.
>
> 0b1 GICv4.1 SGIs capability is offered by hypervisor.
>
> This field is RO.
>
>
> GICD_CTLR.nASSGIreq, bit [8]
>
> Indicates whether guests wants to use HW-based SGIs without Active
> state if the GICv4.1 SGI capability is offered by hypervisor.
> Hypervisor will then try to satisfy the request and switch between
> HW/SW delivered SGIs.
>
> 0b0 If read, indicated that guest is using the old SW-emulated
> SGIs.
> If write, indicates that guest requests to use the old
> SW-emulated SGIs.
>
> 0b1 If read, indicates that guest is using the new HW-based SGIs.
> If write, indicates that guest requests to use the new
> HW-based SGIs. If GICD_TYPER2.nASSGIcap is 0, the write has
> no effect.
>
> Changing this bit is UNPREDICTABLE if the Distributor is enabled.
> KVM may just treat this bit as RO when Distributor stays enabled.
>
>
> See gic_dist_init() in drivers/irqchip/irq-gic-v3.c for an example that
> how Linux guest (since 5.6?) makes use of these bits and benefits from
> the GICv4.1 based vSGIs.
>
> These two bits are Reserved (RAZ/WI) from host's perspective, which is
> good for now.
This write-up is sensible, but it would be a duplication of the
architecture spec once it is publicly available. As much as possible,
I'd like not to duplicate information, as the are bound to diverge as
the architecture spec gets amended. How about we add this to the commit
message instead, as a placeholder until the GICv4.1 doc is available
(hopefully some time next year)?
Thanks,
M.
--
Jazz is not dead. It just smells funny...
Hi Marc,
On 2019/12/28 18:36, Marc Zyngier wrote:
> On Sat, 28 Dec 2019 16:56:54 +0800
> Zenghui Yu <[email protected]> wrote:
>
> Hi Zenghui,
>
>> Hi Marc,
>>
>> On 2019/12/24 19:10, Marc Zyngier wrote:
>>> If running under control of a hypervisor that implements GICv4.1
>>> SGIs, allow the hypervisor to use them at the expense of loosing
>>> the Active state (which we don't care about for SGIs).
>>>
>>> This is trivially done by checking for GICD_TYPER2.nASSGIcap, and
>>> setting GICD_CTLR.nASSGIreq when enabling Group-1 interrupts.
>>>
>>> Signed-off-by: Marc Zyngier <[email protected]>
>>> ---
>>> drivers/irqchip/irq-gic-v3.c | 10 ++++++++--
>>> include/linux/irqchip/arm-gic-v3.h | 2 ++
>>> 2 files changed, 10 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
>>> index 640d4db65b78..624f351c0362 100644
>>> --- a/drivers/irqchip/irq-gic-v3.c
>>> +++ b/drivers/irqchip/irq-gic-v3.c
>>> @@ -724,6 +724,7 @@ static void __init gic_dist_init(void)
>>> unsigned int i;
>>> u64 affinity;
>>> void __iomem *base = gic_data.dist_base;
>>> + u32 val;
>>> > /* Disable the distributor */
>>> writel_relaxed(0, base + GICD_CTLR);
>>> @@ -756,9 +757,14 @@ static void __init gic_dist_init(void)
>>> /* Now do the common stuff, and wait for the distributor to drain */
>>> gic_dist_config(base, GIC_LINE_NR, gic_dist_wait_for_rwp);
>>> > + val = GICD_CTLR_ARE_NS | GICD_CTLR_ENABLE_G1A | GICD_CTLR_ENABLE_G1;
>>> + if (gic_data.rdists.gicd_typer2 & GICD_TYPER2_nASSGIcap) {
>>> + pr_info("Enabling SGIs without active state\n");
>>> + val |= GICD_CTLR_nASSGIreq;
>>> + }
>>> +
>>> /* Enable distributor with ARE, Group1 */
>>> - writel_relaxed(GICD_CTLR_ARE_NS | GICD_CTLR_ENABLE_G1A | GICD_CTLR_ENABLE_G1,
>>> - base + GICD_CTLR);
>>> + writel_relaxed(val, base + GICD_CTLR);
>>> > /*
>>> * Set all global interrupts to the boot CPU only. ARE must be
>>> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
>>> index 9dfe64189d99..72b69f4e6c7b 100644
>>> --- a/include/linux/irqchip/arm-gic-v3.h
>>> +++ b/include/linux/irqchip/arm-gic-v3.h
>>> +#define GICD_CTLR_nASSGIreq (1U << 8)
>>
>>> +#define GICD_TYPER2_nASSGIcap (1U << 8)
>>
>> I thought these two bits are newly added in the specification, which is
>> not available yet... until I've reached patch 29 and 30.
>>
>> So they are actually some "kvm-implemented" bits and can only be used by
>> the KVM guests. I have two questions now:
>>
>> 1) As per the latest GIC specification, these two bits are Reserved
>> (RAZ/WI) from host's perspective, which is good for now. But will
>> they be (unexpectedly) used one day by the future architecture?
>
> These bits are now assigned by the latest revision of the GICv4.1 spec,
> and are roughly defined as you reversed engineered them below.
(Ah, so all what I'm having in the hand is a GICv4.0.5 ;-).
> The idea is that unmodified SW expects SGIs to have an active state,
> because that's the way it has always been so far. Any architecturally
> compliant hypervisor must implement an active state for SGIs.
>
> Now, there are obvious benefits in having HW-accelerated SGI delivery,
> which for GICv4.1 comes at the expense of loosing the active state.
>
> So we need a way to expose the fact that the hypervisor *can* support
> non-active-state SGIs (hence the GICD_TYPER2.nASSGIcap bit), and a way
> for the guest to say "I'm OK with losing the active state for SGIs",
> which corresponds to the GICD_CTLR.nASSGIreq bit.
Thanks for the explanation!
> Note that this isn't a property that ever needs to be exposed by the
> HW, but one that has been architected for the sole benefit of
> hypervisors and guests (a HW implementation would keep them as RAZ/WI).
Yeah, I can understand this point.
>> 2) Only Linux guest will check and make use of these bits now. What if
>> some non-Linux guests wants to run with KVM and use the GICv4.1 based
>> vSGIs? Their developers might have no interests reading at the KVM
>> code... so what about plumbing some descriptions about these bits
>> into somewhere in the documentation (or code)? Like below, mostly
>> copied from your commit messages:
>>
>> ---8<---
>>
>> // Roughly writing, for the ease of reviewing the later patches.
>>
>> No-Active-State SGIs (?) Related Field Descriptions
>> --From Guest's Perspective
>>
>> With GICv4.1, KVM exposes two single bit (in GICD_TYPER2 and GICD_CTLR
>> respectively) for guests, which can be used to probe the GICv4.1 based
>> SGIs support on hypervisor and choose whether guests want the good old
>> SGIs with an active state, or the new, HW-based ones that do not have
>> one.
>>
>> GICD_TYPER2.nASSGIcap, bit [8]
>>
>> Indicates whether guests are running under control of a hypervisor
>> that implements GICv4.1 SGIs, allow the hypervisor to use them at
>> the expense of loosing the Active state.
>>
>> 0b0 GICv4.1 SGIs capability is not offered by hypervisor.
>>
>> 0b1 GICv4.1 SGIs capability is offered by hypervisor.
>>
>> This field is RO.
>>
>>
>> GICD_CTLR.nASSGIreq, bit [8]
>>
>> Indicates whether guests wants to use HW-based SGIs without Active
>> state if the GICv4.1 SGI capability is offered by hypervisor.
>> Hypervisor will then try to satisfy the request and switch between
>> HW/SW delivered SGIs.
>>
>> 0b0 If read, indicated that guest is using the old SW-emulated
>> SGIs.
>> If write, indicates that guest requests to use the old
>> SW-emulated SGIs.
>>
>> 0b1 If read, indicates that guest is using the new HW-based SGIs.
>> If write, indicates that guest requests to use the new
>> HW-based SGIs. If GICD_TYPER2.nASSGIcap is 0, the write has
>> no effect.
>>
>> Changing this bit is UNPREDICTABLE if the Distributor is enabled.
>> KVM may just treat this bit as RO when Distributor stays enabled.
>>
>>
>> See gic_dist_init() in drivers/irqchip/irq-gic-v3.c for an example that
>> how Linux guest (since 5.6?) makes use of these bits and benefits from
>> the GICv4.1 based vSGIs.
>>
>> These two bits are Reserved (RAZ/WI) from host's perspective, which is
>> good for now.
>
> This write-up is sensible, but it would be a duplication of the
> architecture spec once it is publicly available. As much as possible,
> I'd like not to duplicate information, as the are bound to diverge as
> the architecture spec gets amended. How about we add this to the commit
> message instead, as a placeholder until the GICv4.1 doc is available
> (hopefully some time next year)?
Good idea, I'm all for it.
It will (at least) help other reviewers to easily realize what is going
on between guests and the hypervisor.
Thanks,
Zenghui
Hi Marc, [This is from Nianyao]
On 2019/12/24 19:10, Marc Zyngier wrote:
> In order to let a guest buy in the new, active-less SGIs, we
> need to be able to switch between the two modes.
>
> Handle this by stopping all guest activity, transfer the state
> from one mode to the other, and resume the guest.
>
> Signed-off-by: Marc Zyngier <[email protected]>
> ---
> include/kvm/arm_vgic.h | 3 ++
> virt/kvm/arm/vgic/vgic-v3.c | 2 +
> virt/kvm/arm/vgic/vgic-v4.c | 96 +++++++++++++++++++++++++++++++++++++
> virt/kvm/arm/vgic/vgic.h | 1 +
> 4 files changed, 102 insertions(+)
>
> diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
> index 63457908c9c4..69f4164d6477 100644
> --- a/include/kvm/arm_vgic.h
> +++ b/include/kvm/arm_vgic.h
> @@ -231,6 +231,9 @@ struct vgic_dist {
> /* distributor enabled */
> bool enabled;
>
> + /* Wants SGIs without active state */
> + bool nassgireq;
> +
> struct vgic_irq *spis;
>
> struct vgic_io_device dist_iodev;
> diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
> index c2fdea201747..c79a251c4974 100644
> --- a/virt/kvm/arm/vgic/vgic-v3.c
> +++ b/virt/kvm/arm/vgic/vgic-v3.c
> @@ -540,6 +540,8 @@ int vgic_v3_map_resources(struct kvm *kvm)
> goto out;
> }
>
> + if (kvm_vgic_global_state.has_gicv4_1)
> + vgic_v4_configure_vsgis(kvm);
> dist->ready = true;
>
> out:
> diff --git a/virt/kvm/arm/vgic/vgic-v4.c b/virt/kvm/arm/vgic/vgic-v4.c
> index c2fcde104ea2..063785fd2dc7 100644
> --- a/virt/kvm/arm/vgic/vgic-v4.c
> +++ b/virt/kvm/arm/vgic/vgic-v4.c
> @@ -97,6 +97,102 @@ static irqreturn_t vgic_v4_doorbell_handler(int irq, void *info)
> return IRQ_HANDLED;
> }
>
> +static void vgic_v4_sync_sgi_config(struct its_vpe *vpe, struct vgic_irq *irq)
> +{
> + vpe->sgi_config[irq->intid].enabled = irq->enabled;
> + vpe->sgi_config[irq->intid].group = irq->group;
> + vpe->sgi_config[irq->intid].priority = irq->priority;
> +}
> +
> +static void vgic_v4_enable_vsgis(struct kvm_vcpu *vcpu)
> +{
> + struct its_vpe *vpe = &vcpu->arch.vgic_cpu.vgic_v3.its_vpe;
> + int i;
> +
> + /*
> + * With GICv4.1, every virtual SGI can be directly injected. So
> + * let's pretend that they are HW interrupts, tied to a host
> + * IRQ. The SGI code will do its magic.
> + */
> + for (i = 0; i < VGIC_NR_SGIS; i++) {
> + struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, i);
> + struct irq_desc *desc;
> + int ret;
> +
> + if (irq->hw) {
> + vgic_put_irq(vcpu->kvm, irq);
> + continue;
> + }
> +
> + irq->hw = true;
> + irq->host_irq = irq_find_mapping(vpe->sgi_domain, i);
I think we need to check whether irq_find_mapping returns 0.
> + vgic_v4_sync_sgi_config(vpe, irq);
> + /*
> + * SGIs are initialised as disabled. Enable them if
> + * required by the rest of the VGIC init code.
> + */
> + desc = irq_to_desc(irq->host_irq);
> + ret = irq_domain_activate_irq(irq_desc_get_irq_data(desc),
> + false);
If irq->host_irq is not valid , in irq_domain_activate_irq, it will trigger NULL pointer
dereference in host kernel.
I meet a problem here. When hw support GIC4.1, and host kernel is started with
kvm-arm.vgic_v4_enable=0, starting a virtual machine will trigger NULL pointer
dereference in host. The following is error info:
[ 7.913815] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000038
[ 7.913818] Mem abort info:
[ 7.913819] ESR = 0x96000007
[ 7.913821] EC = 0x25: DABT (current EL), IL = 32 bits
[ 7.913823] SET = 0, FnV = 0
[ 7.913825] EA = 0, S1PTW = 0
[ 7.913827] Data abort info:
[ 7.913828] ISV = 0, ISS = 0x00000007
[ 7.913830] CM = 0, WnR = 0
[ 7.913832] user pgtable: 64k pages, 48-bit VAs, pgdp=00000824405a0800
[ 7.913835] [0000000000000038] pgd=0000082444120003, pud=0000082444120003, pmd=0000082444130003, pte=0000000000000000
[ 7.913840] Internal error: Oops: 96000007 [#1] SMP
[ 7.913842] Modules linked in:
[ 7.913845] CPU: 1 PID: 1918 Comm: qemu-system-aar Tainted: G W 5.5.0-rc2-14819-g4e11d8f-dirty #20
[ 7.913849] pstate: 80400009 (Nzcv daif +PAN -UAO)
[ 7.913852] pc : irq_domain_activate_irq+0x0/0x58
[ 7.913854] lr : vgic_v4_configure_vsgis+0x108/0x208
[ 7.913857] sp : ffff8000160af7b0
[ 7.913858] x29: ffff8000160af7b0 x28: ffff082445030000
[ 7.913861] x27: 0000000000000004 x26: ffff0824456032c8
[ 7.913864] x25: 0000000000000000 x24: ffff8000118d1130
[ 7.913868] x23: 0000000000000001 x22: ffff8000118d0000
[ 7.913871] x21: 0000000000000000 x20: ffff082445602360
[ 7.913874] x19: ffff082445603348 x18: 0000000000000010
[ 7.913877] x17: 0000000000000050 x16: 0000000000000001
[ 7.913880] x15: ffff8000110eb258 x14: ffffffffffffffff
[ 7.913883] x13: ffff8000910cfb17 x12: ffff8000110cfb24
[ 7.913886] x11: ffff8000110f8000 x10: 0000000000000040
[ 7.913889] x9 : ffff8000110f8fe0 x8 : ffff8000110f8fd8
[ 7.913892] x7 : ffff082420000270 x6 : 0000000000000000
[ 7.913895] x5 : ffff082420000248 x4 : 0000000000000000
[ 7.913898] x3 : 0000000000000000 x2 : 0000000000000000
[ 7.913901] x1 : 0000000000000000 x0 : 0000000000000028
[ 7.913904] Call trace:
[ 7.913907] irq_domain_activate_irq+0x0/0x58
[ 7.913909] vgic_mmio_write_v3_misc+0xf0/0x100
[ 7.913912] dispatch_mmio_write+0x78/0x100
[ 7.913915] __kvm_io_bus_write+0xbc/0xf8
[ 7.913918] kvm_io_bus_write+0x48/0x80
[ 7.913921] io_mem_abort+0x128/0x288
[ 7.913924] kvm_handle_guest_abort+0x2c0/0xe88
[ 7.913927] handle_exit+0x6c/0x1d8
[ 7.913930] kvm_arch_vcpu_ioctl_run+0x454/0x6e8
[ 7.913932] kvm_vcpu_ioctl+0x310/0x9e0
[ 7.913935] do_vfs_ioctl+0x604/0xbd0
[ 7.913938] ksys_ioctl+0x78/0xa8
[ 7.913940] __arm64_sys_ioctl+0x1c/0x28
[ 7.913943] el0_svc_handler+0x7c/0x188
[ 7.913946] el0_sync_handler+0x138/0x258
[ 7.913948] el0_sync+0x140/0x180
[ 7.913952] Code: a8c87bfd d65f03c0 97fe9008 d503201f (f9400803)
[ 7.913954] ---[ end trace e0d0e4b407d388f3 ]---
Thanks,
> + if (!WARN_ON(ret)) {
> + /* Transfer pending state */
> + ret = irq_set_irqchip_state(irq->host_irq,
> + IRQCHIP_STATE_PENDING,
> + irq->pending_latch);
> + WARN_ON(ret);
> + irq->pending_latch = false;
> + }
> +
> + vgic_put_irq(vcpu->kvm, irq);
> + }
> +}
> +
> +static void vgic_v4_disable_vsgis(struct kvm_vcpu *vcpu)
> +{
> + int i;
> +
> + for (i = 0; i < VGIC_NR_SGIS; i++) {
> + struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, i);
> + struct irq_desc *desc;
> + int ret;
> +
> + if (!irq->hw) {
> + vgic_put_irq(vcpu->kvm, irq);
> + continue;
> + }
> +
> + irq->hw = false;
> + ret = irq_get_irqchip_state(irq->host_irq,
> + IRQCHIP_STATE_PENDING,
> + &irq->pending_latch);
> + WARN_ON(ret);
> +
> + desc = irq_to_desc(irq->host_irq);
> + irq_domain_deactivate_irq(irq_desc_get_irq_data(desc));
> +
> + vgic_put_irq(vcpu->kvm, irq);
> + }
> +}
> +
> +/* Must be called with the kvm lock held */
> +void vgic_v4_configure_vsgis(struct kvm *kvm)
> +{
> + struct vgic_dist *dist = &kvm->arch.vgic;
> + struct kvm_vcpu *vcpu;
> + int i;
> +
> + kvm_arm_halt_guest(kvm);
> +
> + kvm_for_each_vcpu(i, vcpu, kvm) {
> + if (dist->nassgireq)
> + vgic_v4_enable_vsgis(vcpu);
> + else
> + vgic_v4_disable_vsgis(vcpu);
> + }
> +
> + kvm_arm_resume_guest(kvm);
> +}
> +
> /**
> * vgic_v4_init - Initialize the GICv4 data structures
> * @kvm: Pointer to the VM being initialized
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index c7fefd6b1c80..769e4802645e 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -316,5 +316,6 @@ void vgic_its_invalidate_cache(struct kvm *kvm);
> bool vgic_supports_direct_msis(struct kvm *kvm);
> int vgic_v4_init(struct kvm *kvm);
> void vgic_v4_teardown(struct kvm *kvm);
> +void vgic_v4_configure_vsgis(struct kvm *kvm);
>
> #endif
>
Hi,
On 2020/1/15 10:49, Shaokun Zhang wrote:
> Hi Marc, [This is from Nianyao]
>
> On 2019/12/24 19:10, Marc Zyngier wrote:
>> In order to let a guest buy in the new, active-less SGIs, we
>> need to be able to switch between the two modes.
>>
>> Handle this by stopping all guest activity, transfer the state
>> from one mode to the other, and resume the guest.
>>
>> Signed-off-by: Marc Zyngier <[email protected]>
>> ---
[...]
>> diff --git a/virt/kvm/arm/vgic/vgic-v4.c b/virt/kvm/arm/vgic/vgic-v4.c
>> index c2fcde104ea2..063785fd2dc7 100644
>> --- a/virt/kvm/arm/vgic/vgic-v4.c
>> +++ b/virt/kvm/arm/vgic/vgic-v4.c
>> @@ -97,6 +97,102 @@ static irqreturn_t vgic_v4_doorbell_handler(int irq, void *info)
>> return IRQ_HANDLED;
>> }
>>
>> +static void vgic_v4_sync_sgi_config(struct its_vpe *vpe, struct vgic_irq *irq)
>> +{
>> + vpe->sgi_config[irq->intid].enabled = irq->enabled;
>> + vpe->sgi_config[irq->intid].group = irq->group;
>> + vpe->sgi_config[irq->intid].priority = irq->priority;
>> +}
>> +
>> +static void vgic_v4_enable_vsgis(struct kvm_vcpu *vcpu)
>> +{
>> + struct its_vpe *vpe = &vcpu->arch.vgic_cpu.vgic_v3.its_vpe;
>> + int i;
>> +
>> + /*
>> + * With GICv4.1, every virtual SGI can be directly injected. So
>> + * let's pretend that they are HW interrupts, tied to a host
>> + * IRQ. The SGI code will do its magic.
>> + */
>> + for (i = 0; i < VGIC_NR_SGIS; i++) {
>> + struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, i);
>> + struct irq_desc *desc;
>> + int ret;
>> +
>> + if (irq->hw) {
>> + vgic_put_irq(vcpu->kvm, irq);
>> + continue;
>> + }
>> +
>> + irq->hw = true;
>> + irq->host_irq = irq_find_mapping(vpe->sgi_domain, i);
>
> I think we need to check whether irq_find_mapping returns 0.
>
>> + vgic_v4_sync_sgi_config(vpe, irq);
>> + /*
>> + * SGIs are initialised as disabled. Enable them if
>> + * required by the rest of the VGIC init code.
>> + */
>> + desc = irq_to_desc(irq->host_irq);
>> + ret = irq_domain_activate_irq(irq_desc_get_irq_data(desc),
>> + false);
>
> If irq->host_irq is not valid , in irq_domain_activate_irq, it will trigger NULL pointer
> dereference in host kernel.
> I meet a problem here. When hw support GIC4.1, and host kernel is started with
> kvm-arm.vgic_v4_enable=0, starting a virtual machine will trigger NULL pointer
> dereference in host.
I think the thing is that we should _not_ try to configure vSGIs at all
if kvm-arm.vgic_v4_enable=0 (which indicates we don't allow use of the
GICv4 of direct injection).
We currently set kvm_vgic_global_state.has_gicv4_1 to true if HW support
GICv4.1, regardless whatever the gicv4_enable is (see patch#23 -
vgic_v3_probe). I think this is what actually needs fixing.
Thanks,
Zenghui
On 2020-01-15 02:49, Shaokun Zhang wrote:
> Hi Marc, [This is from Nianyao]
>
> On 2019/12/24 19:10, Marc Zyngier wrote:
>> In order to let a guest buy in the new, active-less SGIs, we
>> need to be able to switch between the two modes.
>>
>> Handle this by stopping all guest activity, transfer the state
>> from one mode to the other, and resume the guest.
>>
>> Signed-off-by: Marc Zyngier <[email protected]>
>> ---
>> include/kvm/arm_vgic.h | 3 ++
>> virt/kvm/arm/vgic/vgic-v3.c | 2 +
>> virt/kvm/arm/vgic/vgic-v4.c | 96
>> +++++++++++++++++++++++++++++++++++++
>> virt/kvm/arm/vgic/vgic.h | 1 +
>> 4 files changed, 102 insertions(+)
>>
>> diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
>> index 63457908c9c4..69f4164d6477 100644
>> --- a/include/kvm/arm_vgic.h
>> +++ b/include/kvm/arm_vgic.h
>> @@ -231,6 +231,9 @@ struct vgic_dist {
>> /* distributor enabled */
>> bool enabled;
>>
>> + /* Wants SGIs without active state */
>> + bool nassgireq;
>> +
>> struct vgic_irq *spis;
>>
>> struct vgic_io_device dist_iodev;
>> diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
>> index c2fdea201747..c79a251c4974 100644
>> --- a/virt/kvm/arm/vgic/vgic-v3.c
>> +++ b/virt/kvm/arm/vgic/vgic-v3.c
>> @@ -540,6 +540,8 @@ int vgic_v3_map_resources(struct kvm *kvm)
>> goto out;
>> }
>>
>> + if (kvm_vgic_global_state.has_gicv4_1)
>> + vgic_v4_configure_vsgis(kvm);
>> dist->ready = true;
>>
>> out:
>> diff --git a/virt/kvm/arm/vgic/vgic-v4.c b/virt/kvm/arm/vgic/vgic-v4.c
>> index c2fcde104ea2..063785fd2dc7 100644
>> --- a/virt/kvm/arm/vgic/vgic-v4.c
>> +++ b/virt/kvm/arm/vgic/vgic-v4.c
>> @@ -97,6 +97,102 @@ static irqreturn_t vgic_v4_doorbell_handler(int
>> irq, void *info)
>> return IRQ_HANDLED;
>> }
>>
>> +static void vgic_v4_sync_sgi_config(struct its_vpe *vpe, struct
>> vgic_irq *irq)
>> +{
>> + vpe->sgi_config[irq->intid].enabled = irq->enabled;
>> + vpe->sgi_config[irq->intid].group = irq->group;
>> + vpe->sgi_config[irq->intid].priority = irq->priority;
>> +}
>> +
>> +static void vgic_v4_enable_vsgis(struct kvm_vcpu *vcpu)
>> +{
>> + struct its_vpe *vpe = &vcpu->arch.vgic_cpu.vgic_v3.its_vpe;
>> + int i;
>> +
>> + /*
>> + * With GICv4.1, every virtual SGI can be directly injected. So
>> + * let's pretend that they are HW interrupts, tied to a host
>> + * IRQ. The SGI code will do its magic.
>> + */
>> + for (i = 0; i < VGIC_NR_SGIS; i++) {
>> + struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, i);
>> + struct irq_desc *desc;
>> + int ret;
>> +
>> + if (irq->hw) {
>> + vgic_put_irq(vcpu->kvm, irq);
>> + continue;
>> + }
>> +
>> + irq->hw = true;
>> + irq->host_irq = irq_find_mapping(vpe->sgi_domain, i);
>
> I think we need to check whether irq_find_mapping returns 0.
Why? its_alloc_vcpu_sgis() performs the allocation of all 16 SGIs
in one go. If that fails, we should error out. Conversely, we're
guaranteed that we have a mapping if it hasn't failed.
>
>> + vgic_v4_sync_sgi_config(vpe, irq);
>> + /*
>> + * SGIs are initialised as disabled. Enable them if
>> + * required by the rest of the VGIC init code.
>> + */
>> + desc = irq_to_desc(irq->host_irq);
>> + ret = irq_domain_activate_irq(irq_desc_get_irq_data(desc),
>> + false);
>
> If irq->host_irq is not valid , in irq_domain_activate_irq, it will
> trigger NULL pointer dereference in host kernel.
> I meet a problem here. When hw support GIC4.1, and host kernel is
> started with
> kvm-arm.vgic_v4_enable=0, starting a virtual machine will trigger NULL
> pointer
> dereference in host. The following is error info:
[...]
But the problem here is that we're trying to use GICv4.1 even if GICv4
is disabled. I don't think there is a point in trying to sidestep this
locally if we're doing the wrong thing at top-level, which seems to be
the case.
Thanks,
M.
--
Jazz is not dead. It just smells funny...
On 2020-01-15 03:49, Zenghui Yu wrote:
> Hi,
>
> On 2020/1/15 10:49, Shaokun Zhang wrote:
>> Hi Marc, [This is from Nianyao]
>>
>> On 2019/12/24 19:10, Marc Zyngier wrote:
>>> In order to let a guest buy in the new, active-less SGIs, we
>>> need to be able to switch between the two modes.
>>>
>>> Handle this by stopping all guest activity, transfer the state
>>> from one mode to the other, and resume the guest.
>>>
>>> Signed-off-by: Marc Zyngier <[email protected]>
>>> ---
>
> [...]
>
>>> diff --git a/virt/kvm/arm/vgic/vgic-v4.c
>>> b/virt/kvm/arm/vgic/vgic-v4.c
>>> index c2fcde104ea2..063785fd2dc7 100644
>>> --- a/virt/kvm/arm/vgic/vgic-v4.c
>>> +++ b/virt/kvm/arm/vgic/vgic-v4.c
>>> @@ -97,6 +97,102 @@ static irqreturn_t vgic_v4_doorbell_handler(int
>>> irq, void *info)
>>> return IRQ_HANDLED;
>>> }
>>> +static void vgic_v4_sync_sgi_config(struct its_vpe *vpe, struct
>>> vgic_irq *irq)
>>> +{
>>> + vpe->sgi_config[irq->intid].enabled = irq->enabled;
>>> + vpe->sgi_config[irq->intid].group = irq->group;
>>> + vpe->sgi_config[irq->intid].priority = irq->priority;
>>> +}
>>> +
>>> +static void vgic_v4_enable_vsgis(struct kvm_vcpu *vcpu)
>>> +{
>>> + struct its_vpe *vpe = &vcpu->arch.vgic_cpu.vgic_v3.its_vpe;
>>> + int i;
>>> +
>>> + /*
>>> + * With GICv4.1, every virtual SGI can be directly injected. So
>>> + * let's pretend that they are HW interrupts, tied to a host
>>> + * IRQ. The SGI code will do its magic.
>>> + */
>>> + for (i = 0; i < VGIC_NR_SGIS; i++) {
>>> + struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, i);
>>> + struct irq_desc *desc;
>>> + int ret;
>>> +
>>> + if (irq->hw) {
>>> + vgic_put_irq(vcpu->kvm, irq);
>>> + continue;
>>> + }
>>> +
>>> + irq->hw = true;
>>> + irq->host_irq = irq_find_mapping(vpe->sgi_domain, i);
>>
>> I think we need to check whether irq_find_mapping returns 0.
>>
>>> + vgic_v4_sync_sgi_config(vpe, irq);
>>> + /*
>>> + * SGIs are initialised as disabled. Enable them if
>>> + * required by the rest of the VGIC init code.
>>> + */
>>> + desc = irq_to_desc(irq->host_irq);
>>> + ret = irq_domain_activate_irq(irq_desc_get_irq_data(desc),
>>> + false);
>>
>> If irq->host_irq is not valid , in irq_domain_activate_irq, it will
>> trigger NULL pointer
>> dereference in host kernel.
>> I meet a problem here. When hw support GIC4.1, and host kernel is
>> started with
>> kvm-arm.vgic_v4_enable=0, starting a virtual machine will trigger NULL
>> pointer
>> dereference in host.
>
> I think the thing is that we should _not_ try to configure vSGIs at all
> if kvm-arm.vgic_v4_enable=0 (which indicates we don't allow use of the
> GICv4 of direct injection).
>
> We currently set kvm_vgic_global_state.has_gicv4_1 to true if HW
> support
> GICv4.1, regardless whatever the gicv4_enable is (see patch#23 -
> vgic_v3_probe). I think this is what actually needs fixing.
Yes, my point exactly. I've pushed out a potential fix [1], and I'd be
grateful if you could let me know whether that fixes it for you.
Thanks,
M.
[1]
https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/commit/?h=irq/gic-v4.1-devel&id=b82c2ee1d3fef66fb85793965c344260f618219d
--
Jazz is not dead. It just smells funny...
Hi Marc,
On 2020/1/15 21:32, Marc Zyngier wrote:
> On 2020-01-15 03:49, Zenghui Yu wrote:
>> Hi,
>>
>> On 2020/1/15 10:49, Shaokun Zhang wrote:
>>> Hi Marc, [This is from Nianyao]
>>>
>>> On 2019/12/24 19:10, Marc Zyngier wrote:
>>>> In order to let a guest buy in the new, active-less SGIs, we
>>>> need to be able to switch between the two modes.
>>>>
>>>> Handle this by stopping all guest activity, transfer the state
>>>> from one mode to the other, and resume the guest.
>>>>
>>>> Signed-off-by: Marc Zyngier <[email protected]>
>>>> ---
>>
>> [...]
>>
>>>> diff --git a/virt/kvm/arm/vgic/vgic-v4.c b/virt/kvm/arm/vgic/vgic-v4.c
>>>> index c2fcde104ea2..063785fd2dc7 100644
>>>> --- a/virt/kvm/arm/vgic/vgic-v4.c
>>>> +++ b/virt/kvm/arm/vgic/vgic-v4.c
>>>> @@ -97,6 +97,102 @@ static irqreturn_t vgic_v4_doorbell_handler(int
>>>> irq, void *info)
>>>> Â Â Â Â Â return IRQ_HANDLED;
>>>> Â }
>>>> Â +static void vgic_v4_sync_sgi_config(struct its_vpe *vpe, struct
>>>> vgic_irq *irq)
>>>> +{
>>>> +   vpe->sgi_config[irq->intid].enabled   = irq->enabled;
>>>> +   vpe->sgi_config[irq->intid].group    = irq->group;
>>>> +   vpe->sgi_config[irq->intid].priority   = irq->priority;
>>>> +}
>>>> +
>>>> +static void vgic_v4_enable_vsgis(struct kvm_vcpu *vcpu)
>>>> +{
>>>> +Â Â Â struct its_vpe *vpe = &vcpu->arch.vgic_cpu.vgic_v3.its_vpe;
>>>> +Â Â Â int i;
>>>> +
>>>> +Â Â Â /*
>>>> +Â Â Â Â * With GICv4.1, every virtual SGI can be directly injected. So
>>>> +Â Â Â Â * let's pretend that they are HW interrupts, tied to a host
>>>> +Â Â Â Â * IRQ. The SGI code will do its magic.
>>>> +Â Â Â Â */
>>>> +Â Â Â for (i = 0; i < VGIC_NR_SGIS; i++) {
>>>> +Â Â Â Â Â Â Â struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, i);
>>>> +Â Â Â Â Â Â Â struct irq_desc *desc;
>>>> +Â Â Â Â Â Â Â int ret;
>>>> +
>>>> +Â Â Â Â Â Â Â if (irq->hw) {
>>>> +Â Â Â Â Â Â Â Â Â Â Â vgic_put_irq(vcpu->kvm, irq);
>>>> +Â Â Â Â Â Â Â Â Â Â Â continue;
>>>> +Â Â Â Â Â Â Â }
>>>> +
>>>> +Â Â Â Â Â Â Â irq->hw = true;
>>>> +Â Â Â Â Â Â Â irq->host_irq = irq_find_mapping(vpe->sgi_domain, i);
>>>
>>> I think we need to check whether irq_find_mapping returns 0.
>>>
>>>> +Â Â Â Â Â Â Â vgic_v4_sync_sgi_config(vpe, irq);
>>>> +Â Â Â Â Â Â Â /*
>>>> +Â Â Â Â Â Â Â Â * SGIs are initialised as disabled. Enable them if
>>>> +Â Â Â Â Â Â Â Â * required by the rest of the VGIC init code.
>>>> +Â Â Â Â Â Â Â Â */
>>>> +Â Â Â Â Â Â Â desc = irq_to_desc(irq->host_irq);
>>>> +Â Â Â Â Â Â Â ret = irq_domain_activate_irq(irq_desc_get_irq_data(desc),
>>>> +Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â false);
>>>
>>> If irq->host_irq is not valid , in irq_domain_activate_irq, it will
>>> trigger NULL pointer
>>> dereference in host kernel.
>>> I meet a problem here. When hw support GIC4.1, and host kernel is
>>> started with
>>> kvm-arm.vgic_v4_enable=0, starting a virtual machine will trigger
>>> NULL pointer
>>> dereference in host.
>>
>> I think the thing is that we should _not_ try to configure vSGIs at all
>> if kvm-arm.vgic_v4_enable=0 (which indicates we don't allow use of the
>> GICv4 of direct injection).
>>
>> We currently set kvm_vgic_global_state.has_gicv4_1 to true if HW support
>> GICv4.1, regardless whatever the gicv4_enable is (see patch#23 -
>> vgic_v3_probe). I think this is what actually needs fixing.
>
> Yes, my point exactly. I've pushed out a potential fix [1], and I'd be
> grateful if you could let me know whether that fixes it for you.
I haven't had the appropriate HW yet.. Nianyao or Shaokun can help to
test it tomorrow, I think.
>
> Thanks,
>
> Â Â Â Â Â Â Â M.
>
> [1]
> https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/commit/?h=irq/gic-v4.1-devel&id=b82c2ee1d3fef66fb85793965c344260f618219d
Anyway, this looks good to me.
Thanks,
Zenghui
Hi Marc,
On 2020/1/15 21:32, Marc Zyngier wrote:
> On 2020-01-15 03:49, Zenghui Yu wrote:
>> Hi,
>>
>> On 2020/1/15 10:49, Shaokun Zhang wrote:
>>> Hi Marc, [This is from Nianyao]
>>>
>>> On 2019/12/24 19:10, Marc Zyngier wrote:
>>>> In order to let a guest buy in the new, active-less SGIs, we
>>>> need to be able to switch between the two modes.
>>>>
>>>> Handle this by stopping all guest activity, transfer the state
>>>> from one mode to the other, and resume the guest.
>>>>
>>>> Signed-off-by: Marc Zyngier <[email protected]>
>>>> ---
>>
>> [...]
>>
>>>> diff --git a/virt/kvm/arm/vgic/vgic-v4.c b/virt/kvm/arm/vgic/vgic-v4.c
>>>> index c2fcde104ea2..063785fd2dc7 100644
>>>> --- a/virt/kvm/arm/vgic/vgic-v4.c
>>>> +++ b/virt/kvm/arm/vgic/vgic-v4.c
>>>> @@ -97,6 +97,102 @@ static irqreturn_t vgic_v4_doorbell_handler(int irq, void *info)
>>>> return IRQ_HANDLED;
>>>> }
>>>> +static void vgic_v4_sync_sgi_config(struct its_vpe *vpe, struct vgic_irq *irq)
>>>> +{
>>>> + vpe->sgi_config[irq->intid].enabled = irq->enabled;
>>>> + vpe->sgi_config[irq->intid].group = irq->group;
>>>> + vpe->sgi_config[irq->intid].priority = irq->priority;
>>>> +}
>>>> +
>>>> +static void vgic_v4_enable_vsgis(struct kvm_vcpu *vcpu)
>>>> +{
>>>> + struct its_vpe *vpe = &vcpu->arch.vgic_cpu.vgic_v3.its_vpe;
>>>> + int i;
>>>> +
>>>> + /*
>>>> + * With GICv4.1, every virtual SGI can be directly injected. So
>>>> + * let's pretend that they are HW interrupts, tied to a host
>>>> + * IRQ. The SGI code will do its magic.
>>>> + */
>>>> + for (i = 0; i < VGIC_NR_SGIS; i++) {
>>>> + struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, i);
>>>> + struct irq_desc *desc;
>>>> + int ret;
>>>> +
>>>> + if (irq->hw) {
>>>> + vgic_put_irq(vcpu->kvm, irq);
>>>> + continue;
>>>> + }
>>>> +
>>>> + irq->hw = true;
>>>> + irq->host_irq = irq_find_mapping(vpe->sgi_domain, i);
>>>
>>> I think we need to check whether irq_find_mapping returns 0.
>>>
>>>> + vgic_v4_sync_sgi_config(vpe, irq);
>>>> + /*
>>>> + * SGIs are initialised as disabled. Enable them if
>>>> + * required by the rest of the VGIC init code.
>>>> + */
>>>> + desc = irq_to_desc(irq->host_irq);
>>>> + ret = irq_domain_activate_irq(irq_desc_get_irq_data(desc),
>>>> + false);
>>>
>>> If irq->host_irq is not valid , in irq_domain_activate_irq, it will trigger NULL pointer
>>> dereference in host kernel.
>>> I meet a problem here. When hw support GIC4.1, and host kernel is started with
>>> kvm-arm.vgic_v4_enable=0, starting a virtual machine will trigger NULL pointer
>>> dereference in host.
>>
>> I think the thing is that we should _not_ try to configure vSGIs at all
>> if kvm-arm.vgic_v4_enable=0 (which indicates we don't allow use of the
>> GICv4 of direct injection).
>>
>> We currently set kvm_vgic_global_state.has_gicv4_1 to true if HW support
>> GICv4.1, regardless whatever the gicv4_enable is (see patch#23 -
>> vgic_v3_probe). I think this is what actually needs fixing.
>
> Yes, my point exactly. I've pushed out a potential fix [1], and I'd be
> grateful if you could let me know whether that fixes it for you.
>
Nianyao has tested the patch, it works.
Thanks for your quick reply.
> Thanks,
>
> M.
>
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/commit/?h=irq/gic-v4.1-devel&id=b82c2ee1d3fef66fb85793965c344260f618219d
Hi Marc,
On 2019/12/24 19:10, Marc Zyngier wrote:
> GICv4.1 defines a new VPE table that is potentially shared between
> both the ITSs and the redistributors, following complicated affinity
> rules.
>
> To make things more confusing, the programming of this table at
> the redistributor level is reusing the GICv4.0 GICR_VPROPBASER register
> for something completely different.
>
> The code flow is somewhat complexified by the need to respect the
> affinities required by the HW, meaning that tables can either be
> inherited from a previously discovered ITS or redistributor.
>
> Signed-off-by: Marc Zyngier <[email protected]>
Reviewed-by: Zenghui Yu <[email protected]>
With two very minor concerns below.
[...]
> +static int allocate_vpe_l1_table(void)
> +{
> + void __iomem *vlpi_base = gic_data_rdist_vlpi_base();
> + u64 val, gpsz, npg, pa;
> + unsigned int psz = SZ_64K;
> + unsigned int np, epp, esz;
> + struct page *page;
> +
> + if (!gic_rdists->has_rvpeid)
> + return 0;
> +
> + /*
> + * if VPENDBASER.Valid is set, disable any previously programmed
> + * VPE by setting PendingLast while clearing Valid. This has the
> + * effect of making sure no doorbell will be generated and we can
> + * then safely clear VPROPBASER.Valid.
> + */
> + if (gits_read_vpendbaser(vlpi_base + GICR_VPENDBASER) & GICR_VPENDBASER_Valid)
> + gits_write_vpendbaser(GICR_VPENDBASER_PendingLast,
> + vlpi_base + GICR_VPENDBASER);
I'm confused here. The Valid field resets to 0. Under which scenario
will the Valid==1 while we're doing initialization for this RD?
> +
> + /*
> + * If we can inherit the configuration from another RD, let's do
> + * so. Otherwise, we have to go through the allocation process. We
> + * assume that all RDs have the exact same requirements, as
> + * nothing will work otherwise.
> + */
> + val = inherit_vpe_l1_table_from_rd(&gic_data_rdist()->vpe_table_mask);
> + if (val & GICR_VPROPBASER_4_1_VALID)
> + goto out;
> +
> + gic_data_rdist()->vpe_table_mask = kzalloc(sizeof(cpumask_t), GFP_KERNEL);
> + if (!gic_data_rdist()->vpe_table_mask)
> + return -ENOMEM;
> +
> + val = inherit_vpe_l1_table_from_its();
> + if (val & GICR_VPROPBASER_4_1_VALID)
> + goto out;
> +
> + /* First probe the page size */
> + val = FIELD_PREP(GICR_VPROPBASER_4_1_PAGE_SIZE, GIC_PAGE_SIZE_64K);
> + gits_write_vpropbaser(val, vlpi_base + GICR_VPROPBASER);
> + val = gits_read_vpropbaser(vlpi_base + GICR_VPROPBASER);
> + gpsz = FIELD_GET(GICR_VPROPBASER_4_1_PAGE_SIZE, val);
> + esz = FIELD_GET(GICR_VPROPBASER_4_1_ENTRY_SIZE, val);
> +
> + switch (gpsz) {
> + default:
> + gpsz = GIC_PAGE_SIZE_4K;
> + /* fall through */
> + case GIC_PAGE_SIZE_4K:
> + psz = SZ_4K;
> + break;
> + case GIC_PAGE_SIZE_16K:
> + psz = SZ_16K;
> + break;
> + case GIC_PAGE_SIZE_64K:
> + psz = SZ_64K;
> + break;
> + }
> +
> + /*
> + * Start populating the register from scratch, including RO fields
> + * (which we want to print in debug cases...)
> + */
> + val = 0;
> + val |= FIELD_PREP(GICR_VPROPBASER_4_1_PAGE_SIZE, gpsz);
> + val |= FIELD_PREP(GICR_VPROPBASER_4_1_ENTRY_SIZE, esz);
> +
> + /* How many entries per GIC page? */
> + esz++;
> + epp = psz / (esz * SZ_8);
> +
> + /*
> + * If we need more than just a single L1 page, flag the table
> + * as indirect and compute the number of required L1 pages.
> + */
> + if (epp < ITS_MAX_VPEID) {
> + int nl2;
> +
> + gic_rdists->flags |= RDIST_FLAGS_VPE_INDIRECT;
This flag is set but not used, can we just drop it?
Thanks,
Zenghui
On 2020-01-20 14:03, Zenghui Yu wrote:
> Hi Marc,
>
> On 2019/12/24 19:10, Marc Zyngier wrote:
>> GICv4.1 defines a new VPE table that is potentially shared between
>> both the ITSs and the redistributors, following complicated affinity
>> rules.
>>
>> To make things more confusing, the programming of this table at
>> the redistributor level is reusing the GICv4.0 GICR_VPROPBASER
>> register
>> for something completely different.
>>
>> The code flow is somewhat complexified by the need to respect the
>> affinities required by the HW, meaning that tables can either be
>> inherited from a previously discovered ITS or redistributor.
>>
>> Signed-off-by: Marc Zyngier <[email protected]>
>
> Reviewed-by: Zenghui Yu <[email protected]>
>
> With two very minor concerns below.
>
> [...]
>
>> +static int allocate_vpe_l1_table(void)
>> +{
>> + void __iomem *vlpi_base = gic_data_rdist_vlpi_base();
>> + u64 val, gpsz, npg, pa;
>> + unsigned int psz = SZ_64K;
>> + unsigned int np, epp, esz;
>> + struct page *page;
>> +
>> + if (!gic_rdists->has_rvpeid)
>> + return 0;
>> +
>> + /*
>> + * if VPENDBASER.Valid is set, disable any previously programmed
>> + * VPE by setting PendingLast while clearing Valid. This has the
>> + * effect of making sure no doorbell will be generated and we can
>> + * then safely clear VPROPBASER.Valid.
>> + */
>> + if (gits_read_vpendbaser(vlpi_base + GICR_VPENDBASER) &
>> GICR_VPENDBASER_Valid)
>> + gits_write_vpendbaser(GICR_VPENDBASER_PendingLast,
>> + vlpi_base + GICR_VPENDBASER);
>
> I'm confused here. The Valid field resets to 0. Under which scenario
> will the Valid==1 while we're doing initialization for this RD?
The cases I'm worried about are things like kexec or cpu hotplug,
where we could find that the RD programming is still valid, one way
or another. This is unlikely to hit in practice, but who knows...
>
>> +
>> + /*
>> + * If we can inherit the configuration from another RD, let's do
>> + * so. Otherwise, we have to go through the allocation process. We
>> + * assume that all RDs have the exact same requirements, as
>> + * nothing will work otherwise.
>> + */
>> + val =
>> inherit_vpe_l1_table_from_rd(&gic_data_rdist()->vpe_table_mask);
>> + if (val & GICR_VPROPBASER_4_1_VALID)
>> + goto out;
>> +
>> + gic_data_rdist()->vpe_table_mask = kzalloc(sizeof(cpumask_t),
>> GFP_KERNEL);
>> + if (!gic_data_rdist()->vpe_table_mask)
>> + return -ENOMEM;
>> +
>> + val = inherit_vpe_l1_table_from_its();
>> + if (val & GICR_VPROPBASER_4_1_VALID)
>> + goto out;
>> +
>> + /* First probe the page size */
>> + val = FIELD_PREP(GICR_VPROPBASER_4_1_PAGE_SIZE, GIC_PAGE_SIZE_64K);
>> + gits_write_vpropbaser(val, vlpi_base + GICR_VPROPBASER);
>> + val = gits_read_vpropbaser(vlpi_base + GICR_VPROPBASER);
>> + gpsz = FIELD_GET(GICR_VPROPBASER_4_1_PAGE_SIZE, val);
>> + esz = FIELD_GET(GICR_VPROPBASER_4_1_ENTRY_SIZE, val);
>> +
>> + switch (gpsz) {
>> + default:
>> + gpsz = GIC_PAGE_SIZE_4K;
>> + /* fall through */
>> + case GIC_PAGE_SIZE_4K:
>> + psz = SZ_4K;
>> + break;
>> + case GIC_PAGE_SIZE_16K:
>> + psz = SZ_16K;
>> + break;
>> + case GIC_PAGE_SIZE_64K:
>> + psz = SZ_64K;
>> + break;
>> + }
>> +
>> + /*
>> + * Start populating the register from scratch, including RO fields
>> + * (which we want to print in debug cases...)
>> + */
>> + val = 0;
>> + val |= FIELD_PREP(GICR_VPROPBASER_4_1_PAGE_SIZE, gpsz);
>> + val |= FIELD_PREP(GICR_VPROPBASER_4_1_ENTRY_SIZE, esz);
>> +
>> + /* How many entries per GIC page? */
>> + esz++;
>> + epp = psz / (esz * SZ_8);
>> +
>> + /*
>> + * If we need more than just a single L1 page, flag the table
>> + * as indirect and compute the number of required L1 pages.
>> + */
>> + if (epp < ITS_MAX_VPEID) {
>> + int nl2;
>> +
>> + gic_rdists->flags |= RDIST_FLAGS_VPE_INDIRECT;
>
> This flag is set but not used, can we just drop it?
Yes, good point. I can't even remember what I had in mind with this
flag, so it can't be that important! ;-).
I'll clean that up.
Thanks,
M.
--
Jazz is not dead. It just smells funny...
On 2019/12/24 19:10, Marc Zyngier wrote:
> @@ -4147,6 +4453,8 @@ int __init its_init(struct fwnode_handle *handle, struct rdists *rdists,
> bool has_v4 = false;
> int err;
>
> + gic_rdists = rdists;
> +
> its_parent = parent_domain;
> of_node = to_of_node(handle);
> if (of_node)
> @@ -4159,8 +4467,6 @@ int __init its_init(struct fwnode_handle *handle, struct rdists *rdists,
> return -ENXIO;
> }
>
> - gic_rdists = rdists;
> -
> err = allocate_lpi_tables();
> if (err)
> return err;
And shouldn't this be part of patch#2? (As the new ITS_MAX_VPEID_BITS
would use gic_rdists in the allocation of ITS vPE table.)
But I haven't tested the first two patches separately, I guess it may
crash my box ;-)
Thanks,
Zenghui
The following commit has been merged into the irq/core branch of tip:
Commit-ID: 91bf6395f7b8614a5a9934a0ae9c8b5312d77b29
Gitweb: https://git.kernel.org/tip/91bf6395f7b8614a5a9934a0ae9c8b5312d77b29
Author: Marc Zyngier <[email protected]>
AuthorDate: Tue, 24 Dec 2019 11:10:34
Committer: Marc Zyngier <[email protected]>
CommitterDate: Wed, 22 Jan 2020 14:22:20
irqchip/gic-v4.1: Add VPE residency callback
Making a VPE resident on GICv4.1 is pretty simple, as it is just a
single write to the local redistributor. We just need extra information
about which groups to enable, which the KVM code will have to provide.
Signed-off-by: Marc Zyngier <[email protected]>
Reviewed-by: Zenghui Yu <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
---
drivers/irqchip/irq-gic-v3-its.c | 17 +++++++++++++++++
include/linux/irqchip/arm-gic-v3.h | 9 +++++++++
include/linux/irqchip/arm-gic-v4.h | 5 +++++
3 files changed, 31 insertions(+)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 5ef706e..3adc597 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -3624,12 +3624,29 @@ static void its_vpe_4_1_unmask_irq(struct irq_data *d)
its_vpe_4_1_send_inv(d);
}
+static void its_vpe_4_1_schedule(struct its_vpe *vpe,
+ struct its_cmd_info *info)
+{
+ void __iomem *vlpi_base = gic_data_rdist_vlpi_base();
+ u64 val = 0;
+
+ /* Schedule the VPE */
+ val |= GICR_VPENDBASER_Valid;
+ val |= info->g0en ? GICR_VPENDBASER_4_1_VGRP0EN : 0;
+ val |= info->g1en ? GICR_VPENDBASER_4_1_VGRP1EN : 0;
+ val |= FIELD_PREP(GICR_VPENDBASER_4_1_VPEID, vpe->vpe_id);
+
+ gits_write_vpendbaser(val, vlpi_base + GICR_VPENDBASER);
+}
+
static int its_vpe_4_1_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
{
+ struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
struct its_cmd_info *info = vcpu_info;
switch (info->cmd_type) {
case SCHEDULE_VPE:
+ its_vpe_4_1_schedule(vpe, info);
return 0;
case DESCHEDULE_VPE:
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index 1f17181..822dae6 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -328,6 +328,15 @@
#define GICR_VPENDBASER_Valid (1ULL << 63)
/*
+ * GICv4.1 VPENDBASER, used for VPE residency. On top of these fields,
+ * also use the above Valid, PendingLast and Dirty.
+ */
+#define GICR_VPENDBASER_4_1_DB (1ULL << 62)
+#define GICR_VPENDBASER_4_1_VGRP0EN (1ULL << 59)
+#define GICR_VPENDBASER_4_1_VGRP1EN (1ULL << 58)
+#define GICR_VPENDBASER_4_1_VPEID GENMASK_ULL(15, 0)
+
+/*
* ITS registers, offsets from ITS_base
*/
#define GITS_CTLR 0x0000
diff --git a/include/linux/irqchip/arm-gic-v4.h b/include/linux/irqchip/arm-gic-v4.h
index 498e523..d9c3496 100644
--- a/include/linux/irqchip/arm-gic-v4.h
+++ b/include/linux/irqchip/arm-gic-v4.h
@@ -100,6 +100,11 @@ struct its_cmd_info {
union {
struct its_vlpi_map *map;
u8 config;
+ bool req_db;
+ struct {
+ bool g0en;
+ bool g1en;
+ };
};
};
The following commit has been merged into the irq/core branch of tip:
Commit-ID: d97c97baa214486cc3d64c996a2214475f6cc83c
Gitweb: https://git.kernel.org/tip/d97c97baa214486cc3d64c996a2214475f6cc83c
Author: Marc Zyngier <[email protected]>
AuthorDate: Tue, 24 Dec 2019 11:10:33
Committer: Marc Zyngier <[email protected]>
CommitterDate: Wed, 22 Jan 2020 14:22:20
irqchip/gic-v4.1: Add mask/unmask doorbell callbacks
masking/unmasking doorbells on GICv4.1 relies on a new INVDB command,
which broadcasts the invalidation to all RDs.
Implement the new command as well as the masking callbacks, and plug
the whole thing into the v4.1 VPE irqchip.
Signed-off-by: Marc Zyngier <[email protected]>
Reviewed-by: Zenghui Yu <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
---
drivers/irqchip/irq-gic-v3-its.c | 73 +++++++++++++++++++++++++++++-
include/linux/irqchip/arm-gic-v3.h | 3 +-
2 files changed, 75 insertions(+), 1 deletion(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 2b477e2..5ef706e 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -333,6 +333,10 @@ struct its_cmd_desc {
u16 seq_num;
u16 its_list;
} its_vmovp_cmd;
+
+ struct {
+ struct its_vpe *vpe;
+ } its_invdb_cmd;
};
};
@@ -831,6 +835,21 @@ static struct its_vpe *its_build_vclear_cmd(struct its_node *its,
return valid_vpe(its, map->vpe);
}
+static struct its_vpe *its_build_invdb_cmd(struct its_node *its,
+ struct its_cmd_block *cmd,
+ struct its_cmd_desc *desc)
+{
+ if (WARN_ON(!is_v4_1(its)))
+ return NULL;
+
+ its_encode_cmd(cmd, GITS_CMD_INVDB);
+ its_encode_vpeid(cmd, desc->its_invdb_cmd.vpe->vpe_id);
+
+ its_fixup_cmd(cmd);
+
+ return valid_vpe(its, desc->its_invdb_cmd.vpe);
+}
+
static u64 its_cmd_ptr_to_offset(struct its_node *its,
struct its_cmd_block *ptr)
{
@@ -1239,6 +1258,14 @@ static void its_send_vclear(struct its_device *dev, u32 event_id)
its_send_single_vcommand(dev->its, its_build_vclear_cmd, &desc);
}
+static void its_send_invdb(struct its_node *its, struct its_vpe *vpe)
+{
+ struct its_cmd_desc desc;
+
+ desc.its_invdb_cmd.vpe = vpe;
+ its_send_single_vcommand(its, its_build_invdb_cmd, &desc);
+}
+
/*
* irqchip functions - assumes MSI, mostly.
*/
@@ -3553,6 +3580,50 @@ static struct irq_chip its_vpe_irq_chip = {
.irq_set_vcpu_affinity = its_vpe_set_vcpu_affinity,
};
+static struct its_node *find_4_1_its(void)
+{
+ static struct its_node *its = NULL;
+
+ if (!its) {
+ list_for_each_entry(its, &its_nodes, entry) {
+ if (is_v4_1(its))
+ return its;
+ }
+
+ /* Oops? */
+ its = NULL;
+ }
+
+ return its;
+}
+
+static void its_vpe_4_1_send_inv(struct irq_data *d)
+{
+ struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
+ struct its_node *its;
+
+ /*
+ * GICv4.1 wants doorbells to be invalidated using the
+ * INVDB command in order to be broadcast to all RDs. Send
+ * it to the first valid ITS, and let the HW do its magic.
+ */
+ its = find_4_1_its();
+ if (its)
+ its_send_invdb(its, vpe);
+}
+
+static void its_vpe_4_1_mask_irq(struct irq_data *d)
+{
+ lpi_write_config(d->parent_data, LPI_PROP_ENABLED, 0);
+ its_vpe_4_1_send_inv(d);
+}
+
+static void its_vpe_4_1_unmask_irq(struct irq_data *d)
+{
+ lpi_write_config(d->parent_data, 0, LPI_PROP_ENABLED);
+ its_vpe_4_1_send_inv(d);
+}
+
static int its_vpe_4_1_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
{
struct its_cmd_info *info = vcpu_info;
@@ -3574,6 +3645,8 @@ static int its_vpe_4_1_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
static struct irq_chip its_vpe_4_1_irq_chip = {
.name = "GICv4.1-vpe",
+ .irq_mask = its_vpe_4_1_mask_irq,
+ .irq_unmask = its_vpe_4_1_unmask_irq,
.irq_eoi = irq_chip_eoi_parent,
.irq_set_affinity = its_vpe_set_affinity,
.irq_set_vcpu_affinity = its_vpe_4_1_set_vcpu_affinity,
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index db0a111..1f17181 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -484,8 +484,9 @@
#define GITS_CMD_VMAPTI GITS_CMD_GICv4(GITS_CMD_MAPTI)
#define GITS_CMD_VMOVI GITS_CMD_GICv4(GITS_CMD_MOVI)
#define GITS_CMD_VSYNC GITS_CMD_GICv4(GITS_CMD_SYNC)
-/* VMOVP is the odd one, as it doesn't have a physical counterpart */
+/* VMOVP and INVDB are the odd ones, as they dont have a physical counterpart */
#define GITS_CMD_VMOVP GITS_CMD_GICv4(2)
+#define GITS_CMD_INVDB GITS_CMD_GICv4(0xe)
/*
* ITS error numbers
The following commit has been merged into the irq/core branch of tip:
Commit-ID: dd3f050a216ef7c8ce21ba48fd3b2ece2155382f
Gitweb: https://git.kernel.org/tip/dd3f050a216ef7c8ce21ba48fd3b2ece2155382f
Author: Marc Zyngier <[email protected]>
AuthorDate: Tue, 24 Dec 2019 11:10:31
Committer: Marc Zyngier <[email protected]>
CommitterDate: Wed, 22 Jan 2020 14:22:20
irqchip/gic-v4.1: Implement the v4.1 flavour of VMOVP
With GICv4.1, VMOVP is extended to allow a default doorbell to be
specified, as well as a validity bit for this doorbell. As an added
bonus, VMOVP isn't required anymore of moving a VPE between
redistributors that share the same affinity.
Let's add this support to the VMOVP builder, and make sure we don't
issue the command if we don't really need to.
Signed-off-by: Marc Zyngier <[email protected]>
Reviewed-by: Zenghui Yu <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
---
drivers/irqchip/irq-gic-v3-its.c | 40 +++++++++++++++++++++++++------
1 file changed, 33 insertions(+), 7 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 9bc8adf..53a7663 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -470,6 +470,17 @@ static void its_encode_vmapp_default_db(struct its_cmd_block *cmd,
its_mask_encode(&cmd->raw_cmd[1], vpe_db_lpi, 31, 0);
}
+static void its_encode_vmovp_default_db(struct its_cmd_block *cmd,
+ u32 vpe_db_lpi)
+{
+ its_mask_encode(&cmd->raw_cmd[3], vpe_db_lpi, 31, 0);
+}
+
+static void its_encode_db(struct its_cmd_block *cmd, bool db)
+{
+ its_mask_encode(&cmd->raw_cmd[2], db, 63, 63);
+}
+
static inline void its_fixup_cmd(struct its_cmd_block *cmd)
{
/* Let's fixup BE commands */
@@ -756,6 +767,11 @@ static struct its_vpe *its_build_vmovp_cmd(struct its_node *its,
its_encode_vpeid(cmd, desc->its_vmovp_cmd.vpe->vpe_id);
its_encode_target(cmd, target);
+ if (is_v4_1(its)) {
+ its_encode_db(cmd, true);
+ its_encode_vmovp_default_db(cmd, desc->its_vmovp_cmd.vpe->vpe_db_lpi);
+ }
+
its_fixup_cmd(cmd);
return valid_vpe(its, desc->its_vmovp_cmd.vpe);
@@ -3327,7 +3343,7 @@ static int its_vpe_set_affinity(struct irq_data *d,
bool force)
{
struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
- int cpu = cpumask_first(mask_val);
+ int from, cpu = cpumask_first(mask_val);
/*
* Changing affinity is mega expensive, so let's be as lazy as
@@ -3335,14 +3351,24 @@ static int its_vpe_set_affinity(struct irq_data *d,
* into the proxy device, we need to move the doorbell
* interrupt to its new location.
*/
- if (vpe->col_idx != cpu) {
- int from = vpe->col_idx;
+ if (vpe->col_idx == cpu)
+ goto out;
- vpe->col_idx = cpu;
- its_send_vmovp(vpe);
- its_vpe_db_proxy_move(vpe, from, cpu);
- }
+ from = vpe->col_idx;
+ vpe->col_idx = cpu;
+
+ /*
+ * GICv4.1 allows us to skip VMOVP if moving to a cpu whose RD
+ * is sharing its VPE table with the current one.
+ */
+ if (gic_data_rdist_cpu(cpu)->vpe_table_mask &&
+ cpumask_test_cpu(from, gic_data_rdist_cpu(cpu)->vpe_table_mask))
+ goto out;
+ its_send_vmovp(vpe);
+ its_vpe_db_proxy_move(vpe, from, cpu);
+
+out:
irq_data_update_effective_affinity(d, cpumask_of(cpu));
return IRQ_SET_MASK_OK_DONE;
The following commit has been merged into the irq/core branch of tip:
Commit-ID: e64fab1a1477dbf0c355691914511612ba312932
Gitweb: https://git.kernel.org/tip/e64fab1a1477dbf0c355691914511612ba312932
Author: Marc Zyngier <[email protected]>
AuthorDate: Tue, 24 Dec 2019 11:10:35
Committer: Marc Zyngier <[email protected]>
CommitterDate: Wed, 22 Jan 2020 14:22:20
irqchip/gic-v4.1: Add VPE eviction callback
When descheduling a VPE, special care must be taken to tell the GIC
about whether we want to receive a doorbell or not. This is a
major improvement on GICv4.0, where the doorbell had to be separately
enabled/disabled.
Signed-off-by: Marc Zyngier <[email protected]>
Reviewed-by: Zenghui Yu <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
---
drivers/irqchip/irq-gic-v3-its.c | 53 ++++++++++++++++++++++++-------
1 file changed, 42 insertions(+), 11 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 3adc597..69b16e5 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -2630,7 +2630,7 @@ static int __init allocate_lpi_tables(void)
return 0;
}
-static u64 its_clear_vpend_valid(void __iomem *vlpi_base)
+static u64 its_clear_vpend_valid(void __iomem *vlpi_base, u64 clr, u64 set)
{
u32 count = 1000000; /* 1s! */
bool clean;
@@ -2638,6 +2638,8 @@ static u64 its_clear_vpend_valid(void __iomem *vlpi_base)
val = gits_read_vpendbaser(vlpi_base + GICR_VPENDBASER);
val &= ~GICR_VPENDBASER_Valid;
+ val &= ~clr;
+ val |= set;
gits_write_vpendbaser(val, vlpi_base + GICR_VPENDBASER);
do {
@@ -2650,6 +2652,11 @@ static u64 its_clear_vpend_valid(void __iomem *vlpi_base)
}
} while (!clean && count);
+ if (unlikely(val & GICR_VPENDBASER_Dirty)) {
+ pr_err_ratelimited("ITS virtual pending table not cleaning\n");
+ val |= GICR_VPENDBASER_PendingLast;
+ }
+
return val;
}
@@ -2758,7 +2765,7 @@ static void its_cpu_init_lpis(void)
* ancient programming gets left in and has possibility of
* corrupting memory.
*/
- val = its_clear_vpend_valid(vlpi_base);
+ val = its_clear_vpend_valid(vlpi_base, 0, 0);
WARN_ON(val & GICR_VPENDBASER_Dirty);
}
@@ -3438,16 +3445,10 @@ static void its_vpe_deschedule(struct its_vpe *vpe)
void __iomem *vlpi_base = gic_data_rdist_vlpi_base();
u64 val;
- val = its_clear_vpend_valid(vlpi_base);
+ val = its_clear_vpend_valid(vlpi_base, 0, 0);
- if (unlikely(val & GICR_VPENDBASER_Dirty)) {
- pr_err_ratelimited("ITS virtual pending table not cleaning\n");
- vpe->idai = false;
- vpe->pending_last = true;
- } else {
- vpe->idai = !!(val & GICR_VPENDBASER_IDAI);
- vpe->pending_last = !!(val & GICR_VPENDBASER_PendingLast);
- }
+ vpe->idai = !!(val & GICR_VPENDBASER_IDAI);
+ vpe->pending_last = !!(val & GICR_VPENDBASER_PendingLast);
}
static void its_vpe_invall(struct its_vpe *vpe)
@@ -3639,6 +3640,35 @@ static void its_vpe_4_1_schedule(struct its_vpe *vpe,
gits_write_vpendbaser(val, vlpi_base + GICR_VPENDBASER);
}
+static void its_vpe_4_1_deschedule(struct its_vpe *vpe,
+ struct its_cmd_info *info)
+{
+ void __iomem *vlpi_base = gic_data_rdist_vlpi_base();
+ u64 val;
+
+ if (info->req_db) {
+ /*
+ * vPE is going to block: make the vPE non-resident with
+ * PendingLast clear and DB set. The GIC guarantees that if
+ * we read-back PendingLast clear, then a doorbell will be
+ * delivered when an interrupt comes.
+ */
+ val = its_clear_vpend_valid(vlpi_base,
+ GICR_VPENDBASER_PendingLast,
+ GICR_VPENDBASER_4_1_DB);
+ vpe->pending_last = !!(val & GICR_VPENDBASER_PendingLast);
+ } else {
+ /*
+ * We're not blocking, so just make the vPE non-resident
+ * with PendingLast set, indicating that we'll be back.
+ */
+ val = its_clear_vpend_valid(vlpi_base,
+ 0,
+ GICR_VPENDBASER_PendingLast);
+ vpe->pending_last = true;
+ }
+}
+
static int its_vpe_4_1_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
{
struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
@@ -3650,6 +3680,7 @@ static int its_vpe_4_1_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
return 0;
case DESCHEDULE_VPE:
+ its_vpe_4_1_deschedule(vpe, info);
return 0;
case INVALL_VPE:
The following commit has been merged into the irq/core branch of tip:
Commit-ID: b4a4bd0f2629ec2ece7690de1b4721529da29871
Gitweb: https://git.kernel.org/tip/b4a4bd0f2629ec2ece7690de1b4721529da29871
Author: Marc Zyngier <[email protected]>
AuthorDate: Tue, 24 Dec 2019 11:10:36
Committer: Marc Zyngier <[email protected]>
CommitterDate: Wed, 22 Jan 2020 14:22:21
irqchip/gic-v4.1: Add VPE INVALL callback
GICv4.1 redistributors have a VPE-aware INVALL register. Progress!
We can now emulate a guest-requested INVALL without emiting a
VINVALL command.
Signed-off-by: Marc Zyngier <[email protected]>
Reviewed-by: Zenghui Yu <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
---
drivers/irqchip/irq-gic-v3-its.c | 14 ++++++++++++++
include/linux/irqchip/arm-gic-v3.h | 6 ++++++
2 files changed, 20 insertions(+)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 69b16e5..1d8d96a 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -3669,6 +3669,19 @@ static void its_vpe_4_1_deschedule(struct its_vpe *vpe,
}
}
+static void its_vpe_4_1_invall(struct its_vpe *vpe)
+{
+ void __iomem *rdbase;
+ u64 val;
+
+ val = GICR_INVALLR_V;
+ val |= FIELD_PREP(GICR_INVALLR_VPEID, vpe->vpe_id);
+
+ /* Target the redistributor this vPE is currently known on */
+ rdbase = per_cpu_ptr(gic_rdists->rdist, vpe->col_idx)->rd_base;
+ gic_write_lpir(val, rdbase + GICR_INVALLR);
+}
+
static int its_vpe_4_1_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
{
struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
@@ -3684,6 +3697,7 @@ static int its_vpe_4_1_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
return 0;
case INVALL_VPE:
+ its_vpe_4_1_invall(vpe);
return 0;
default:
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index 822dae6..49ed6fa 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -247,6 +247,12 @@
#define GICR_TYPER_COMMON_LPI_AFF GENMASK_ULL(25, 24)
#define GICR_TYPER_AFFINITY GENMASK_ULL(63, 32)
+#define GICR_INVLPIR_VPEID GENMASK_ULL(47, 32)
+#define GICR_INVLPIR_V GENMASK_ULL(63, 63)
+
+#define GICR_INVALLR_VPEID GICR_INVLPIR_VPEID
+#define GICR_INVALLR_V GICR_INVLPIR_V
+
#define GIC_V3_REDIST_SIZE 0x20000
#define LPI_PROP_GROUP1 (1 << 1)
The following commit has been merged into the irq/core branch of tip:
Commit-ID: 0684c7046590dd1e8047e187aaf4c7910cc35bce
Gitweb: https://git.kernel.org/tip/0684c7046590dd1e8047e187aaf4c7910cc35bce
Author: Marc Zyngier <[email protected]>
AuthorDate: Tue, 24 Dec 2019 11:10:30
Committer: Marc Zyngier <[email protected]>
CommitterDate: Wed, 22 Jan 2020 14:22:20
irqchip/gic-v4.1: Don't use the VPE proxy if RVPEID is set
The infamous VPE proxy device isn't used with GICv4.1 because:
- we can invalidate any LPI from the DirectLPI MMIO interface
- the ITS and redistributors understand the life cycle of
the doorbell, so we don't need to enable/disable it all
the time
So let's escape early from the proxy related functions.
Signed-off-by: Marc Zyngier <[email protected]>
Reviewed-by: Zenghui Yu <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
---
drivers/irqchip/irq-gic-v3-its.c | 25 ++++++++++++++++++++++++-
1 file changed, 24 insertions(+), 1 deletion(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index fbf4ca7..9bc8adf 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -3212,7 +3212,7 @@ static const struct irq_domain_ops its_domain_ops = {
/*
* This is insane.
*
- * If a GICv4 doesn't implement Direct LPIs (which is extremely
+ * If a GICv4.0 doesn't implement Direct LPIs (which is extremely
* likely), the only way to perform an invalidate is to use a fake
* device to issue an INV command, implying that the LPI has first
* been mapped to some event on that device. Since this is not exactly
@@ -3220,9 +3220,20 @@ static const struct irq_domain_ops its_domain_ops = {
* only issue an UNMAP if we're short on available slots.
*
* Broken by design(tm).
+ *
+ * GICv4.1, on the other hand, mandates that we're able to invalidate
+ * by writing to a MMIO register. It doesn't implement the whole of
+ * DirectLPI, but that's good enough. And most of the time, we don't
+ * even have to invalidate anything, as the redistributor can be told
+ * whether to generate a doorbell or not (we thus leave it enabled,
+ * always).
*/
static void its_vpe_db_proxy_unmap_locked(struct its_vpe *vpe)
{
+ /* GICv4.1 doesn't use a proxy, so nothing to do here */
+ if (gic_rdists->has_rvpeid)
+ return;
+
/* Already unmapped? */
if (vpe->vpe_proxy_event == -1)
return;
@@ -3245,6 +3256,10 @@ static void its_vpe_db_proxy_unmap_locked(struct its_vpe *vpe)
static void its_vpe_db_proxy_unmap(struct its_vpe *vpe)
{
+ /* GICv4.1 doesn't use a proxy, so nothing to do here */
+ if (gic_rdists->has_rvpeid)
+ return;
+
if (!gic_rdists->has_direct_lpi) {
unsigned long flags;
@@ -3256,6 +3271,10 @@ static void its_vpe_db_proxy_unmap(struct its_vpe *vpe)
static void its_vpe_db_proxy_map_locked(struct its_vpe *vpe)
{
+ /* GICv4.1 doesn't use a proxy, so nothing to do here */
+ if (gic_rdists->has_rvpeid)
+ return;
+
/* Already mapped? */
if (vpe->vpe_proxy_event != -1)
return;
@@ -3278,6 +3297,10 @@ static void its_vpe_db_proxy_move(struct its_vpe *vpe, int from, int to)
unsigned long flags;
struct its_collection *target_col;
+ /* GICv4.1 doesn't use a proxy, so nothing to do here */
+ if (gic_rdists->has_rvpeid)
+ return;
+
if (gic_rdists->has_direct_lpi) {
void __iomem *rdbase;
The following commit has been merged into the irq/core branch of tip:
Commit-ID: 29c647f3b5ae1a20221d477442dcdf058cea4a21
Gitweb: https://git.kernel.org/tip/29c647f3b5ae1a20221d477442dcdf058cea4a21
Author: Marc Zyngier <[email protected]>
AuthorDate: Tue, 24 Dec 2019 11:10:32
Committer: Marc Zyngier <[email protected]>
CommitterDate: Wed, 22 Jan 2020 14:22:20
irqchip/gic-v4.1: Plumb skeletal VPE irqchip
Just like for GICv4.0, each VPE has its own doorbell interrupt, and
thus an irqchip that manages them. Since the doorbell management is
quite different on GICv4.1, let's introduce an almost empty irqchip
the will get populated over the next new patches.
Signed-off-by: Marc Zyngier <[email protected]>
Reviewed-by: Zenghui Yu <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
---
drivers/irqchip/irq-gic-v3-its.c | 32 ++++++++++++++++++++++++++++++-
1 file changed, 31 insertions(+), 1 deletion(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 53a7663..2b477e2 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -3553,6 +3553,32 @@ static struct irq_chip its_vpe_irq_chip = {
.irq_set_vcpu_affinity = its_vpe_set_vcpu_affinity,
};
+static int its_vpe_4_1_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
+{
+ struct its_cmd_info *info = vcpu_info;
+
+ switch (info->cmd_type) {
+ case SCHEDULE_VPE:
+ return 0;
+
+ case DESCHEDULE_VPE:
+ return 0;
+
+ case INVALL_VPE:
+ return 0;
+
+ default:
+ return -EINVAL;
+ }
+}
+
+static struct irq_chip its_vpe_4_1_irq_chip = {
+ .name = "GICv4.1-vpe",
+ .irq_eoi = irq_chip_eoi_parent,
+ .irq_set_affinity = its_vpe_set_affinity,
+ .irq_set_vcpu_affinity = its_vpe_4_1_set_vcpu_affinity,
+};
+
static int its_vpe_id_alloc(void)
{
return ida_simple_get(&its_vpeid_ida, 0, ITS_MAX_VPEID, GFP_KERNEL);
@@ -3633,6 +3659,7 @@ static void its_vpe_irq_domain_free(struct irq_domain *domain,
static int its_vpe_irq_domain_alloc(struct irq_domain *domain, unsigned int virq,
unsigned int nr_irqs, void *args)
{
+ struct irq_chip *irqchip = &its_vpe_irq_chip;
struct its_vm *vm = args;
unsigned long *bitmap;
struct page *vprop_page;
@@ -3660,6 +3687,9 @@ static int its_vpe_irq_domain_alloc(struct irq_domain *domain, unsigned int virq
vm->nr_db_lpis = nr_ids;
vm->vprop_page = vprop_page;
+ if (gic_rdists->has_rvpeid)
+ irqchip = &its_vpe_4_1_irq_chip;
+
for (i = 0; i < nr_irqs; i++) {
vm->vpes[i]->vpe_db_lpi = base + i;
err = its_vpe_init(vm->vpes[i]);
@@ -3670,7 +3700,7 @@ static int its_vpe_irq_domain_alloc(struct irq_domain *domain, unsigned int virq
if (err)
break;
irq_domain_set_hwirq_and_chip(domain, virq + i, i,
- &its_vpe_irq_chip, vm->vpes[i]);
+ irqchip, vm->vpes[i]);
set_bit(i, bitmap);
}
The following commit has been merged into the irq/core branch of tip:
Commit-ID: f4a81f5a853e0b7c38bfad3afd6d0365d654e777
Gitweb: https://git.kernel.org/tip/f4a81f5a853e0b7c38bfad3afd6d0365d654e777
Author: Marc Zyngier <[email protected]>
AuthorDate: Tue, 24 Dec 2019 11:10:38
Committer: Marc Zyngier <[email protected]>
CommitterDate: Wed, 22 Jan 2020 14:22:21
irqchip/gic-v4.1: Allow direct invalidation of VLPIs
Just like for INVALL, GICv4.1 has grown a VPE-aware INVLPI register.
Let's plumb it in and make use of the DirectLPI code in that case.
Signed-off-by: Marc Zyngier <[email protected]>
Reviewed-by: Zenghui Yu <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
---
drivers/irqchip/irq-gic-v3-its.c | 55 ++++++++++++++++++-----------
include/linux/irqchip/arm-gic-v3.h | 1 +-
2 files changed, 37 insertions(+), 19 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 53e91c9..f717586 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -227,11 +227,27 @@ static struct its_vlpi_map *dev_event_to_vlpi_map(struct its_device *its_dev,
return &its_dev->event_map.vlpi_maps[event];
}
-static struct its_collection *irq_to_col(struct irq_data *d)
+static struct its_vlpi_map *get_vlpi_map(struct irq_data *d)
+{
+ if (irqd_is_forwarded_to_vcpu(d)) {
+ struct its_device *its_dev = irq_data_get_irq_chip_data(d);
+ u32 event = its_get_event_id(d);
+
+ return dev_event_to_vlpi_map(its_dev, event);
+ }
+
+ return NULL;
+}
+
+static int irq_to_cpuid(struct irq_data *d)
{
struct its_device *its_dev = irq_data_get_irq_chip_data(d);
+ struct its_vlpi_map *map = get_vlpi_map(d);
+
+ if (map)
+ return map->vpe->col_idx;
- return dev_event_to_col(its_dev, its_get_event_id(d));
+ return its_dev->event_map.col_map[its_get_event_id(d)];
}
static struct its_collection *valid_col(struct its_collection *col)
@@ -1269,18 +1285,6 @@ static void its_send_invdb(struct its_node *its, struct its_vpe *vpe)
/*
* irqchip functions - assumes MSI, mostly.
*/
-static struct its_vlpi_map *get_vlpi_map(struct irq_data *d)
-{
- if (irqd_is_forwarded_to_vcpu(d)) {
- struct its_device *its_dev = irq_data_get_irq_chip_data(d);
- u32 event = its_get_event_id(d);
-
- return dev_event_to_vlpi_map(its_dev, event);
- }
-
- return NULL;
-}
-
static void lpi_write_config(struct irq_data *d, u8 clr, u8 set)
{
struct its_vlpi_map *map = get_vlpi_map(d);
@@ -1323,13 +1327,25 @@ static void wait_for_syncr(void __iomem *rdbase)
static void direct_lpi_inv(struct irq_data *d)
{
- struct its_collection *col;
+ struct its_vlpi_map *map = get_vlpi_map(d);
void __iomem *rdbase;
+ u64 val;
+
+ if (map) {
+ struct its_device *its_dev = irq_data_get_irq_chip_data(d);
+
+ WARN_ON(!is_v4_1(its_dev->its));
+
+ val = GICR_INVLPIR_V;
+ val |= FIELD_PREP(GICR_INVLPIR_VPEID, map->vpe->vpe_id);
+ val |= FIELD_PREP(GICR_INVLPIR_INTID, map->vintid);
+ } else {
+ val = d->hwirq;
+ }
/* Target the redistributor this LPI is currently routed to */
- col = irq_to_col(d);
- rdbase = per_cpu_ptr(gic_rdists->rdist, col->col_id)->rd_base;
- gic_write_lpir(d->hwirq, rdbase + GICR_INVLPIR);
+ rdbase = per_cpu_ptr(gic_rdists->rdist, irq_to_cpuid(d))->rd_base;
+ gic_write_lpir(val, rdbase + GICR_INVLPIR);
wait_for_syncr(rdbase);
}
@@ -1339,7 +1355,8 @@ static void lpi_update_config(struct irq_data *d, u8 clr, u8 set)
struct its_device *its_dev = irq_data_get_irq_chip_data(d);
lpi_write_config(d, clr, set);
- if (gic_rdists->has_direct_lpi && !irqd_is_forwarded_to_vcpu(d))
+ if (gic_rdists->has_direct_lpi &&
+ (is_v4_1(its_dev->its) || !irqd_is_forwarded_to_vcpu(d)))
direct_lpi_inv(d);
else if (!irqd_is_forwarded_to_vcpu(d))
its_send_inv(its_dev, its_get_event_id(d));
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index 49ed6fa..f0b8ca7 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -247,6 +247,7 @@
#define GICR_TYPER_COMMON_LPI_AFF GENMASK_ULL(25, 24)
#define GICR_TYPER_AFFINITY GENMASK_ULL(63, 32)
+#define GICR_INVLPIR_INTID GENMASK_ULL(31, 0)
#define GICR_INVLPIR_VPEID GENMASK_ULL(47, 32)
#define GICR_INVLPIR_V GENMASK_ULL(63, 63)
The following commit has been merged into the irq/core branch of tip:
Commit-ID: 3858d4dfdfb845e51ee8b4045f61ccba2c3111ee
Gitweb: https://git.kernel.org/tip/3858d4dfdfb845e51ee8b4045f61ccba2c3111ee
Author: Marc Zyngier <[email protected]>
AuthorDate: Tue, 24 Dec 2019 11:10:37
Committer: Marc Zyngier <[email protected]>
CommitterDate: Wed, 22 Jan 2020 14:22:21
irqchip/gic-v4.1: Suppress per-VLPI doorbell
Since GICv4.1 gives us a per-VPE doorbell, avoid programming anything
else on VMOVI/VMAPI/VMAPTI and on any other action that would have
otherwise resulted in a per-VLPI doorbell to be programmed.
Signed-off-by: Marc Zyngier <[email protected]>
Reviewed-by: Zenghui Yu <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
---
drivers/irqchip/irq-gic-v3-its.c | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 1d8d96a..53e91c9 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -718,7 +718,7 @@ static struct its_vpe *its_build_vmapti_cmd(struct its_node *its,
{
u32 db;
- if (desc->its_vmapti_cmd.db_enabled)
+ if (!is_v4_1(its) && desc->its_vmapti_cmd.db_enabled)
db = desc->its_vmapti_cmd.vpe->vpe_db_lpi;
else
db = 1023;
@@ -741,7 +741,7 @@ static struct its_vpe *its_build_vmovi_cmd(struct its_node *its,
{
u32 db;
- if (desc->its_vmovi_cmd.db_enabled)
+ if (!is_v4_1(its) && desc->its_vmovi_cmd.db_enabled)
db = desc->its_vmovi_cmd.vpe->vpe_db_lpi;
else
db = 1023;
@@ -1353,6 +1353,13 @@ static void its_vlpi_set_doorbell(struct irq_data *d, bool enable)
u32 event = its_get_event_id(d);
struct its_vlpi_map *map;
+ /*
+ * GICv4.1 does away with the per-LPI nonsense, nothing to do
+ * here.
+ */
+ if (is_v4_1(its_dev->its))
+ return;
+
map = dev_event_to_vlpi_map(its_dev, event);
if (map->db_enabled == enable)
The following commit has been merged into the irq/core branch of tip:
Commit-ID: b25319d279b63781b972c4966b4082193e69afac
Gitweb: https://git.kernel.org/tip/b25319d279b63781b972c4966b4082193e69afac
Author: Marc Zyngier <[email protected]>
AuthorDate: Tue, 24 Dec 2019 11:10:24
Committer: Marc Zyngier <[email protected]>
CommitterDate: Wed, 22 Jan 2020 14:22:19
irqchip/gic-v3: Detect GICv4.1 supporting RVPEID
GICv4.1 supports the RVPEID ("Residency per vPE ID"), which allows for
a much efficient way of making virtual CPUs resident (to allow direct
injection of interrupts).
The functionnality needs to be discovered on each and every redistributor
in the system, and disabled if the settings are inconsistent.
Signed-off-by: Marc Zyngier <[email protected]>
Reviewed-by: Zenghui Yu <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
---
drivers/irqchip/irq-gic-v3.c | 21 ++++++++++++++++++---
include/linux/irqchip/arm-gic-v3.h | 2 ++
2 files changed, 20 insertions(+), 3 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index d621801..ffcb018 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -858,8 +858,21 @@ static int __gic_update_rdist_properties(struct redist_region *region,
void __iomem *ptr)
{
u64 typer = gic_read_typer(ptr + GICR_TYPER);
+
gic_data.rdists.has_vlpis &= !!(typer & GICR_TYPER_VLPIS);
- gic_data.rdists.has_direct_lpi &= !!(typer & GICR_TYPER_DirectLPIS);
+
+ /* RVPEID implies some form of DirectLPI, no matter what the doc says... :-/ */
+ gic_data.rdists.has_rvpeid &= !!(typer & GICR_TYPER_RVPEID);
+ gic_data.rdists.has_direct_lpi &= (!!(typer & GICR_TYPER_DirectLPIS) |
+ gic_data.rdists.has_rvpeid);
+
+ /* Detect non-sensical configurations */
+ if (WARN_ON_ONCE(gic_data.rdists.has_rvpeid && !gic_data.rdists.has_vlpis)) {
+ gic_data.rdists.has_direct_lpi = false;
+ gic_data.rdists.has_vlpis = false;
+ gic_data.rdists.has_rvpeid = false;
+ }
+
gic_data.ppi_nr = min(GICR_TYPER_NR_PPIS(typer), gic_data.ppi_nr);
return 1;
@@ -872,9 +885,10 @@ static void gic_update_rdist_properties(void)
if (WARN_ON(gic_data.ppi_nr == UINT_MAX))
gic_data.ppi_nr = 0;
pr_info("%d PPIs implemented\n", gic_data.ppi_nr);
- pr_info("%sVLPI support, %sdirect LPI support\n",
+ pr_info("%sVLPI support, %sdirect LPI support, %sRVPEID support\n",
!gic_data.rdists.has_vlpis ? "no " : "",
- !gic_data.rdists.has_direct_lpi ? "no " : "");
+ !gic_data.rdists.has_direct_lpi ? "no " : "",
+ !gic_data.rdists.has_rvpeid ? "no " : "");
}
/* Check whether it's single security state view */
@@ -1566,6 +1580,7 @@ static int __init gic_init_bases(void __iomem *dist_base,
&gic_data);
irq_domain_update_bus_token(gic_data.domain, DOMAIN_BUS_WIRED);
gic_data.rdists.rdist = alloc_percpu(typeof(*gic_data.rdists.rdist));
+ gic_data.rdists.has_rvpeid = true;
gic_data.rdists.has_vlpis = true;
gic_data.rdists.has_direct_lpi = true;
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index de991d6..9a5f85d 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -234,6 +234,7 @@
#define GICR_TYPER_VLPIS (1U << 1)
#define GICR_TYPER_DirectLPIS (1U << 3)
#define GICR_TYPER_LAST (1U << 4)
+#define GICR_TYPER_RVPEID (1U << 7)
#define GIC_V3_REDIST_SIZE 0x20000
@@ -615,6 +616,7 @@ struct rdists {
u64 flags;
u32 gicd_typer;
bool has_vlpis;
+ bool has_rvpeid;
bool has_direct_lpi;
};
The following commit has been merged into the irq/core branch of tip:
Commit-ID: 5e5168461c22c8738d31d4ee12a5cbc2ab0aa440
Gitweb: https://git.kernel.org/tip/5e5168461c22c8738d31d4ee12a5cbc2ab0aa440
Author: Marc Zyngier <[email protected]>
AuthorDate: Tue, 24 Dec 2019 11:10:28
Committer: Marc Zyngier <[email protected]>
CommitterDate: Wed, 22 Jan 2020 14:22:19
irqchip/gic-v4.1: VPE table (aka GICR_VPROPBASER) allocation
GICv4.1 defines a new VPE table that is potentially shared between
both the ITSs and the redistributors, following complicated affinity
rules.
To make things more confusing, the programming of this table at
the redistributor level is reusing the GICv4.0 GICR_VPROPBASER register
for something completely different.
The code flow is somewhat complexified by the need to respect the
affinities required by the HW, meaning that tables can either be
inherited from a previously discovered ITS or redistributor.
Signed-off-by: Marc Zyngier <[email protected]>
Reviewed-by: Zenghui Yu <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
---
arch/arm/include/asm/arch_gicv3.h | 2 +-
arch/arm64/include/asm/arch_gicv3.h | 1 +-
drivers/irqchip/irq-gic-v3-its.c | 310 ++++++++++++++++++++++++++-
include/linux/irqchip/arm-gic-v3.h | 33 ++-
4 files changed, 339 insertions(+), 7 deletions(-)
diff --git a/arch/arm/include/asm/arch_gicv3.h b/arch/arm/include/asm/arch_gicv3.h
index fa50bb0..b5752f0 100644
--- a/arch/arm/include/asm/arch_gicv3.h
+++ b/arch/arm/include/asm/arch_gicv3.h
@@ -10,6 +10,7 @@
#ifndef __ASSEMBLY__
#include <linux/io.h>
+#include <linux/io-64-nonatomic-lo-hi.h>
#include <asm/barrier.h>
#include <asm/cacheflush.h>
#include <asm/cp15.h>
@@ -327,6 +328,7 @@ static inline u64 __gic_readq_nonatomic(const volatile void __iomem *addr)
/*
* GITS_VPROPBASER - hi and lo bits may be accessed independently.
*/
+#define gits_read_vpropbaser(c) __gic_readq_nonatomic(c)
#define gits_write_vpropbaser(v, c) __gic_writeq_nonatomic(v, c)
/*
diff --git a/arch/arm64/include/asm/arch_gicv3.h b/arch/arm64/include/asm/arch_gicv3.h
index 89e4c8b..4750fc8 100644
--- a/arch/arm64/include/asm/arch_gicv3.h
+++ b/arch/arm64/include/asm/arch_gicv3.h
@@ -141,6 +141,7 @@ static inline u32 gic_read_rpr(void)
#define gicr_read_pendbaser(c) readq_relaxed(c)
#define gits_write_vpropbaser(v, c) writeq_relaxed(v, c)
+#define gits_read_vpropbaser(c) readq_relaxed(c)
#define gits_write_vpendbaser(v, c) writeq_relaxed(v, c)
#define gits_read_vpendbaser(c) readq_relaxed(c)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 6b0a8d6..9124f7f 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -106,6 +106,7 @@ struct its_node {
u64 typer;
u64 cbaser_save;
u32 ctlr_save;
+ u32 mpidr;
struct list_head its_device_list;
u64 flags;
unsigned long list_nr;
@@ -116,6 +117,7 @@ struct its_node {
};
#define is_v4(its) (!!((its)->typer & GITS_TYPER_VLPIS))
+#define is_v4_1(its) (!!((its)->typer & GITS_TYPER_VMAPP))
#define device_ids(its) (FIELD_GET(GITS_TYPER_DEVBITS, (its)->typer) + 1)
#define ITS_ITT_ALIGN SZ_256
@@ -2100,6 +2102,65 @@ static bool its_parse_indirect_baser(struct its_node *its,
return indirect;
}
+static u32 compute_common_aff(u64 val)
+{
+ u32 aff, clpiaff;
+
+ aff = FIELD_GET(GICR_TYPER_AFFINITY, val);
+ clpiaff = FIELD_GET(GICR_TYPER_COMMON_LPI_AFF, val);
+
+ return aff & ~(GENMASK(31, 0) >> (clpiaff * 8));
+}
+
+static u32 compute_its_aff(struct its_node *its)
+{
+ u64 val;
+ u32 svpet;
+
+ /*
+ * Reencode the ITS SVPET and MPIDR as a GICR_TYPER, and compute
+ * the resulting affinity. We then use that to see if this match
+ * our own affinity.
+ */
+ svpet = FIELD_GET(GITS_TYPER_SVPET, its->typer);
+ val = FIELD_PREP(GICR_TYPER_COMMON_LPI_AFF, svpet);
+ val |= FIELD_PREP(GICR_TYPER_AFFINITY, its->mpidr);
+ return compute_common_aff(val);
+}
+
+static struct its_node *find_sibling_its(struct its_node *cur_its)
+{
+ struct its_node *its;
+ u32 aff;
+
+ if (!FIELD_GET(GITS_TYPER_SVPET, cur_its->typer))
+ return NULL;
+
+ aff = compute_its_aff(cur_its);
+
+ list_for_each_entry(its, &its_nodes, entry) {
+ u64 baser;
+
+ if (!is_v4_1(its) || its == cur_its)
+ continue;
+
+ if (!FIELD_GET(GITS_TYPER_SVPET, its->typer))
+ continue;
+
+ if (aff != compute_its_aff(its))
+ continue;
+
+ /* GICv4.1 guarantees that the vPE table is GITS_BASER2 */
+ baser = its->tables[2].val;
+ if (!(baser & GITS_BASER_VALID))
+ continue;
+
+ return its;
+ }
+
+ return NULL;
+}
+
static void its_free_tables(struct its_node *its)
{
int i;
@@ -2142,6 +2203,17 @@ static int its_alloc_tables(struct its_node *its)
break;
case GITS_BASER_TYPE_VCPU:
+ if (is_v4_1(its)) {
+ struct its_node *sibling;
+
+ WARN_ON(i != 2);
+ if ((sibling = find_sibling_its(its))) {
+ *baser = sibling->tables[2];
+ its_write_baser(its, baser, baser->val);
+ continue;
+ }
+ }
+
indirect = its_parse_indirect_baser(its, baser,
psz, &order,
ITS_MAX_VPEID_BITS);
@@ -2163,6 +2235,220 @@ static int its_alloc_tables(struct its_node *its)
return 0;
}
+static u64 inherit_vpe_l1_table_from_its(void)
+{
+ struct its_node *its;
+ u64 val;
+ u32 aff;
+
+ val = gic_read_typer(gic_data_rdist_rd_base() + GICR_TYPER);
+ aff = compute_common_aff(val);
+
+ list_for_each_entry(its, &its_nodes, entry) {
+ u64 baser, addr;
+
+ if (!is_v4_1(its))
+ continue;
+
+ if (!FIELD_GET(GITS_TYPER_SVPET, its->typer))
+ continue;
+
+ if (aff != compute_its_aff(its))
+ continue;
+
+ /* GICv4.1 guarantees that the vPE table is GITS_BASER2 */
+ baser = its->tables[2].val;
+ if (!(baser & GITS_BASER_VALID))
+ continue;
+
+ /* We have a winner! */
+ val = GICR_VPROPBASER_4_1_VALID;
+ if (baser & GITS_BASER_INDIRECT)
+ val |= GICR_VPROPBASER_4_1_INDIRECT;
+ val |= FIELD_PREP(GICR_VPROPBASER_4_1_PAGE_SIZE,
+ FIELD_GET(GITS_BASER_PAGE_SIZE_MASK, baser));
+ switch (FIELD_GET(GITS_BASER_PAGE_SIZE_MASK, baser)) {
+ case GIC_PAGE_SIZE_64K:
+ addr = GITS_BASER_ADDR_48_to_52(baser);
+ break;
+ default:
+ addr = baser & GENMASK_ULL(47, 12);
+ break;
+ }
+ val |= FIELD_PREP(GICR_VPROPBASER_4_1_ADDR, addr >> 12);
+ val |= FIELD_PREP(GICR_VPROPBASER_SHAREABILITY_MASK,
+ FIELD_GET(GITS_BASER_SHAREABILITY_MASK, baser));
+ val |= FIELD_PREP(GICR_VPROPBASER_INNER_CACHEABILITY_MASK,
+ FIELD_GET(GITS_BASER_INNER_CACHEABILITY_MASK, baser));
+ val |= FIELD_PREP(GICR_VPROPBASER_4_1_SIZE, GITS_BASER_NR_PAGES(baser) - 1);
+
+ return val;
+ }
+
+ return 0;
+}
+
+static u64 inherit_vpe_l1_table_from_rd(cpumask_t **mask)
+{
+ u32 aff;
+ u64 val;
+ int cpu;
+
+ val = gic_read_typer(gic_data_rdist_rd_base() + GICR_TYPER);
+ aff = compute_common_aff(val);
+
+ for_each_possible_cpu(cpu) {
+ void __iomem *base = gic_data_rdist_cpu(cpu)->rd_base;
+ u32 tmp;
+
+ if (!base || cpu == smp_processor_id())
+ continue;
+
+ val = gic_read_typer(base + GICR_TYPER);
+ tmp = compute_common_aff(val);
+ if (tmp != aff)
+ continue;
+
+ /*
+ * At this point, we have a victim. This particular CPU
+ * has already booted, and has an affinity that matches
+ * ours wrt CommonLPIAff. Let's use its own VPROPBASER.
+ * Make sure we don't write the Z bit in that case.
+ */
+ val = gits_read_vpropbaser(base + SZ_128K + GICR_VPROPBASER);
+ val &= ~GICR_VPROPBASER_4_1_Z;
+
+ *mask = gic_data_rdist_cpu(cpu)->vpe_table_mask;
+
+ return val;
+ }
+
+ return 0;
+}
+
+static int allocate_vpe_l1_table(void)
+{
+ void __iomem *vlpi_base = gic_data_rdist_vlpi_base();
+ u64 val, gpsz, npg, pa;
+ unsigned int psz = SZ_64K;
+ unsigned int np, epp, esz;
+ struct page *page;
+
+ if (!gic_rdists->has_rvpeid)
+ return 0;
+
+ /*
+ * if VPENDBASER.Valid is set, disable any previously programmed
+ * VPE by setting PendingLast while clearing Valid. This has the
+ * effect of making sure no doorbell will be generated and we can
+ * then safely clear VPROPBASER.Valid.
+ */
+ if (gits_read_vpendbaser(vlpi_base + GICR_VPENDBASER) & GICR_VPENDBASER_Valid)
+ gits_write_vpendbaser(GICR_VPENDBASER_PendingLast,
+ vlpi_base + GICR_VPENDBASER);
+
+ /*
+ * If we can inherit the configuration from another RD, let's do
+ * so. Otherwise, we have to go through the allocation process. We
+ * assume that all RDs have the exact same requirements, as
+ * nothing will work otherwise.
+ */
+ val = inherit_vpe_l1_table_from_rd(&gic_data_rdist()->vpe_table_mask);
+ if (val & GICR_VPROPBASER_4_1_VALID)
+ goto out;
+
+ gic_data_rdist()->vpe_table_mask = kzalloc(sizeof(cpumask_t), GFP_KERNEL);
+ if (!gic_data_rdist()->vpe_table_mask)
+ return -ENOMEM;
+
+ val = inherit_vpe_l1_table_from_its();
+ if (val & GICR_VPROPBASER_4_1_VALID)
+ goto out;
+
+ /* First probe the page size */
+ val = FIELD_PREP(GICR_VPROPBASER_4_1_PAGE_SIZE, GIC_PAGE_SIZE_64K);
+ gits_write_vpropbaser(val, vlpi_base + GICR_VPROPBASER);
+ val = gits_read_vpropbaser(vlpi_base + GICR_VPROPBASER);
+ gpsz = FIELD_GET(GICR_VPROPBASER_4_1_PAGE_SIZE, val);
+ esz = FIELD_GET(GICR_VPROPBASER_4_1_ENTRY_SIZE, val);
+
+ switch (gpsz) {
+ default:
+ gpsz = GIC_PAGE_SIZE_4K;
+ /* fall through */
+ case GIC_PAGE_SIZE_4K:
+ psz = SZ_4K;
+ break;
+ case GIC_PAGE_SIZE_16K:
+ psz = SZ_16K;
+ break;
+ case GIC_PAGE_SIZE_64K:
+ psz = SZ_64K;
+ break;
+ }
+
+ /*
+ * Start populating the register from scratch, including RO fields
+ * (which we want to print in debug cases...)
+ */
+ val = 0;
+ val |= FIELD_PREP(GICR_VPROPBASER_4_1_PAGE_SIZE, gpsz);
+ val |= FIELD_PREP(GICR_VPROPBASER_4_1_ENTRY_SIZE, esz);
+
+ /* How many entries per GIC page? */
+ esz++;
+ epp = psz / (esz * SZ_8);
+
+ /*
+ * If we need more than just a single L1 page, flag the table
+ * as indirect and compute the number of required L1 pages.
+ */
+ if (epp < ITS_MAX_VPEID) {
+ int nl2;
+
+ val |= GICR_VPROPBASER_4_1_INDIRECT;
+
+ /* Number of L2 pages required to cover the VPEID space */
+ nl2 = DIV_ROUND_UP(ITS_MAX_VPEID, epp);
+
+ /* Number of L1 pages to point to the L2 pages */
+ npg = DIV_ROUND_UP(nl2 * SZ_8, psz);
+ } else {
+ npg = 1;
+ }
+
+ val |= FIELD_PREP(GICR_VPROPBASER_4_1_SIZE, npg);
+
+ /* Right, that's the number of CPU pages we need for L1 */
+ np = DIV_ROUND_UP(npg * psz, PAGE_SIZE);
+
+ pr_debug("np = %d, npg = %lld, psz = %d, epp = %d, esz = %d\n",
+ np, npg, psz, epp, esz);
+ page = alloc_pages(GFP_KERNEL | __GFP_ZERO, get_order(np * PAGE_SIZE));
+ if (!page)
+ return -ENOMEM;
+
+ gic_data_rdist()->vpe_l1_page = page;
+ pa = virt_to_phys(page_address(page));
+ WARN_ON(!IS_ALIGNED(pa, psz));
+
+ val |= FIELD_PREP(GICR_VPROPBASER_4_1_ADDR, pa >> 12);
+ val |= GICR_VPROPBASER_RaWb;
+ val |= GICR_VPROPBASER_InnerShareable;
+ val |= GICR_VPROPBASER_4_1_Z;
+ val |= GICR_VPROPBASER_4_1_VALID;
+
+out:
+ gits_write_vpropbaser(val, vlpi_base + GICR_VPROPBASER);
+ cpumask_set_cpu(smp_processor_id(), gic_data_rdist()->vpe_table_mask);
+
+ pr_debug("CPU%d: VPROPBASER = %llx %*pbl\n",
+ smp_processor_id(), val,
+ cpumask_pr_args(gic_data_rdist()->vpe_table_mask));
+
+ return 0;
+}
+
static int its_alloc_collections(struct its_node *its)
{
int i;
@@ -2362,7 +2648,7 @@ static void its_cpu_init_lpis(void)
val |= GICR_CTLR_ENABLE_LPIS;
writel_relaxed(val, rbase + GICR_CTLR);
- if (gic_rdists->has_vlpis) {
+ if (gic_rdists->has_vlpis && !gic_rdists->has_rvpeid) {
void __iomem *vlpi_base = gic_data_rdist_vlpi_base();
/*
@@ -2386,6 +2672,16 @@ static void its_cpu_init_lpis(void)
WARN_ON(val & GICR_VPENDBASER_Dirty);
}
+ if (allocate_vpe_l1_table()) {
+ /*
+ * If the allocation has failed, we're in massive trouble.
+ * Disable direct injection, and pray that no VM was
+ * already running...
+ */
+ gic_rdists->has_rvpeid = false;
+ gic_rdists->has_vlpis = false;
+ }
+
/* Make sure the GIC has seen the above */
dsb(sy);
out:
@@ -3788,6 +4084,14 @@ static int __init its_probe_one(struct resource *res,
} else {
pr_info("ITS@%pa: Single VMOVP capable\n", &res->start);
}
+
+ if (is_v4_1(its)) {
+ u32 svpet = FIELD_GET(GITS_TYPER_SVPET, typer);
+ its->mpidr = readl_relaxed(its_base + GITS_MPIDR);
+
+ pr_info("ITS@%pa: Using GICv4.1 mode %08x %08x\n",
+ &res->start, its->mpidr, svpet);
+ }
}
its->numa_node = numa_node;
@@ -4148,6 +4452,8 @@ int __init its_init(struct fwnode_handle *handle, struct rdists *rdists,
bool has_v4 = false;
int err;
+ gic_rdists = rdists;
+
its_parent = parent_domain;
of_node = to_of_node(handle);
if (of_node)
@@ -4160,8 +4466,6 @@ int __init its_init(struct fwnode_handle *handle, struct rdists *rdists,
return -ENXIO;
}
- gic_rdists = rdists;
-
err = allocate_lpi_tables();
if (err)
return err;
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index 9dfe641..db0a111 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -102,6 +102,11 @@
#define GIC_V3_DIST_SIZE 0x10000
+#define GIC_PAGE_SIZE_4K 0ULL
+#define GIC_PAGE_SIZE_16K 1ULL
+#define GIC_PAGE_SIZE_64K 2ULL
+#define GIC_PAGE_SIZE_MASK 3ULL
+
/*
* Re-Distributor registers, offsets from RD_base
*/
@@ -239,6 +244,8 @@
#define GICR_TYPER_DirectLPIS (1U << 3)
#define GICR_TYPER_LAST (1U << 4)
#define GICR_TYPER_RVPEID (1U << 7)
+#define GICR_TYPER_COMMON_LPI_AFF GENMASK_ULL(25, 24)
+#define GICR_TYPER_AFFINITY GENMASK_ULL(63, 32)
#define GIC_V3_REDIST_SIZE 0x20000
@@ -277,6 +284,18 @@
#define GICR_VPROPBASER_RaWaWt GIC_BASER_CACHEABILITY(GICR_VPROPBASER, INNER, RaWaWt)
#define GICR_VPROPBASER_RaWaWb GIC_BASER_CACHEABILITY(GICR_VPROPBASER, INNER, RaWaWb)
+/*
+ * GICv4.1 VPROPBASER reinvention. A subtle mix between the old
+ * VPROPBASER and ITS_BASER. Just not quite any of the two.
+ */
+#define GICR_VPROPBASER_4_1_VALID (1ULL << 63)
+#define GICR_VPROPBASER_4_1_ENTRY_SIZE GENMASK_ULL(61, 59)
+#define GICR_VPROPBASER_4_1_INDIRECT (1ULL << 55)
+#define GICR_VPROPBASER_4_1_PAGE_SIZE GENMASK_ULL(54, 53)
+#define GICR_VPROPBASER_4_1_Z (1ULL << 52)
+#define GICR_VPROPBASER_4_1_ADDR GENMASK_ULL(51, 12)
+#define GICR_VPROPBASER_4_1_SIZE GENMASK_ULL(6, 0)
+
#define GICR_VPENDBASER 0x0078
#define GICR_VPENDBASER_SHAREABILITY_SHIFT (10)
@@ -314,6 +333,7 @@
#define GITS_CTLR 0x0000
#define GITS_IIDR 0x0004
#define GITS_TYPER 0x0008
+#define GITS_MPIDR 0x0018
#define GITS_CBASER 0x0080
#define GITS_CWRITER 0x0088
#define GITS_CREADR 0x0090
@@ -347,6 +367,8 @@
#define GITS_TYPER_HCC_SHIFT 24
#define GITS_TYPER_HCC(r) (((r) >> GITS_TYPER_HCC_SHIFT) & 0xff)
#define GITS_TYPER_VMOVP (1ULL << 37)
+#define GITS_TYPER_VMAPP (1ULL << 40)
+#define GITS_TYPER_SVPET GENMASK_ULL(42, 41)
#define GITS_IIDR_REV_SHIFT 12
#define GITS_IIDR_REV_MASK (0xf << GITS_IIDR_REV_SHIFT)
@@ -417,10 +439,11 @@
#define GITS_BASER_InnerShareable \
GIC_BASER_SHAREABILITY(GITS_BASER, InnerShareable)
#define GITS_BASER_PAGE_SIZE_SHIFT (8)
-#define GITS_BASER_PAGE_SIZE_4K (0ULL << GITS_BASER_PAGE_SIZE_SHIFT)
-#define GITS_BASER_PAGE_SIZE_16K (1ULL << GITS_BASER_PAGE_SIZE_SHIFT)
-#define GITS_BASER_PAGE_SIZE_64K (2ULL << GITS_BASER_PAGE_SIZE_SHIFT)
-#define GITS_BASER_PAGE_SIZE_MASK (3ULL << GITS_BASER_PAGE_SIZE_SHIFT)
+#define __GITS_BASER_PSZ(sz) (GIC_PAGE_SIZE_ ## sz << GITS_BASER_PAGE_SIZE_SHIFT)
+#define GITS_BASER_PAGE_SIZE_4K __GITS_BASER_PSZ(4K)
+#define GITS_BASER_PAGE_SIZE_16K __GITS_BASER_PSZ(16K)
+#define GITS_BASER_PAGE_SIZE_64K __GITS_BASER_PSZ(64K)
+#define GITS_BASER_PAGE_SIZE_MASK __GITS_BASER_PSZ(MASK)
#define GITS_BASER_PAGES_MAX 256
#define GITS_BASER_PAGES_SHIFT (0)
#define GITS_BASER_NR_PAGES(r) (((r) & 0xff) + 1)
@@ -612,8 +635,10 @@ struct rdists {
struct {
void __iomem *rd_base;
struct page *pend_page;
+ struct page *vpe_l1_page;
phys_addr_t phys_base;
bool lpi_enabled;
+ cpumask_t *vpe_table_mask;
} __percpu *rdist;
phys_addr_t prop_table_pa;
void *prop_table_va;
The following commit has been merged into the irq/core branch of tip:
Commit-ID: 64edfaa9a2342a3ce34f8cb982c2c2df84db4de3
Gitweb: https://git.kernel.org/tip/64edfaa9a2342a3ce34f8cb982c2c2df84db4de3
Author: Marc Zyngier <[email protected]>
AuthorDate: Tue, 24 Dec 2019 11:10:29
Committer: Marc Zyngier <[email protected]>
CommitterDate: Wed, 22 Jan 2020 14:22:19
irqchip/gic-v4.1: Implement the v4.1 flavour of VMAPP
The ITS VMAPP command gains some new fields with GICv4.1:
- a default doorbell, which allows a single doorbell to be used for
all the VLPIs routed to a given VPE
- a pointer to the configuration table (instead of having it in a register
that gets context switched)
- a flag indicating whether this is the first map or the last unmap for
this particular VPE
- a flag indicating whether the pending table is known to be zeroed, or not
Plumb in the new fields in the VMAPP builder, and add the map/unmap
refcounting so that the ITS can do the right thing.
Signed-off-by: Marc Zyngier <[email protected]>
Reviewed-by: Zenghui Yu <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
---
drivers/irqchip/irq-gic-v3-its.c | 60 ++++++++++++++++++++++++++---
include/linux/irqchip/arm-gic-v4.h | 18 +++++++--
2 files changed, 69 insertions(+), 9 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 9124f7f..fbf4ca7 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -449,6 +449,27 @@ static void its_encode_vpt_size(struct its_cmd_block *cmd, u8 vpt_size)
its_mask_encode(&cmd->raw_cmd[3], vpt_size, 4, 0);
}
+static void its_encode_vconf_addr(struct its_cmd_block *cmd, u64 vconf_pa)
+{
+ its_mask_encode(&cmd->raw_cmd[0], vconf_pa >> 16, 51, 16);
+}
+
+static void its_encode_alloc(struct its_cmd_block *cmd, bool alloc)
+{
+ its_mask_encode(&cmd->raw_cmd[0], alloc, 8, 8);
+}
+
+static void its_encode_ptz(struct its_cmd_block *cmd, bool ptz)
+{
+ its_mask_encode(&cmd->raw_cmd[0], ptz, 9, 9);
+}
+
+static void its_encode_vmapp_default_db(struct its_cmd_block *cmd,
+ u32 vpe_db_lpi)
+{
+ its_mask_encode(&cmd->raw_cmd[1], vpe_db_lpi, 31, 0);
+}
+
static inline void its_fixup_cmd(struct its_cmd_block *cmd)
{
/* Let's fixup BE commands */
@@ -632,19 +653,45 @@ static struct its_vpe *its_build_vmapp_cmd(struct its_node *its,
struct its_cmd_block *cmd,
struct its_cmd_desc *desc)
{
- unsigned long vpt_addr;
+ unsigned long vpt_addr, vconf_addr;
u64 target;
-
- vpt_addr = virt_to_phys(page_address(desc->its_vmapp_cmd.vpe->vpt_page));
- target = desc->its_vmapp_cmd.col->target_address + its->vlpi_redist_offset;
+ bool alloc;
its_encode_cmd(cmd, GITS_CMD_VMAPP);
its_encode_vpeid(cmd, desc->its_vmapp_cmd.vpe->vpe_id);
its_encode_valid(cmd, desc->its_vmapp_cmd.valid);
+
+ if (!desc->its_vmapp_cmd.valid) {
+ if (is_v4_1(its)) {
+ alloc = !atomic_dec_return(&desc->its_vmapp_cmd.vpe->vmapp_count);
+ its_encode_alloc(cmd, alloc);
+ }
+
+ goto out;
+ }
+
+ vpt_addr = virt_to_phys(page_address(desc->its_vmapp_cmd.vpe->vpt_page));
+ target = desc->its_vmapp_cmd.col->target_address + its->vlpi_redist_offset;
+
its_encode_target(cmd, target);
its_encode_vpt_addr(cmd, vpt_addr);
its_encode_vpt_size(cmd, LPI_NRBITS - 1);
+ if (!is_v4_1(its))
+ goto out;
+
+ vconf_addr = virt_to_phys(page_address(desc->its_vmapp_cmd.vpe->its_vm->vprop_page));
+
+ alloc = !atomic_fetch_inc(&desc->its_vmapp_cmd.vpe->vmapp_count);
+
+ its_encode_alloc(cmd, alloc);
+
+ /* We can only signal PTZ when alloc==1. Why do we have two bits? */
+ its_encode_ptz(cmd, alloc);
+ its_encode_vconf_addr(cmd, vconf_addr);
+ its_encode_vmapp_default_db(cmd, desc->its_vmapp_cmd.vpe->vpe_db_lpi);
+
+out:
its_fixup_cmd(cmd);
return valid_vpe(its, desc->its_vmapp_cmd.vpe);
@@ -3492,7 +3539,10 @@ static int its_vpe_init(struct its_vpe *vpe)
vpe->vpe_id = vpe_id;
vpe->vpt_page = vpt_page;
- vpe->vpe_proxy_event = -1;
+ if (gic_rdists->has_rvpeid)
+ atomic_set(&vpe->vmapp_count, 0);
+ else
+ vpe->vpe_proxy_event = -1;
return 0;
}
diff --git a/include/linux/irqchip/arm-gic-v4.h b/include/linux/irqchip/arm-gic-v4.h
index 5dbcfc6..498e523 100644
--- a/include/linux/irqchip/arm-gic-v4.h
+++ b/include/linux/irqchip/arm-gic-v4.h
@@ -39,8 +39,20 @@ struct its_vpe {
irq_hw_number_t vpe_db_lpi;
/* VPE resident */
bool resident;
- /* VPE proxy mapping */
- int vpe_proxy_event;
+ union {
+ /* GICv4.0 implementations */
+ struct {
+ /* VPE proxy mapping */
+ int vpe_proxy_event;
+ /* Implementation Defined Area Invalid */
+ bool idai;
+ };
+ /* GICv4.1 implementations */
+ struct {
+ atomic_t vmapp_count;
+ };
+ };
+
/*
* This collection ID is used to indirect the target
* redistributor for this VPE. The ID itself isn't involved in
@@ -49,8 +61,6 @@ struct its_vpe {
u16 col_idx;
/* Unique (system-wide) VPE identifier */
u16 vpe_id;
- /* Implementation Defined Area Invalid */
- bool idai;
/* Pending VLPIs on schedule out? */
bool pending_last;
};
On 24.12.19 11:10:26, Marc Zyngier wrote:
> Despite the architecture spec being extremely clear about the fact
> that reserved registers in the GIC distributor memory map are RES0
> (and thus are not allowed to generate an exception), the Cavium
> ThunderX (aka TX1) SoC explodes as such:
>
> [ 0.000000] GICv3: GIC: Using split EOI/Deactivate mode
> [ 0.000000] GICv3: 128 SPIs implemented
> [ 0.000000] GICv3: 0 Extended SPIs implemented
> [ 0.000000] Internal error: synchronous external abort: 96000210 [#1] SMP
> [ 0.000000] Modules linked in:
> [ 0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.4.0-rc4-00035-g3cf6a3d5725f #7956
> [ 0.000000] Hardware name: cavium,thunder-88xx (DT)
> [ 0.000000] pstate: 60000085 (nZCv daIf -PAN -UAO)
> [ 0.000000] pc : __raw_readl+0x0/0x8
> [ 0.000000] lr : gic_init_bases+0x110/0x560
> [ 0.000000] sp : ffff800011243d90
> [ 0.000000] x29: ffff800011243d90 x28: 0000000000000000
> [ 0.000000] x27: 0000000000000018 x26: 0000000000000002
> [ 0.000000] x25: ffff8000116f0000 x24: ffff000fbe6a2c80
> [ 0.000000] x23: 0000000000000000 x22: ffff010fdc322b68
> [ 0.000000] x21: ffff800010a7a208 x20: 00000000009b0404
> [ 0.000000] x19: ffff80001124dad0 x18: 0000000000000010
> [ 0.000000] x17: 000000004d8d492b x16: 00000000f67eb9af
> [ 0.000000] x15: ffffffffffffffff x14: ffff800011249908
> [ 0.000000] x13: ffff800091243ae7 x12: ffff800011243af4
> [ 0.000000] x11: ffff80001126e000 x10: ffff800011243a70
> [ 0.000000] x9 : 00000000ffffffd0 x8 : ffff80001069c828
> [ 0.000000] x7 : 0000000000000059 x6 : ffff8000113fb4d1
> [ 0.000000] x5 : 0000000000000001 x4 : 0000000000000000
> [ 0.000000] x3 : 0000000000000000 x2 : 0000000000000000
> [ 0.000000] x1 : 0000000000000000 x0 : ffff8000116f000c
> [ 0.000000] Call trace:
> [ 0.000000] __raw_readl+0x0/0x8
> [ 0.000000] gic_of_init+0x188/0x224
> [ 0.000000] of_irq_init+0x200/0x3cc
> [ 0.000000] irqchip_init+0x1c/0x40
> [ 0.000000] init_IRQ+0x160/0x1d0
> [ 0.000000] start_kernel+0x2ec/0x4b8
> [ 0.000000] Code: a8c47bfd d65f03c0 d538d080 d65f03c0 (b9400000)
>
> when reading the GICv4.1 GICD_TYPER2 register, which is unexpected...
>
> Work around it by adding a new quirk flagging all the A1 revisions
> of the distributor, but it remains unknown whether this could affect
> other revisions of this SoC (or even other SoCs from the same silicon
> vendor).
>
> Signed-off-by: Marc Zyngier <[email protected]>
> ---
> drivers/irqchip/irq-gic-v3.c | 23 ++++++++++++++++++++++-
> 1 file changed, 22 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
> index 286f98222878..640d4db65b78 100644
> --- a/drivers/irqchip/irq-gic-v3.c
> +++ b/drivers/irqchip/irq-gic-v3.c
> @@ -34,6 +34,7 @@
> #define GICD_INT_NMI_PRI (GICD_INT_DEF_PRI & ~0x80)
>
> #define FLAGS_WORKAROUND_GICR_WAKER_MSM8996 (1ULL << 0)
> +#define FLAGS_WORKAROUND_GICD_TYPER2_TX1 (1ULL << 1)
>
> struct redist_region {
> void __iomem *redist_base;
> @@ -1464,6 +1465,15 @@ static bool gic_enable_quirk_msm8996(void *data)
> return true;
> }
>
> +static bool gic_enable_quirk_tx1(void *data)
> +{
> + struct gic_chip_data *d = data;
> +
> + d->flags |= FLAGS_WORKAROUND_GICD_TYPER2_TX1;
> +
> + return true;
> +}
> +
> static bool gic_enable_quirk_hip06_07(void *data)
> {
> struct gic_chip_data *d = data;
> @@ -1502,6 +1512,12 @@ static const struct gic_quirk gic_quirks[] = {
> .mask = 0xffffffff,
> .init = gic_enable_quirk_hip06_07,
> },
> + {
> + .desc = "GICv3: Cavium TX1 GICD_TYPER2 erratum",
There is no errata number yet.
> + .iidr = 0xa100034c,
> + .mask = 0xfff00fff,
> + .init = gic_enable_quirk_tx1,
All TX1 and OcteonTX parts are affected, which is a0-a7 and b0-b7. So
the iidr/mask should be:
.iidr = 0xa000034c,
.mask = 0xe8f00fff,
> + },
> {
> }
> };
> @@ -1577,7 +1593,12 @@ static int __init gic_init_bases(void __iomem *dist_base,
> pr_info("%d SPIs implemented\n", GIC_LINE_NR - 32);
> pr_info("%d Extended SPIs implemented\n", GIC_ESPI_NR);
>
> - gic_data.rdists.gicd_typer2 = readl_relaxed(gic_data.dist_base + GICD_TYPER2);
> + /*
> + * ThunderX1 explodes on reading GICD_TYPER2, in total violation
> + * of the spec (which says that reserved addresses are RES0).
> + */
> + if (!(gic_data.flags & FLAGS_WORKAROUND_GICD_TYPER2_TX1))
> + gic_data.rdists.gicd_typer2 = readl_relaxed(gic_data.dist_base + GICD_TYPER2);
You already said that checking for ArchRev of GICD_PIDR2 isn't an
option here. Though, it could...
Thanks,
-Robert
>
> gic_data.domain = irq_domain_create_tree(handle, &gic_irq_domain_ops,
> &gic_data);
> --
> 2.20.1
>
Hi Robert,
On 2020-03-09 22:11, Robert Richter wrote:
> On 24.12.19 11:10:26, Marc Zyngier wrote:
>> Despite the architecture spec being extremely clear about the fact
>> that reserved registers in the GIC distributor memory map are RES0
>> (and thus are not allowed to generate an exception), the Cavium
>> ThunderX (aka TX1) SoC explodes as such:
>>
>> [ 0.000000] GICv3: GIC: Using split EOI/Deactivate mode
>> [ 0.000000] GICv3: 128 SPIs implemented
>> [ 0.000000] GICv3: 0 Extended SPIs implemented
>> [ 0.000000] Internal error: synchronous external abort: 96000210
>> [#1] SMP
>> [ 0.000000] Modules linked in:
>> [ 0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted
>> 5.4.0-rc4-00035-g3cf6a3d5725f #7956
>> [ 0.000000] Hardware name: cavium,thunder-88xx (DT)
>> [ 0.000000] pstate: 60000085 (nZCv daIf -PAN -UAO)
>> [ 0.000000] pc : __raw_readl+0x0/0x8
>> [ 0.000000] lr : gic_init_bases+0x110/0x560
>> [ 0.000000] sp : ffff800011243d90
>> [ 0.000000] x29: ffff800011243d90 x28: 0000000000000000
>> [ 0.000000] x27: 0000000000000018 x26: 0000000000000002
>> [ 0.000000] x25: ffff8000116f0000 x24: ffff000fbe6a2c80
>> [ 0.000000] x23: 0000000000000000 x22: ffff010fdc322b68
>> [ 0.000000] x21: ffff800010a7a208 x20: 00000000009b0404
>> [ 0.000000] x19: ffff80001124dad0 x18: 0000000000000010
>> [ 0.000000] x17: 000000004d8d492b x16: 00000000f67eb9af
>> [ 0.000000] x15: ffffffffffffffff x14: ffff800011249908
>> [ 0.000000] x13: ffff800091243ae7 x12: ffff800011243af4
>> [ 0.000000] x11: ffff80001126e000 x10: ffff800011243a70
>> [ 0.000000] x9 : 00000000ffffffd0 x8 : ffff80001069c828
>> [ 0.000000] x7 : 0000000000000059 x6 : ffff8000113fb4d1
>> [ 0.000000] x5 : 0000000000000001 x4 : 0000000000000000
>> [ 0.000000] x3 : 0000000000000000 x2 : 0000000000000000
>> [ 0.000000] x1 : 0000000000000000 x0 : ffff8000116f000c
>> [ 0.000000] Call trace:
>> [ 0.000000] __raw_readl+0x0/0x8
>> [ 0.000000] gic_of_init+0x188/0x224
>> [ 0.000000] of_irq_init+0x200/0x3cc
>> [ 0.000000] irqchip_init+0x1c/0x40
>> [ 0.000000] init_IRQ+0x160/0x1d0
>> [ 0.000000] start_kernel+0x2ec/0x4b8
>> [ 0.000000] Code: a8c47bfd d65f03c0 d538d080 d65f03c0 (b9400000)
>>
>> when reading the GICv4.1 GICD_TYPER2 register, which is unexpected...
>>
>> Work around it by adding a new quirk flagging all the A1 revisions
>> of the distributor, but it remains unknown whether this could affect
>> other revisions of this SoC (or even other SoCs from the same silicon
>> vendor).
>>
>> Signed-off-by: Marc Zyngier <[email protected]>
>> ---
>> drivers/irqchip/irq-gic-v3.c | 23 ++++++++++++++++++++++-
>> 1 file changed, 22 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/irqchip/irq-gic-v3.c
>> b/drivers/irqchip/irq-gic-v3.c
>> index 286f98222878..640d4db65b78 100644
>> --- a/drivers/irqchip/irq-gic-v3.c
>> +++ b/drivers/irqchip/irq-gic-v3.c
>> @@ -34,6 +34,7 @@
>> #define GICD_INT_NMI_PRI (GICD_INT_DEF_PRI & ~0x80)
>>
>> #define FLAGS_WORKAROUND_GICR_WAKER_MSM8996 (1ULL << 0)
>> +#define FLAGS_WORKAROUND_GICD_TYPER2_TX1 (1ULL << 1)
>>
>> struct redist_region {
>> void __iomem *redist_base;
>> @@ -1464,6 +1465,15 @@ static bool gic_enable_quirk_msm8996(void
>> *data)
>> return true;
>> }
>>
>> +static bool gic_enable_quirk_tx1(void *data)
>> +{
>> + struct gic_chip_data *d = data;
>> +
>> + d->flags |= FLAGS_WORKAROUND_GICD_TYPER2_TX1;
>> +
>> + return true;
>> +}
>> +
>> static bool gic_enable_quirk_hip06_07(void *data)
>> {
>> struct gic_chip_data *d = data;
>> @@ -1502,6 +1512,12 @@ static const struct gic_quirk gic_quirks[] = {
>> .mask = 0xffffffff,
>> .init = gic_enable_quirk_hip06_07,
>> },
>> + {
>> + .desc = "GICv3: Cavium TX1 GICD_TYPER2 erratum",
>
> There is no errata number yet.
Please let me know when/if you obtain one.
>
>> + .iidr = 0xa100034c,
>> + .mask = 0xfff00fff,
>> + .init = gic_enable_quirk_tx1,
>
> All TX1 and OcteonTX parts are affected, which is a0-a7 and b0-b7. So
> the iidr/mask should be:
>
> .iidr = 0xa000034c,
> .mask = 0xe8f00fff,
Thanks, that's pretty helpful. I'll update the patch with these values
and the corresponding description.
>> + },
>> {
>> }
>> };
>> @@ -1577,7 +1593,12 @@ static int __init gic_init_bases(void __iomem
>> *dist_base,
>> pr_info("%d SPIs implemented\n", GIC_LINE_NR - 32);
>> pr_info("%d Extended SPIs implemented\n", GIC_ESPI_NR);
>>
>> - gic_data.rdists.gicd_typer2 = readl_relaxed(gic_data.dist_base +
>> GICD_TYPER2);
>> + /*
>> + * ThunderX1 explodes on reading GICD_TYPER2, in total violation
>> + * of the spec (which says that reserved addresses are RES0).
>> + */
>> + if (!(gic_data.flags & FLAGS_WORKAROUND_GICD_TYPER2_TX1))
>> + gic_data.rdists.gicd_typer2 = readl_relaxed(gic_data.dist_base +
>> GICD_TYPER2);
>
> You already said that checking for ArchRev of GICD_PIDR2 isn't an
> option here. Though, it could...
Once GICv3.2 starts using this register as well (because GICD_TYPER is
already completely full), we'd have to fix it again. There is also the
thing
you hinted at in the other thread: TX1 will generate a SEA on every
reserved
GICD registers, so we may need to protect more than just this one over
time,
and maybe more than just in the distributor.
Thanks,
M.
--
Jazz is not dead. It just smells funny...
On 10.03.20 11:41:09, Marc Zyngier wrote:
> On 2020-03-09 22:11, Robert Richter wrote:
> > On 24.12.19 11:10:26, Marc Zyngier wrote:
> > > @@ -1502,6 +1512,12 @@ static const struct gic_quirk gic_quirks[] = {
> > > .mask = 0xffffffff,
> > > .init = gic_enable_quirk_hip06_07,
> > > },
> > > + {
> > > + .desc = "GICv3: Cavium TX1 GICD_TYPER2 erratum",
> >
> > There is no errata number yet.
>
> Please let me know when/if you obtain one.
Yes, will do.
>
> >
> > > + .iidr = 0xa100034c,
> > > + .mask = 0xfff00fff,
> > > + .init = gic_enable_quirk_tx1,
> >
> > All TX1 and OcteonTX parts are affected, which is a0-a7 and b0-b7. So
> > the iidr/mask should be:
> >
> > .iidr = 0xa000034c,
> > .mask = 0xe8f00fff,
>
> Thanks, that's pretty helpful. I'll update the patch with these values
> and the corresponding description.
Thanks for the update.
>
> > > + },
> > > {
> > > }
> > > };
> > > @@ -1577,7 +1593,12 @@ static int __init gic_init_bases(void __iomem
> > > *dist_base,
> > > pr_info("%d SPIs implemented\n", GIC_LINE_NR - 32);
> > > pr_info("%d Extended SPIs implemented\n", GIC_ESPI_NR);
> > >
> > > - gic_data.rdists.gicd_typer2 = readl_relaxed(gic_data.dist_base +
> > > GICD_TYPER2);
> > > + /*
> > > + * ThunderX1 explodes on reading GICD_TYPER2, in total violation
> > > + * of the spec (which says that reserved addresses are RES0).
> > > + */
> > > + if (!(gic_data.flags & FLAGS_WORKAROUND_GICD_TYPER2_TX1))
> > > + gic_data.rdists.gicd_typer2 = readl_relaxed(gic_data.dist_base +
> > > GICD_TYPER2);
> >
> > You already said that checking for ArchRev of GICD_PIDR2 isn't an
> > option here. Though, it could...
>
> Once GICv3.2 starts using this register as well (because GICD_TYPER is
> already completely full), we'd have to fix it again. There is also the thing
> you hinted at in the other thread: TX1 will generate a SEA on every reserved
> GICD registers, so we may need to protect more than just this one over time,
> and maybe more than just in the distributor.
I am fine with both. And right, it would be possibly needed for other
ranges too.
Thanks,
-Robert
>
> Thanks,
>
> M.
> --
> Jazz is not dead. It just smells funny...
Hi Marc,
On 10.03.20 11:41:09, Marc Zyngier wrote:
> On 2020-03-09 22:11, Robert Richter wrote:
> > On 24.12.19 11:10:26, Marc Zyngier wrote:
> > > @@ -1502,6 +1512,12 @@ static const struct gic_quirk gic_quirks[] = {
> > > .mask = 0xffffffff,
> > > .init = gic_enable_quirk_hip06_07,
> > > },
> > > + {
> > > + .desc = "GICv3: Cavium TX1 GICD_TYPER2 erratum",
> >
> > There is no errata number yet.
>
> Please let me know when/if you obtain one.
GIC-38539: GIC faults when accessing reserved GICD_TYPER2 register
Applies to (covered with iidr mask below):
ThunderX: CN88xx
OCTEON TX: CN83xx, CN81xx
OCTEON TX2: CN93xx, CN96xx, CN98xx, CNF95xx*
Issue: Access to GIC reserved registers results in an exception.
Notes:
1) This applies to other reserved registers too.
2) The errata number is unique over all IP blocks, so a macro
CAVIUM_ERRATUM_38539 is ok.
>
> >
> > > + .iidr = 0xa100034c,
> > > + .mask = 0xfff00fff,
> > > + .init = gic_enable_quirk_tx1,
> >
> > All TX1 and OcteonTX parts are affected, which is a0-a7 and b0-b7. So
> > the iidr/mask should be:
> >
> > .iidr = 0xa000034c,
> > .mask = 0xe8f00fff,
>
> Thanks, that's pretty helpful. I'll update the patch with these values
> and the corresponding description.
>
> > > + },
> > > {
> > > }
> > > };
Thank you for addressing this.
-Robert
Hi Robert,
On 2020-03-11 08:45, Robert Richter wrote:
> Hi Marc,
>
> On 10.03.20 11:41:09, Marc Zyngier wrote:
>> On 2020-03-09 22:11, Robert Richter wrote:
>> > On 24.12.19 11:10:26, Marc Zyngier wrote:
>
>> > > @@ -1502,6 +1512,12 @@ static const struct gic_quirk gic_quirks[] = {
>> > > .mask = 0xffffffff,
>> > > .init = gic_enable_quirk_hip06_07,
>> > > },
>> > > + {
>> > > + .desc = "GICv3: Cavium TX1 GICD_TYPER2 erratum",
>> >
>> > There is no errata number yet.
>>
>> Please let me know when/if you obtain one.
>
> GIC-38539: GIC faults when accessing reserved GICD_TYPER2 register
>
> Applies to (covered with iidr mask below):
>
> ThunderX: CN88xx
> OCTEON TX: CN83xx, CN81xx
> OCTEON TX2: CN93xx, CN96xx, CN98xx, CNF95xx*
>
> Issue: Access to GIC reserved registers results in an exception.
> Notes:
> 1) This applies to other reserved registers too.
> 2) The errata number is unique over all IP blocks, so a macro
> CAVIUM_ERRATUM_38539 is ok.
Great, thanks a lot for chasing this. One question though: does this
apply to the distributor only? Or to all reserved registers regardless
of the architectural block they are in?
It won't change the workaround for now, but knowing the scope of the
erratum will help future developments.
Thanks,
M.
--
Jazz is not dead. It just smells funny...
On 11.03.20 09:03:48, Marc Zyngier wrote:
> Hi Robert,
>
> On 2020-03-11 08:45, Robert Richter wrote:
> > Hi Marc,
> >
> > On 10.03.20 11:41:09, Marc Zyngier wrote:
> > > On 2020-03-09 22:11, Robert Richter wrote:
> > > > On 24.12.19 11:10:26, Marc Zyngier wrote:
> >
> > > > > @@ -1502,6 +1512,12 @@ static const struct gic_quirk gic_quirks[] = {
> > > > > .mask = 0xffffffff,
> > > > > .init = gic_enable_quirk_hip06_07,
> > > > > },
> > > > > + {
> > > > > + .desc = "GICv3: Cavium TX1 GICD_TYPER2 erratum",
> > > >
> > > > There is no errata number yet.
> > >
> > > Please let me know when/if you obtain one.
> >
> > GIC-38539: GIC faults when accessing reserved GICD_TYPER2 register
> >
> > Applies to (covered with iidr mask below):
> >
> > ThunderX: CN88xx
> > OCTEON TX: CN83xx, CN81xx
> > OCTEON TX2: CN93xx, CN96xx, CN98xx, CNF95xx*
> >
> > Issue: Access to GIC reserved registers results in an exception.
> > Notes:
> > 1) This applies to other reserved registers too.
> > 2) The errata number is unique over all IP blocks, so a macro
> > CAVIUM_ERRATUM_38539 is ok.
>
> Great, thanks a lot for chasing this. One question though: does this
> apply to the distributor only? Or to all reserved registers regardless
> of the architectural block they are in?
It is the whole GIC IP block covered by GICv3 spec. It was implemented
during a very early state of the spec and the RES0 requirement was
added later to reserved registers. CN8xxx GIC CSR access checking is
stricter implemented, so if no register is present, an access results
in a fault other than simply returning a RAZ/WI.
Thanks,
-Robert
>
> It won't change the workaround for now, but knowing the scope of the
> erratum will help future developments.
>
> Thanks,
>
> M.
> --
> Jazz is not dead. It just smells funny...