The GICv3 LPI allocator has served us well so far, but a number of new
use cases have recently showed up:
- A new extension to the GICv3 architecture allows a hypervisor to
dramatically restrict the range of available LPIs. This means that
our current policy of allocating LPIs in blocks of 32 may quickly
deplete the number of devices that get LPIs
- New and currently undisclosed busses seem to come with thousands of
devices, each requiring a single LPI. Again, our current allocation
policy means they quickly run out of LPIs.
Simply expanding the bitmap doesn't seem to be a great idea, so let's
change the LPI allocator altogether. This means we can move individual
busses to a more minimal allocation scheme, though we only do it for
PCI at the moment (Platform MSI looks like the Far West, and I'm
clueless about the FSL MC thing).
This is a pretty invasive change, and I'm thus cc'ing the usual
suspects that have access to weird and wonderful HW to verify
everything still works as expected, and let me know if we can relax
the allocation for their own pet bus implementation.
Only lightly tested in a KVM guest (PCI).
Marc Zyngier (7):
irqchip/gic-v3-its: Refactor LPI allocator
irqchip/gic-v3-its: Use full range of LPIs
irqchip/gic-v3-its: Move minimum LPI requirements to individual busses
irqchip/gic-v3-its: Drop chunk allocation compatibility
irqchip/gic-v3: Expose GICD_TYPER in the rdist structure
irqchip/gic-v3-its: Honor hypervisor enforced LPI range
irqchip/gic-v3-its: Reduce minimum LPI allocation to 1 for PCI devices
drivers/irqchip/irq-gic-v3-its-fsl-mc-msi.c | 3 +
drivers/irqchip/irq-gic-v3-its-pci-msi.c | 16 +-
drivers/irqchip/irq-gic-v3-its-platform-msi.c | 2 +
drivers/irqchip/irq-gic-v3-its.c | 225 ++++++++++++------
drivers/irqchip/irq-gic-v3.c | 4 +-
include/linux/irqchip/arm-gic-v3.h | 3 +-
6 files changed, 169 insertions(+), 84 deletions(-)
--
2.17.1
Our current LPI allocator relies on a bitmap, each bit representing
a chunk of 32 LPIs, meaning that each device gets allocated LPIs
in multiple of 32. It served us well so far, but new use cases now
require much more finer grain allocations, down the the individual
LPI.
Given the size of the IntID space (up to 32bit), it isn't practical
to continue using a bitmap, so let's use a different data structure
altogether.
We switch to a list, where each element represent a contiguous range
of LPIs. On allocation, we simply grab the first group big enough to
satisfy the allocation, and substract what we need from it. If the
group becomes empty, we just remove it. On freeing interrupts, we
insert a new group of interrupt in the list, sort it and fuse the
adjacent groups.
This makes freeing interrupt much more expensive than allocating
them (an unusual behaviour), but that's fine as long as we consider
that freeing interrupts is an extremely rare event.
We still allocate interrupts in blocks of 32 for the time being,
but subsequent patches will relax this.
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3-its.c | 191 +++++++++++++++++++++----------
1 file changed, 129 insertions(+), 62 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 5377d7e2afba..6d148c2108b9 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -23,6 +23,8 @@
#include <linux/dma-iommu.h>
#include <linux/interrupt.h>
#include <linux/irqdomain.h>
+#include <linux/list.h>
+#include <linux/list_sort.h>
#include <linux/log2.h>
#include <linux/mm.h>
#include <linux/msi.h>
@@ -1405,112 +1407,177 @@ static struct irq_chip its_irq_chip = {
.irq_set_vcpu_affinity = its_irq_set_vcpu_affinity,
};
+
/*
* How we allocate LPIs:
*
- * The GIC has id_bits bits for interrupt identifiers. From there, we
- * must subtract 8192 which are reserved for SGIs/PPIs/SPIs. Then, as
- * we allocate LPIs by chunks of 32, we can shift the whole thing by 5
- * bits to the right.
+ * lpi_range_list contains ranges of LPIs that are to available to
+ * allocate from. To allocate LPIs, just pick the first range that
+ * fits the required allocation, and reduce it by the required
+ * amount. Once empty, remove the range from the list.
+ *
+ * To free a range of LPIs, add a free range to the list, sort it and
+ * merge the result if the new range happens to be adjacent to an
+ * already free block.
*
- * This gives us (((1UL << id_bits) - 8192) >> 5) possible allocations.
+ * The consequence of the above is that allocation is cost is low, but
+ * freeing is expensive. We assumes that freeing rarely occurs.
+ */
+
+/*
+ * Compatibility defines until we fully refactor the allocator
*/
#define IRQS_PER_CHUNK_SHIFT 5
#define IRQS_PER_CHUNK (1UL << IRQS_PER_CHUNK_SHIFT)
#define ITS_MAX_LPI_NRBITS 16 /* 64K LPIs */
-static unsigned long *lpi_bitmap;
-static u32 lpi_chunks;
-static DEFINE_SPINLOCK(lpi_lock);
+static DEFINE_MUTEX(lpi_range_lock);
+static LIST_HEAD(lpi_range_list);
-static int its_lpi_to_chunk(int lpi)
+struct lpi_range {
+ struct list_head entry;
+ u32 base_id;
+ u32 span;
+};
+
+static struct lpi_range *mk_lpi_range(u32 base, u32 span)
{
- return (lpi - 8192) >> IRQS_PER_CHUNK_SHIFT;
+ struct lpi_range *range;
+
+ range = kzalloc(sizeof(*range), GFP_KERNEL);
+ if (range) {
+ INIT_LIST_HEAD(&range->entry);
+ range->base_id = base;
+ range->span = span;
+ }
+
+ return range;
}
-static int its_chunk_to_lpi(int chunk)
+static int lpi_range_cmp(void *priv, struct list_head *a, struct list_head *b)
{
- return (chunk << IRQS_PER_CHUNK_SHIFT) + 8192;
+ struct lpi_range *ra, *rb;
+
+ ra = container_of(a, struct lpi_range, entry);
+ rb = container_of(b, struct lpi_range, entry);
+
+ return rb->base_id - ra->base_id;
}
-static int __init its_lpi_init(u32 id_bits)
+static void merge_lpi_ranges(void)
{
- lpi_chunks = its_lpi_to_chunk(1UL << id_bits);
+ struct lpi_range *range, *tmp;
- lpi_bitmap = kcalloc(BITS_TO_LONGS(lpi_chunks), sizeof(long),
- GFP_KERNEL);
- if (!lpi_bitmap) {
- lpi_chunks = 0;
- return -ENOMEM;
+ list_for_each_entry_safe(range, tmp, &lpi_range_list, entry) {
+ if (!list_is_last(&range->entry, &lpi_range_list) &&
+ (tmp->base_id == (range->base_id + range->span))) {
+ tmp->base_id = range->base_id;
+ tmp->span += range->span;
+ list_del(&range->entry);
+ kfree(range);
+ }
}
+}
- pr_info("ITS: Allocated %d chunks for LPIs\n", (int)lpi_chunks);
- return 0;
+static int alloc_lpi_range(u32 nr_lpis, u32 *base)
+{
+ struct lpi_range *range, *tmp;
+ int err = -ENOSPC;
+
+ mutex_lock(&lpi_range_lock);
+
+ list_for_each_entry_safe(range, tmp, &lpi_range_list, entry) {
+ if (range->span >= nr_lpis) {
+ *base = range->base_id;
+ range->base_id += nr_lpis;
+ range->span -= nr_lpis;
+
+ if (range->span == 0) {
+ list_del(&range->entry);
+ kfree(range);
+ }
+
+ err = 0;
+ break;
+ }
+ }
+
+ mutex_unlock(&lpi_range_lock);
+
+ pr_debug("ITS: alloc %u:%u\n", *base, nr_lpis);
+ return err;
}
-static unsigned long *its_lpi_alloc_chunks(int nr_irqs, int *base, int *nr_ids)
+static int free_lpi_range(u32 base, u32 nr_lpis)
{
- unsigned long *bitmap = NULL;
- int chunk_id;
- int nr_chunks;
- int i;
+ struct lpi_range *new;
+ int err = 0;
- nr_chunks = DIV_ROUND_UP(nr_irqs, IRQS_PER_CHUNK);
+ mutex_lock(&lpi_range_lock);
- spin_lock(&lpi_lock);
+ new = mk_lpi_range(base, nr_lpis);
+ if (!new) {
+ err = -ENOMEM;
+ goto out;
+ }
+
+ list_add(&new->entry, &lpi_range_list);
+ list_sort(NULL, &lpi_range_list, lpi_range_cmp);
+ merge_lpi_ranges();
+out:
+ mutex_unlock(&lpi_range_lock);
+ return err;
+}
+
+static int __init its_lpi_init(u32 id_bits)
+{
+ u32 lpis = (1UL << id_bits) - 8192;
+ int err;
+
+ /*
+ * Initializing the allocator is just the same as freeing the
+ * full range of LPIs.
+ */
+ err = free_lpi_range(8192, lpis);
+ pr_debug("ITS: Allocator initialized for %u LPIs\n", lpis);
+ return err;
+}
+
+static unsigned long *its_lpi_alloc_chunks(int nr_irqs, u32 *base, int *nr_ids)
+{
+ unsigned long *bitmap = NULL;
+ int err = 0;
+ int nr_lpis;
+
+ nr_lpis = round_up(nr_irqs, IRQS_PER_CHUNK);
do {
- chunk_id = bitmap_find_next_zero_area(lpi_bitmap, lpi_chunks,
- 0, nr_chunks, 0);
- if (chunk_id < lpi_chunks)
+ err = alloc_lpi_range(nr_lpis, base);
+ if (!err)
break;
- nr_chunks--;
- } while (nr_chunks > 0);
+ nr_lpis -= IRQS_PER_CHUNK;
+ } while (nr_lpis > 0);
- if (!nr_chunks)
+ if (err)
goto out;
- bitmap = kcalloc(BITS_TO_LONGS(nr_chunks * IRQS_PER_CHUNK),
- sizeof(long),
- GFP_ATOMIC);
+ bitmap = kcalloc(BITS_TO_LONGS(nr_lpis), sizeof (long), GFP_ATOMIC);
if (!bitmap)
goto out;
- for (i = 0; i < nr_chunks; i++)
- set_bit(chunk_id + i, lpi_bitmap);
-
- *base = its_chunk_to_lpi(chunk_id);
- *nr_ids = nr_chunks * IRQS_PER_CHUNK;
+ *nr_ids = nr_lpis;
out:
- spin_unlock(&lpi_lock);
-
if (!bitmap)
*base = *nr_ids = 0;
return bitmap;
}
-static void its_lpi_free_chunks(unsigned long *bitmap, int base, int nr_ids)
+static void its_lpi_free_chunks(unsigned long *bitmap, u32 base, u32 nr_ids)
{
- int lpi;
-
- spin_lock(&lpi_lock);
-
- for (lpi = base; lpi < (base + nr_ids); lpi += IRQS_PER_CHUNK) {
- int chunk = its_lpi_to_chunk(lpi);
-
- BUG_ON(chunk > lpi_chunks);
- if (test_bit(chunk, lpi_bitmap)) {
- clear_bit(chunk, lpi_bitmap);
- } else {
- pr_err("Bad LPI chunk %d\n", chunk);
- }
- }
-
- spin_unlock(&lpi_lock);
-
+ WARN_ON(free_lpi_range(base, nr_ids));
kfree(bitmap);
}
--
2.17.1
As we used to represent the LPI range using a bitmap, we were reducing
the number of LPIs to at most 64k in order to preserve memory.
With our new allocator, there is no such need, as dealing with 2^16
or 2^32 LPIs takes the same amount of memory.
So let's use the number of IntID bits reported by the GIC instead of
an arbitrary limit.
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3-its.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 6d148c2108b9..c98e49f8c341 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -1429,7 +1429,6 @@ static struct irq_chip its_irq_chip = {
*/
#define IRQS_PER_CHUNK_SHIFT 5
#define IRQS_PER_CHUNK (1UL << IRQS_PER_CHUNK_SHIFT)
-#define ITS_MAX_LPI_NRBITS 16 /* 64K LPIs */
static DEFINE_MUTEX(lpi_range_lock);
static LIST_HEAD(lpi_range_list);
@@ -1610,7 +1609,7 @@ static int __init its_alloc_lpi_tables(void)
{
phys_addr_t paddr;
- lpi_id_bits = min_t(u32, gic_rdists->id_bits, ITS_MAX_LPI_NRBITS);
+ lpi_id_bits = gic_rdists->id_bits;
gic_rdists->prop_page = its_allocate_prop_table(GFP_NOWAIT);
if (!gic_rdists->prop_page) {
pr_err("Failed to allocate PROPBASE\n");
--
2.17.1
At the moment, the core ITS driver imposes the allocation to be
in chunks of 32. As we want to relax this on a per bus basis, let's
move the the the allocation constraints to each bus.
No functionnal change.
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3-its-fsl-mc-msi.c | 3 +++
drivers/irqchip/irq-gic-v3-its-pci-msi.c | 6 ++++--
drivers/irqchip/irq-gic-v3-its-platform-msi.c | 2 ++
drivers/irqchip/irq-gic-v3-its.c | 11 +++++++----
4 files changed, 16 insertions(+), 6 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3-its-fsl-mc-msi.c b/drivers/irqchip/irq-gic-v3-its-fsl-mc-msi.c
index 4eca5c763766..606efa64adff 100644
--- a/drivers/irqchip/irq-gic-v3-its-fsl-mc-msi.c
+++ b/drivers/irqchip/irq-gic-v3-its-fsl-mc-msi.c
@@ -45,6 +45,9 @@ static int its_fsl_mc_msi_prepare(struct irq_domain *msi_domain,
*/
info->scratchpad[0].ul = mc_bus_dev->icid;
msi_info = msi_get_domain_info(msi_domain->parent);
+
+ /* Allocate at least 32 MSIs, and always as a power of 2 */
+ nvec = max_t(int, 32, roundup_pow_of_two(nvec));
return msi_info->ops->msi_prepare(msi_domain->parent, dev, nvec, info);
}
diff --git a/drivers/irqchip/irq-gic-v3-its-pci-msi.c b/drivers/irqchip/irq-gic-v3-its-pci-msi.c
index 25a98de5cfb2..75c3cafabc6a 100644
--- a/drivers/irqchip/irq-gic-v3-its-pci-msi.c
+++ b/drivers/irqchip/irq-gic-v3-its-pci-msi.c
@@ -86,8 +86,10 @@ static int its_pci_msi_prepare(struct irq_domain *domain, struct device *dev,
/* ITS specific DeviceID, as the core ITS ignores dev. */
info->scratchpad[0].ul = pci_msi_domain_get_msi_rid(domain, pdev);
- return msi_info->ops->msi_prepare(domain->parent,
- dev, max(nvec, alias_count), info);
+ /* Allocate at least 32 MSIs, and always as a power of 2 */
+ nvec = max(nvec, alias_count);
+ nvec = max_t(int, 32, roundup_pow_of_two(nvec));
+ return msi_info->ops->msi_prepare(domain->parent, dev, nvec, info);
}
static struct msi_domain_ops its_pci_msi_ops = {
diff --git a/drivers/irqchip/irq-gic-v3-its-platform-msi.c b/drivers/irqchip/irq-gic-v3-its-platform-msi.c
index 8881a053c173..7b8e87b493fe 100644
--- a/drivers/irqchip/irq-gic-v3-its-platform-msi.c
+++ b/drivers/irqchip/irq-gic-v3-its-platform-msi.c
@@ -73,6 +73,8 @@ static int its_pmsi_prepare(struct irq_domain *domain, struct device *dev,
/* ITS specific DeviceID, as the core ITS ignores dev. */
info->scratchpad[0].ul = dev_id;
+ /* Allocate at least 32 MSIs, and always as a power of 2 */
+ nvec = max_t(int, 32, roundup_pow_of_two(nvec));
return msi_info->ops->msi_prepare(domain->parent,
dev, nvec, info);
}
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index c98e49f8c341..cfe65776bb4c 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -2179,12 +2179,15 @@ static struct its_device *its_create_device(struct its_node *its, u32 dev_id,
if (!its_alloc_device_table(its, dev_id))
return NULL;
+ if (WARN_ON(!is_power_of_2(nvecs)))
+ nvecs = roundup_pow_of_two(nvecs);
+
dev = kzalloc(sizeof(*dev), GFP_KERNEL);
/*
- * We allocate at least one chunk worth of LPIs bet device,
- * and thus that many ITEs. The device may require less though.
+ * Even if the device wants a single LPI, the ITT must be
+ * sized as a power of two (and you need at least one bit...).
*/
- nr_ites = max(IRQS_PER_CHUNK, roundup_pow_of_two(nvecs));
+ nr_ites = max(2, nvecs);
sz = nr_ites * its->ite_size;
sz = max(sz, ITS_ITT_ALIGN) + ITS_ITT_ALIGN - 1;
itt = kzalloc(sz, GFP_KERNEL);
@@ -2833,7 +2836,7 @@ static int its_vpe_irq_domain_alloc(struct irq_domain *domain, unsigned int virq
BUG_ON(!vm);
- bitmap = its_lpi_alloc_chunks(nr_irqs, &base, &nr_ids);
+ bitmap = its_lpi_alloc_chunks(roundup_pow_of_two(nr_irqs), &base, &nr_ids);
if (!bitmap)
return -ENOMEM;
--
2.17.1
The chunk allocation system is now officially dead, so let's
remove it.
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3-its.c | 41 +++++++++++++-------------------
1 file changed, 16 insertions(+), 25 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index cfe65776bb4c..554c9d040c5a 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -1424,12 +1424,6 @@ static struct irq_chip its_irq_chip = {
* freeing is expensive. We assumes that freeing rarely occurs.
*/
-/*
- * Compatibility defines until we fully refactor the allocator
- */
-#define IRQS_PER_CHUNK_SHIFT 5
-#define IRQS_PER_CHUNK (1UL << IRQS_PER_CHUNK_SHIFT)
-
static DEFINE_MUTEX(lpi_range_lock);
static LIST_HEAD(lpi_range_list);
@@ -1542,30 +1536,27 @@ static int __init its_lpi_init(u32 id_bits)
return err;
}
-static unsigned long *its_lpi_alloc_chunks(int nr_irqs, u32 *base, int *nr_ids)
+static unsigned long *its_lpi_alloc(int nr_irqs, u32 *base, int *nr_ids)
{
unsigned long *bitmap = NULL;
int err = 0;
- int nr_lpis;
-
- nr_lpis = round_up(nr_irqs, IRQS_PER_CHUNK);
do {
- err = alloc_lpi_range(nr_lpis, base);
+ err = alloc_lpi_range(nr_irqs, base);
if (!err)
break;
- nr_lpis -= IRQS_PER_CHUNK;
- } while (nr_lpis > 0);
+ nr_irqs /= 2;
+ } while (nr_irqs > 0);
if (err)
goto out;
- bitmap = kcalloc(BITS_TO_LONGS(nr_lpis), sizeof (long), GFP_ATOMIC);
+ bitmap = kcalloc(BITS_TO_LONGS(nr_irqs), sizeof (long), GFP_ATOMIC);
if (!bitmap)
goto out;
- *nr_ids = nr_lpis;
+ *nr_ids = nr_irqs;
out:
if (!bitmap)
@@ -1574,7 +1565,7 @@ static unsigned long *its_lpi_alloc_chunks(int nr_irqs, u32 *base, int *nr_ids)
return bitmap;
}
-static void its_lpi_free_chunks(unsigned long *bitmap, u32 base, u32 nr_ids)
+static void its_lpi_free(unsigned long *bitmap, u32 base, u32 nr_ids)
{
WARN_ON(free_lpi_range(base, nr_ids));
kfree(bitmap);
@@ -2192,7 +2183,7 @@ static struct its_device *its_create_device(struct its_node *its, u32 dev_id,
sz = max(sz, ITS_ITT_ALIGN) + ITS_ITT_ALIGN - 1;
itt = kzalloc(sz, GFP_KERNEL);
if (alloc_lpis) {
- lpi_map = its_lpi_alloc_chunks(nvecs, &lpi_base, &nr_lpis);
+ lpi_map = its_lpi_alloc(nvecs, &lpi_base, &nr_lpis);
if (lpi_map)
col_map = kcalloc(nr_lpis, sizeof(*col_map),
GFP_KERNEL);
@@ -2420,9 +2411,9 @@ static void its_irq_domain_free(struct irq_domain *domain, unsigned int virq,
/* If all interrupts have been freed, start mopping the floor */
if (bitmap_empty(its_dev->event_map.lpi_map,
its_dev->event_map.nr_lpis)) {
- its_lpi_free_chunks(its_dev->event_map.lpi_map,
- its_dev->event_map.lpi_base,
- its_dev->event_map.nr_lpis);
+ its_lpi_free(its_dev->event_map.lpi_map,
+ its_dev->event_map.lpi_base,
+ its_dev->event_map.nr_lpis);
kfree(its_dev->event_map.col_map);
/* Unmap device/itt */
@@ -2821,7 +2812,7 @@ static void its_vpe_irq_domain_free(struct irq_domain *domain,
}
if (bitmap_empty(vm->db_bitmap, vm->nr_db_lpis)) {
- its_lpi_free_chunks(vm->db_bitmap, vm->db_lpi_base, vm->nr_db_lpis);
+ its_lpi_free(vm->db_bitmap, vm->db_lpi_base, vm->nr_db_lpis);
its_free_prop_table(vm->vprop_page);
}
}
@@ -2836,18 +2827,18 @@ static int its_vpe_irq_domain_alloc(struct irq_domain *domain, unsigned int virq
BUG_ON(!vm);
- bitmap = its_lpi_alloc_chunks(roundup_pow_of_two(nr_irqs), &base, &nr_ids);
+ bitmap = its_lpi_alloc(roundup_pow_of_two(nr_irqs), &base, &nr_ids);
if (!bitmap)
return -ENOMEM;
if (nr_ids < nr_irqs) {
- its_lpi_free_chunks(bitmap, base, nr_ids);
+ its_lpi_free(bitmap, base, nr_ids);
return -ENOMEM;
}
vprop_page = its_allocate_prop_table(GFP_KERNEL);
if (!vprop_page) {
- its_lpi_free_chunks(bitmap, base, nr_ids);
+ its_lpi_free(bitmap, base, nr_ids);
return -ENOMEM;
}
@@ -2874,7 +2865,7 @@ static int its_vpe_irq_domain_alloc(struct irq_domain *domain, unsigned int virq
if (i > 0)
its_vpe_irq_domain_free(domain, virq, i - 1);
- its_lpi_free_chunks(bitmap, base, nr_ids);
+ its_lpi_free(bitmap, base, nr_ids);
its_free_prop_table(vprop_page);
}
--
2.17.1
Instead of exposing the GIC distributor IntID field in the rdist
structure that is passed to the ITS, let's replace it with a
copy of the whole GICD_TYPER register. We are going to need
some of this information at a later time.
No functionnal change.
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3-its.c | 2 +-
drivers/irqchip/irq-gic-v3.c | 4 ++--
include/linux/irqchip/arm-gic-v3.h | 2 +-
3 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 554c9d040c5a..8c7e8c235faf 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -1600,7 +1600,7 @@ static int __init its_alloc_lpi_tables(void)
{
phys_addr_t paddr;
- lpi_id_bits = gic_rdists->id_bits;
+ lpi_id_bits = GICD_TYPER_ID_BITS(gic_rdists->gicd_typer);
gic_rdists->prop_page = its_allocate_prop_table(GFP_NOWAIT);
if (!gic_rdists->prop_page) {
pr_err("Failed to allocate PROPBASE\n");
diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index 76ea56d779a1..e214181b77b7 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -877,7 +877,7 @@ static struct irq_chip gic_eoimode1_chip = {
.flags = IRQCHIP_SET_TYPE_MASKED,
};
-#define GIC_ID_NR (1U << gic_data.rdists.id_bits)
+#define GIC_ID_NR (1U << GICD_TYPER_ID_BITS(gic_data.rdists.gicd_typer))
static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq,
irq_hw_number_t hw)
@@ -1091,7 +1091,7 @@ static int __init gic_init_bases(void __iomem *dist_base,
* The GIC only supports up to 1020 interrupt sources (SGI+PPI+SPI)
*/
typer = readl_relaxed(gic_data.dist_base + GICD_TYPER);
- gic_data.rdists.id_bits = GICD_TYPER_ID_BITS(typer);
+ gic_data.rdists.gicd_typer = typer;
gic_irqs = GICD_TYPER_IRQS(typer);
if (gic_irqs > 1020)
gic_irqs = 1020;
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index cbb872c1b607..396cd99af02f 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -576,8 +576,8 @@ struct rdists {
phys_addr_t phys_base;
} __percpu *rdist;
struct page *prop_page;
- int id_bits;
u64 flags;
+ u32 gicd_typer;
bool has_vlpis;
bool has_direct_lpi;
};
--
2.17.1
A recent extension to the GIC architecture allows a hypervisor to
arbitrarily reduce the number of LPIs available to a guest, no
matter what the GIC says about the valid range of IntIDs.
Let's factor in this information when computing the number of
available LPIs
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3-its.c | 9 +++++++++
include/linux/irqchip/arm-gic-v3.h | 1 +
2 files changed, 10 insertions(+)
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 8c7e8c235faf..903ca1c19553 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -1525,8 +1525,17 @@ static int free_lpi_range(u32 base, u32 nr_lpis)
static int __init its_lpi_init(u32 id_bits)
{
u32 lpis = (1UL << id_bits) - 8192;
+ u32 numlpis;
int err;
+ numlpis = 1UL << GICD_TYPER_NUM_LPIS(gic_rdists->gicd_typer);
+
+ if (numlpis > 1 && !WARN_ON(numlpis > lpis)) {
+ lpis = numlpis;
+ pr_info("ITS: Using hypervisor restricted LPI range [%u]\n",
+ lpis);
+ }
+
/*
* Initializing the allocator is just the same as freeing the
* full range of LPIs.
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index 396cd99af02f..9d2ea3e907d0 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -73,6 +73,7 @@
#define GICD_TYPER_MBIS (1U << 16)
#define GICD_TYPER_ID_BITS(typer) ((((typer) >> 19) & 0x1f) + 1)
+#define GICD_TYPER_NUM_LPIS(typer) ((((typer) >> 11) & 0x1f) + 1)
#define GICD_TYPER_IRQS(typer) ((((typer) & 0x1f) + 1) * 32)
#define GICD_IROUTER_SPI_MODE_ONE (0U << 31)
--
2.17.1
Allocating a minimum of 32 LPIs per PCI device, let's reduce it to
be just 1, as most devices do not need that many interrupts.
We still have to special-case DevID 0, as there is plenty of broken
HW around where the PCI RID is not presented as a DevID to the ITS,
and all the devices are presented as DevID 0. In this case, we keep
the 32 minimal allocation.
Signed-off-by: Marc Zyngier <[email protected]>
---
drivers/irqchip/irq-gic-v3-its-pci-msi.c | 14 +++++++++++---
1 file changed, 11 insertions(+), 3 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3-its-pci-msi.c b/drivers/irqchip/irq-gic-v3-its-pci-msi.c
index 75c3cafabc6a..8d6d009d1d58 100644
--- a/drivers/irqchip/irq-gic-v3-its-pci-msi.c
+++ b/drivers/irqchip/irq-gic-v3-its-pci-msi.c
@@ -66,7 +66,7 @@ static int its_pci_msi_prepare(struct irq_domain *domain, struct device *dev,
{
struct pci_dev *pdev, *alias_dev;
struct msi_domain_info *msi_info;
- int alias_count = 0;
+ int alias_count = 0, minnvec = 1;
if (!dev_is_pci(dev))
return -EINVAL;
@@ -86,9 +86,17 @@ static int its_pci_msi_prepare(struct irq_domain *domain, struct device *dev,
/* ITS specific DeviceID, as the core ITS ignores dev. */
info->scratchpad[0].ul = pci_msi_domain_get_msi_rid(domain, pdev);
- /* Allocate at least 32 MSIs, and always as a power of 2 */
+ /*
+ * Always allocate a power of 2, and special case device 0 for
+ * broken systems where the DevID is not wired (and all devices
+ * appear as DevID 0). For that reason, we generously allocate a
+ * minimum of 32 MSIs for DevID 0. If you want more because all
+ * your devices are aliasing to DevID 0, consider fixing your HW.
+ */
nvec = max(nvec, alias_count);
- nvec = max_t(int, 32, roundup_pow_of_two(nvec));
+ if (!info->scratchpad[0].ul)
+ minnvec = 32;
+ nvec = max_t(int, minnvec, roundup_pow_of_two(nvec));
return msi_info->ops->msi_prepare(domain->parent, dev, nvec, info);
}
--
2.17.1
Hi Marc,
> -----Original Message-----
> From: Marc Zyngier [mailto:[email protected]]
> Sent: 20 June 2018 14:53
> To: [email protected]
> Cc: Thomas Gleixner <[email protected]>; Ard Biesheuvel
> <[email protected]>; Shanker Donthineni
> <[email protected]>; Shameerali Kolothum Thodi
> <[email protected]>; MaJun <[email protected]>;
> Laurentiu Tudor <[email protected]>; Lei Zhang
> <[email protected]>
> Subject: [PATCH 6/7] irqchip/gic-v3-its: Honor hypervisor enforced LPI range
>
> A recent extension to the GIC architecture allows a hypervisor to
> arbitrarily reduce the number of LPIs available to a guest, no
> matter what the GIC says about the valid range of IntIDs.
>
> Let's factor in this information when computing the number of
> available LPIs
On our D05 board, this limits the lpis to 2 and results in MSI irq alloc fails:
[ 0.000000] ITS: Using hypervisor restricted LPI range [2]
....
[ 10.543889] ixgbe 000a:11:00.1: Failed to allocate MSI interrupt, falling back to legacy. Error: -12
> Signed-off-by: Marc Zyngier <[email protected]>
> ---
> drivers/irqchip/irq-gic-v3-its.c | 9 +++++++++
> include/linux/irqchip/arm-gic-v3.h | 1 +
> 2 files changed, 10 insertions(+)
>
> diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
> index 8c7e8c235faf..903ca1c19553 100644
> --- a/drivers/irqchip/irq-gic-v3-its.c
> +++ b/drivers/irqchip/irq-gic-v3-its.c
> @@ -1525,8 +1525,17 @@ static int free_lpi_range(u32 base, u32 nr_lpis)
> static int __init its_lpi_init(u32 id_bits)
> {
> u32 lpis = (1UL << id_bits) - 8192;
> + u32 numlpis;
> int err;
>
> + numlpis = 1UL << GICD_TYPER_NUM_LPIS(gic_rdists->gicd_typer);
> +
> + if (numlpis > 1 && !WARN_ON(numlpis > lpis)) {
> + lpis = numlpis;
> + pr_info("ITS: Using hypervisor restricted LPI range [%u]\n",
> + lpis);
> + }
I don't have the GICv3 extension doc, but did you intent to check for,
if (numlpis > 2 && !WARN_ON(numlpis > lpis)) {
as it looks like D05 returns 0 for bits 11-15 and that makes numlpis=2.
Please let me know.
Thanks,
Shameer
> +
> /*
> * Initializing the allocator is just the same as freeing the
> * full range of LPIs.
> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-
> v3.h
> index 396cd99af02f..9d2ea3e907d0 100644
> --- a/include/linux/irqchip/arm-gic-v3.h
> +++ b/include/linux/irqchip/arm-gic-v3.h
> @@ -73,6 +73,7 @@
> #define GICD_TYPER_MBIS (1U << 16)
>
> #define GICD_TYPER_ID_BITS(typer) ((((typer) >> 19) & 0x1f) + 1)
> +#define GICD_TYPER_NUM_LPIS(typer) ((((typer) >> 11) & 0x1f) + 1)
> #define GICD_TYPER_IRQS(typer) ((((typer) & 0x1f) + 1) * 32)
>
> #define GICD_IROUTER_SPI_MODE_ONE (0U << 31)
> --
> 2.17.1
Hi Shammer,
On Fri, 22 Jun 2018 17:31:40 +0100,
Shameerali Kolothum Thodi <[email protected]> wrote:
>
> Hi Marc,
>
> > -----Original Message-----
> > From: Marc Zyngier [mailto:[email protected]]
> > Sent: 20 June 2018 14:53
> > To: [email protected]
> > Cc: Thomas Gleixner <[email protected]>; Ard Biesheuvel
> > <[email protected]>; Shanker Donthineni
> > <[email protected]>; Shameerali Kolothum Thodi
> > <[email protected]>; MaJun <[email protected]>;
> > Laurentiu Tudor <[email protected]>; Lei Zhang
> > <[email protected]>
> > Subject: [PATCH 6/7] irqchip/gic-v3-its: Honor hypervisor enforced LPI range
> >
> > A recent extension to the GIC architecture allows a hypervisor to
> > arbitrarily reduce the number of LPIs available to a guest, no
> > matter what the GIC says about the valid range of IntIDs.
> >
> > Let's factor in this information when computing the number of
> > available LPIs
>
> On our D05 board, this limits the lpis to 2 and results in MSI irq
> alloc fails:
>
> [ 0.000000] ITS: Using hypervisor restricted LPI range [2]
> ....
> [ 10.543889] ixgbe 000a:11:00.1: Failed to allocate MSI interrupt, falling back to legacy. Error: -12
>
> > Signed-off-by: Marc Zyngier <[email protected]>
> > ---
> > drivers/irqchip/irq-gic-v3-its.c | 9 +++++++++
> > include/linux/irqchip/arm-gic-v3.h | 1 +
> > 2 files changed, 10 insertions(+)
> >
> > diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
> > index 8c7e8c235faf..903ca1c19553 100644
> > --- a/drivers/irqchip/irq-gic-v3-its.c
> > +++ b/drivers/irqchip/irq-gic-v3-its.c
> > @@ -1525,8 +1525,17 @@ static int free_lpi_range(u32 base, u32 nr_lpis)
> > static int __init its_lpi_init(u32 id_bits)
> > {
> > u32 lpis = (1UL << id_bits) - 8192;
> > + u32 numlpis;
> > int err;
> >
> > + numlpis = 1UL << GICD_TYPER_NUM_LPIS(gic_rdists->gicd_typer);
> > +
> > + if (numlpis > 1 && !WARN_ON(numlpis > lpis)) {
> > + lpis = numlpis;
> > + pr_info("ITS: Using hypervisor restricted LPI range [%u]\n",
> > + lpis);
> > + }
>
> I don't have the GICv3 extension doc, but did you intent to check for,
>
> if (numlpis > 2 && !WARN_ON(numlpis > lpis)) {
>
> as it looks like D05 returns 0 for bits 11-15 and that makes numlpis=2.
Absolutely. This really is a silly bug, thanks for catching that one.
I've pushed out an update on my irq/lpi-allocator branch.
Thanks,
M.
--
Jazz is not dead, it just smell funny.
Hi Marc
This patches is necessary for our device, thanks a lot for your patches.
We have done the tests for your patches on our prototype CPU chip.
All of tests's results are PASSED.
Below is the detail of our tests.
We did tests for 2 points.
point 1: No level down for existing device such as nvme, network interface card.
what we did: iozone benchmark on nvme, ssh command.
point 2: Our original device can work well.
what we did: Test set for our original device.
And we have done the review, we think the patches is no problem.
But we found a spelling mistake in you comments.
> + * The consequence of the above is that allocation is cost is low, but
I propose the following is correct.
+ * The consequence of the above is that allocation cost is low, but
Best Regards,
Lei Zhang
> -----Original Message-----
> From: Marc Zyngier [mailto:[email protected]]
> Sent: Wednesday, June 20, 2018 10:52 PM
> To: [email protected]
> Cc: Thomas Gleixner; Ard Biesheuvel; Shanker Donthineni; Shameer
> Kolothum; MaJun; Laurentiu Tudor; Zhang, Lei/張 雷
> Subject: [PATCH 0/7] irqchip/gic-v3: LPI allocation refactoring
>
> The GICv3 LPI allocator has served us well so far, but a number of new
> use cases have recently showed up:
>
> - A new extension to the GICv3 architecture allows a hypervisor to
> dramatically restrict the range of available LPIs. This means that
> our current policy of allocating LPIs in blocks of 32 may quickly
> deplete the number of devices that get LPIs
>
> - New and currently undisclosed busses seem to come with thousands of
> devices, each requiring a single LPI. Again, our current allocation
> policy means they quickly run out of LPIs.
>
> Simply expanding the bitmap doesn't seem to be a great idea, so let's
> change the LPI allocator altogether. This means we can move individual
> busses to a more minimal allocation scheme, though we only do it for
> PCI at the moment (Platform MSI looks like the Far West, and I'm
> clueless about the FSL MC thing).
>
> This is a pretty invasive change, and I'm thus cc'ing the usual
> suspects that have access to weird and wonderful HW to verify
> everything still works as expected, and let me know if we can relax
> the allocation for their own pet bus implementation.
>
> Only lightly tested in a KVM guest (PCI).
>
>
> Marc Zyngier (7):
> irqchip/gic-v3-its: Refactor LPI allocator
> irqchip/gic-v3-its: Use full range of LPIs
> irqchip/gic-v3-its: Move minimum LPI requirements to individual busses
> irqchip/gic-v3-its: Drop chunk allocation compatibility
> irqchip/gic-v3: Expose GICD_TYPER in the rdist structure
> irqchip/gic-v3-its: Honor hypervisor enforced LPI range
> irqchip/gic-v3-its: Reduce minimum LPI allocation to 1 for PCI devices
>
> drivers/irqchip/irq-gic-v3-its-fsl-mc-msi.c | 3 +
> drivers/irqchip/irq-gic-v3-its-pci-msi.c | 16 +-
> drivers/irqchip/irq-gic-v3-its-platform-msi.c | 2 +
> drivers/irqchip/irq-gic-v3-its.c | 225
> ++++++++++++------
> drivers/irqchip/irq-gic-v3.c | 4 +-
> include/linux/irqchip/arm-gic-v3.h | 3 +-
> 6 files changed, 169 insertions(+), 84 deletions(-)
>
> --
> 2.17.1
>
>