The current Multi MSI data programming fails if multiple end points
requesting MSI and multi MSI are connected with switch, i.e the current
multi MSI data being given is not considering the number of vectors
being requested in case of multi MSI.
Ex: Two EP's connected via switch, EP1 requesting single MSI first,
EP2 requesting Multi MSI of count four. The current code gives
MSI data 0x0 to EP1 and 0x1 to EP2, but EP2 can modify lower two bits
due to which EP2 also sends interrupt with MSI data 0x0 which results
in always invoking virq of EP1 due to which EP2 MSI interrupt never
gets handled.
Fix Multi MSI data programming with required alignment by
using number of vectors being requested.
Fixes: ab597d35ef11 ("PCI: xilinx-nwl: Add support for Xilinx NWL PCIe
Host Controller")
Signed-off-by: Bharat Kumar Gogada <[email protected]>
---
V4:
- Using a different bitmap registration API whcih serves single and multi
MSI requests.
---
drivers/pci/controller/pcie-xilinx-nwl.c | 11 +++++------
1 file changed, 5 insertions(+), 6 deletions(-)
diff --git a/drivers/pci/controller/pcie-xilinx-nwl.c b/drivers/pci/controller/pcie-xilinx-nwl.c
index 81538d7..a9e07b8 100644
--- a/drivers/pci/controller/pcie-xilinx-nwl.c
+++ b/drivers/pci/controller/pcie-xilinx-nwl.c
@@ -483,15 +483,13 @@ static int nwl_irq_domain_alloc(struct irq_domain *domain, unsigned int virq,
int i;
mutex_lock(&msi->lock);
- bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0,
- nr_irqs, 0);
- if (bit >= INT_PCI_MSI_NR) {
+ bit = bitmap_find_free_region(msi->bitmap, INT_PCI_MSI_NR,
+ get_count_order(nr_irqs));
+ if (bit < 0) {
mutex_unlock(&msi->lock);
return -ENOSPC;
}
- bitmap_set(msi->bitmap, bit, nr_irqs);
-
for (i = 0; i < nr_irqs; i++) {
irq_domain_set_info(domain, virq + i, bit + i, &nwl_irq_chip,
domain->host_data, handle_simple_irq,
@@ -509,7 +507,8 @@ static void nwl_irq_domain_free(struct irq_domain *domain, unsigned int virq,
struct nwl_msi *msi = &pcie->msi;
mutex_lock(&msi->lock);
- bitmap_clear(msi->bitmap, data->hwirq, nr_irqs);
+ bitmap_release_region(msi->bitmap, data->hwirq,
+ get_count_order(nr_irqs));
mutex_unlock(&msi->lock);
}
--
2.7.4
On Wed, 12 Jun 2019 11:17:59 +0100,
Bharat Kumar Gogada <[email protected]> wrote:
>
> The current Multi MSI data programming fails if multiple end points
> requesting MSI and multi MSI are connected with switch, i.e the current
> multi MSI data being given is not considering the number of vectors
> being requested in case of multi MSI.
> Ex: Two EP's connected via switch, EP1 requesting single MSI first,
> EP2 requesting Multi MSI of count four. The current code gives
> MSI data 0x0 to EP1 and 0x1 to EP2, but EP2 can modify lower two bits
> due to which EP2 also sends interrupt with MSI data 0x0 which results
> in always invoking virq of EP1 due to which EP2 MSI interrupt never
> gets handled.
I think there is a much simpler explanation for this: Multi-MSI
mandates that the base interrupt number is naturally aligned to its
size. Having switches in the middle is just a way to expose the issue,
but you could see it failing with a single end-point and two MSIs that
are assigned on an odd boundary.
>
> Fix Multi MSI data programming with required alignment by
> using number of vectors being requested.
>
> Fixes: ab597d35ef11 ("PCI: xilinx-nwl: Add support for Xilinx NWL PCIe
> Host Controller")
>
> Signed-off-by: Bharat Kumar Gogada <[email protected]>
> ---
> V4:
> - Using a different bitmap registration API whcih serves single and multi
> MSI requests.
> ---
> drivers/pci/controller/pcie-xilinx-nwl.c | 11 +++++------
> 1 file changed, 5 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/pci/controller/pcie-xilinx-nwl.c b/drivers/pci/controller/pcie-xilinx-nwl.c
> index 81538d7..a9e07b8 100644
> --- a/drivers/pci/controller/pcie-xilinx-nwl.c
> +++ b/drivers/pci/controller/pcie-xilinx-nwl.c
> @@ -483,15 +483,13 @@ static int nwl_irq_domain_alloc(struct irq_domain *domain, unsigned int virq,
> int i;
>
> mutex_lock(&msi->lock);
> - bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0,
> - nr_irqs, 0);
> - if (bit >= INT_PCI_MSI_NR) {
> + bit = bitmap_find_free_region(msi->bitmap, INT_PCI_MSI_NR,
> + get_count_order(nr_irqs));
> + if (bit < 0) {
> mutex_unlock(&msi->lock);
> return -ENOSPC;
> }
>
> - bitmap_set(msi->bitmap, bit, nr_irqs);
> -
> for (i = 0; i < nr_irqs; i++) {
> irq_domain_set_info(domain, virq + i, bit + i, &nwl_irq_chip,
> domain->host_data, handle_simple_irq,
> @@ -509,7 +507,8 @@ static void nwl_irq_domain_free(struct irq_domain *domain, unsigned int virq,
> struct nwl_msi *msi = &pcie->msi;
>
> mutex_lock(&msi->lock);
> - bitmap_clear(msi->bitmap, data->hwirq, nr_irqs);
> + bitmap_release_region(msi->bitmap, data->hwirq,
> + get_count_order(nr_irqs));
> mutex_unlock(&msi->lock);
> }
>
> --
> 2.7.4
>
As for the body of the patch:
Suggested-by: Marc Zyngier <[email protected]>
Acked-by: Marc Zyngier <[email protected]>
Thanks,
M.
--
Jazz is not dead, it just smells funny.
On Wed, Jun 12, 2019 at 02:01:56PM +0100, Marc Zyngier wrote:
> On Wed, 12 Jun 2019 11:17:59 +0100,
> Bharat Kumar Gogada <[email protected]> wrote:
> >
> > The current Multi MSI data programming fails if multiple end points
> > requesting MSI and multi MSI are connected with switch, i.e the current
> > multi MSI data being given is not considering the number of vectors
> > being requested in case of multi MSI.
> > Ex: Two EP's connected via switch, EP1 requesting single MSI first,
> > EP2 requesting Multi MSI of count four. The current code gives
> > MSI data 0x0 to EP1 and 0x1 to EP2, but EP2 can modify lower two bits
> > due to which EP2 also sends interrupt with MSI data 0x0 which results
> > in always invoking virq of EP1 due to which EP2 MSI interrupt never
> > gets handled.
>
> I think there is a much simpler explanation for this: Multi-MSI
> mandates that the base interrupt number is naturally aligned to its
> size. Having switches in the middle is just a way to expose the issue,
> but you could see it failing with a single end-point and two MSIs that
> are assigned on an odd boundary.
Agreed, I will rewrite the commit log with a link to the specs,
a switch has no role to play in this bug.
Lorenzo
> > Fix Multi MSI data programming with required alignment by
> > using number of vectors being requested.
> >
> > Fixes: ab597d35ef11 ("PCI: xilinx-nwl: Add support for Xilinx NWL PCIe
> > Host Controller")
> >
> > Signed-off-by: Bharat Kumar Gogada <[email protected]>
> > ---
> > V4:
> > - Using a different bitmap registration API whcih serves single and multi
> > MSI requests.
> > ---
> > drivers/pci/controller/pcie-xilinx-nwl.c | 11 +++++------
> > 1 file changed, 5 insertions(+), 6 deletions(-)
> >
> > diff --git a/drivers/pci/controller/pcie-xilinx-nwl.c b/drivers/pci/controller/pcie-xilinx-nwl.c
> > index 81538d7..a9e07b8 100644
> > --- a/drivers/pci/controller/pcie-xilinx-nwl.c
> > +++ b/drivers/pci/controller/pcie-xilinx-nwl.c
> > @@ -483,15 +483,13 @@ static int nwl_irq_domain_alloc(struct irq_domain *domain, unsigned int virq,
> > int i;
> >
> > mutex_lock(&msi->lock);
> > - bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0,
> > - nr_irqs, 0);
> > - if (bit >= INT_PCI_MSI_NR) {
> > + bit = bitmap_find_free_region(msi->bitmap, INT_PCI_MSI_NR,
> > + get_count_order(nr_irqs));
> > + if (bit < 0) {
> > mutex_unlock(&msi->lock);
> > return -ENOSPC;
> > }
> >
> > - bitmap_set(msi->bitmap, bit, nr_irqs);
> > -
> > for (i = 0; i < nr_irqs; i++) {
> > irq_domain_set_info(domain, virq + i, bit + i, &nwl_irq_chip,
> > domain->host_data, handle_simple_irq,
> > @@ -509,7 +507,8 @@ static void nwl_irq_domain_free(struct irq_domain *domain, unsigned int virq,
> > struct nwl_msi *msi = &pcie->msi;
> >
> > mutex_lock(&msi->lock);
> > - bitmap_clear(msi->bitmap, data->hwirq, nr_irqs);
> > + bitmap_release_region(msi->bitmap, data->hwirq,
> > + get_count_order(nr_irqs));
> > mutex_unlock(&msi->lock);
> > }
> >
> > --
> > 2.7.4
> >
>
> As for the body of the patch:
>
> Suggested-by: Marc Zyngier <[email protected]>
> Acked-by: Marc Zyngier <[email protected]>
>
> Thanks,
>
> M.
>
> --
> Jazz is not dead, it just smells funny.
On Wed, Jun 12, 2019 at 03:47:59PM +0530, Bharat Kumar Gogada wrote:
> The current Multi MSI data programming fails if multiple end points
> requesting MSI and multi MSI are connected with switch, i.e the current
> multi MSI data being given is not considering the number of vectors
> being requested in case of multi MSI.
> Ex: Two EP's connected via switch, EP1 requesting single MSI first,
> EP2 requesting Multi MSI of count four. The current code gives
> MSI data 0x0 to EP1 and 0x1 to EP2, but EP2 can modify lower two bits
> due to which EP2 also sends interrupt with MSI data 0x0 which results
> in always invoking virq of EP1 due to which EP2 MSI interrupt never
> gets handled.
>
> Fix Multi MSI data programming with required alignment by
> using number of vectors being requested.
>
> Fixes: ab597d35ef11 ("PCI: xilinx-nwl: Add support for Xilinx NWL PCIe
> Host Controller")
>
> Signed-off-by: Bharat Kumar Gogada <[email protected]>
> ---
> V4:
> - Using a different bitmap registration API whcih serves single and multi
> MSI requests.
> ---
> drivers/pci/controller/pcie-xilinx-nwl.c | 11 +++++------
> 1 file changed, 5 insertions(+), 6 deletions(-)
Applied to pci/xilinx for v5.3, please have a look and check if
the commit log I wrote provides a clear description of the issue.
Lorenzo
> diff --git a/drivers/pci/controller/pcie-xilinx-nwl.c b/drivers/pci/controller/pcie-xilinx-nwl.c
> index 81538d7..a9e07b8 100644
> --- a/drivers/pci/controller/pcie-xilinx-nwl.c
> +++ b/drivers/pci/controller/pcie-xilinx-nwl.c
> @@ -483,15 +483,13 @@ static int nwl_irq_domain_alloc(struct irq_domain *domain, unsigned int virq,
> int i;
>
> mutex_lock(&msi->lock);
> - bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0,
> - nr_irqs, 0);
> - if (bit >= INT_PCI_MSI_NR) {
> + bit = bitmap_find_free_region(msi->bitmap, INT_PCI_MSI_NR,
> + get_count_order(nr_irqs));
> + if (bit < 0) {
> mutex_unlock(&msi->lock);
> return -ENOSPC;
> }
>
> - bitmap_set(msi->bitmap, bit, nr_irqs);
> -
> for (i = 0; i < nr_irqs; i++) {
> irq_domain_set_info(domain, virq + i, bit + i, &nwl_irq_chip,
> domain->host_data, handle_simple_irq,
> @@ -509,7 +507,8 @@ static void nwl_irq_domain_free(struct irq_domain *domain, unsigned int virq,
> struct nwl_msi *msi = &pcie->msi;
>
> mutex_lock(&msi->lock);
> - bitmap_clear(msi->bitmap, data->hwirq, nr_irqs);
> + bitmap_release_region(msi->bitmap, data->hwirq,
> + get_count_order(nr_irqs));
> mutex_unlock(&msi->lock);
> }
>
> --
> 2.7.4
>
>
> On Wed, Jun 12, 2019 at 03:47:59PM +0530, Bharat Kumar Gogada wrote:
> > The current Multi MSI data programming fails if multiple end points
> > requesting MSI and multi MSI are connected with switch, i.e the
> > current multi MSI data being given is not considering the number of
> > vectors being requested in case of multi MSI.
> > Ex: Two EP's connected via switch, EP1 requesting single MSI first,
> > EP2 requesting Multi MSI of count four. The current code gives MSI
> > data 0x0 to EP1 and 0x1 to EP2, but EP2 can modify lower two bits due
> > to which EP2 also sends interrupt with MSI data 0x0 which results in
> > always invoking virq of EP1 due to which EP2 MSI interrupt never gets
> > handled.
> >
> > Fix Multi MSI data programming with required alignment by using number
> > of vectors being requested.
> >
> > Fixes: ab597d35ef11 ("PCI: xilinx-nwl: Add support for Xilinx NWL PCIe
> > Host Controller")
> >
> > Signed-off-by: Bharat Kumar Gogada <[email protected]>
> > ---
> > V4:
> > - Using a different bitmap registration API whcih serves single and multi
> > MSI requests.
> > ---
> > drivers/pci/controller/pcie-xilinx-nwl.c | 11 +++++------
> > 1 file changed, 5 insertions(+), 6 deletions(-)
>
> Applied to pci/xilinx for v5.3, please have a look and check if the commit log
> I wrote provides a clear description of the issue.
>
> Lorenzo
Thanks Lorenzo and Marc.
Lorenzo, can you please point to link for above commit.
Regards,
Bharat
> > diff --git a/drivers/pci/controller/pcie-xilinx-nwl.c
> > b/drivers/pci/controller/pcie-xilinx-nwl.c
> > index 81538d7..a9e07b8 100644
> > --- a/drivers/pci/controller/pcie-xilinx-nwl.c
> > +++ b/drivers/pci/controller/pcie-xilinx-nwl.c
> > @@ -483,15 +483,13 @@ static int nwl_irq_domain_alloc(struct
> irq_domain *domain, unsigned int virq,
> > int i;
> >
> > mutex_lock(&msi->lock);
> > - bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0,
> > - nr_irqs, 0);
> > - if (bit >= INT_PCI_MSI_NR) {
> > + bit = bitmap_find_free_region(msi->bitmap, INT_PCI_MSI_NR,
> > + get_count_order(nr_irqs));
> > + if (bit < 0) {
> > mutex_unlock(&msi->lock);
> > return -ENOSPC;
> > }
> >
> > - bitmap_set(msi->bitmap, bit, nr_irqs);
> > -
> > for (i = 0; i < nr_irqs; i++) {
> > irq_domain_set_info(domain, virq + i, bit + i, &nwl_irq_chip,
> > domain->host_data, handle_simple_irq, @@
> -509,7 +507,8 @@ static
> > void nwl_irq_domain_free(struct irq_domain *domain, unsigned int virq,
> > struct nwl_msi *msi = &pcie->msi;
> >
> > mutex_lock(&msi->lock);
> > - bitmap_clear(msi->bitmap, data->hwirq, nr_irqs);
> > + bitmap_release_region(msi->bitmap, data->hwirq,
> > + get_count_order(nr_irqs));
> > mutex_unlock(&msi->lock);
> > }
> >
> > --
> > 2.7.4
> >
On Tue, Jun 18, 2019 at 12:28:02PM +0000, Bharat Kumar Gogada wrote:
[...]
> > Applied to pci/xilinx for v5.3, please have a look and check if the commit log
> > I wrote provides a clear description of the issue.
> >
> > Lorenzo
> Thanks Lorenzo and Marc.
> Lorenzo, can you please point to link for above commit.
I did already.
Anyway:
https://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/pci.git/commit/?h=pci/xilinx&id=46c1bfcfcd873f8754f733e4258121748bcae3a3
> Regards,
> Bharat
> > > diff --git a/drivers/pci/controller/pcie-xilinx-nwl.c
> > > b/drivers/pci/controller/pcie-xilinx-nwl.c
> > > index 81538d7..a9e07b8 100644
> > > --- a/drivers/pci/controller/pcie-xilinx-nwl.c
> > > +++ b/drivers/pci/controller/pcie-xilinx-nwl.c
> > > @@ -483,15 +483,13 @@ static int nwl_irq_domain_alloc(struct
> > irq_domain *domain, unsigned int virq,
> > > int i;
> > >
> > > mutex_lock(&msi->lock);
> > > - bit = bitmap_find_next_zero_area(msi->bitmap, INT_PCI_MSI_NR, 0,
> > > - nr_irqs, 0);
> > > - if (bit >= INT_PCI_MSI_NR) {
> > > + bit = bitmap_find_free_region(msi->bitmap, INT_PCI_MSI_NR,
> > > + get_count_order(nr_irqs));
> > > + if (bit < 0) {
> > > mutex_unlock(&msi->lock);
> > > return -ENOSPC;
> > > }
> > >
> > > - bitmap_set(msi->bitmap, bit, nr_irqs);
> > > -
> > > for (i = 0; i < nr_irqs; i++) {
> > > irq_domain_set_info(domain, virq + i, bit + i, &nwl_irq_chip,
> > > domain->host_data, handle_simple_irq, @@
> > -509,7 +507,8 @@ static
> > > void nwl_irq_domain_free(struct irq_domain *domain, unsigned int virq,
> > > struct nwl_msi *msi = &pcie->msi;
> > >
> > > mutex_lock(&msi->lock);
> > > - bitmap_clear(msi->bitmap, data->hwirq, nr_irqs);
> > > + bitmap_release_region(msi->bitmap, data->hwirq,
> > > + get_count_order(nr_irqs));
> > > mutex_unlock(&msi->lock);
> > > }
> > >
> > > --
> > > 2.7.4
> > >