Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
From:   =?UTF-8?Q?Lu=C3=ADs_Matallui?= <matallui@gmail.com>
Date:   Thu, 2 Apr 2020 19:35:01 -0600
Message-ID: <CAE3Pknn88NZNqiaHahM4HVvTU82hdMcbhGWWb4AyXXYMbcGuaQ@mail.gmail.com>
Subject: Help with IRQ-MSI-IRQ bridges
To:     linux-kernel@vger.kernel.org
Cc:     maz@kernel.org
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

Hi,

I've got this SoC which uses IRQ-MSI and MSI-IRQ bridges in order to
get interrupts from devices external to the ARM subsystem.
I already got some pointers from Maz and have been able to create the
drivers with the stacked domains and can now see the mappings working
fine across domains.

Maz pointed me to the Marvell mvebu-gicp (for my MSI controller, a.k.a
MSI-IRQ bridge) and to mvebu-icu for the MSI client (IRQ-MSI bridge).

I now have the interrupts working, but it seems like I'm missing a
bunch of them. And therefore my device doesn't work properly.
The main difference between my HW and Marvell's is that my IRQs are
not level-triggered and the MSIs don't support the two messages for
level-triggered interrupts.

To illustrate my system:

DEV --line--> IRQ-MSI Bridge (MSIC) --msi--> MSI-IRQ Bridge (GICP)
--line--> GICv2

For MSIC, all I can do is configure the address and data for the MSI,
and I believe on every rising edge of the Device IRQ, an MSI is sent.
For GICP, all I have is a doorbell and a way to enable/disable it, and
whenever the doorbell is enabled and has a value != 0, the IRQ line to
GICv2 gets asserted.

The first thing I noticed is that when I get an interrupt, the IRQ
flow goes like:

  handle_irq();
  irq_eoi();

So, I guess my first question here is, how can I guarantee that I
don't get another MSI whilst in handle_irq()?
If I do, then I will clear the doorbell on irq_eoi() (because that's
my only choice) and will lose the queued IRQs.

It also seems that I'm missing IRQs in the beginning after probing the
device, and before it was working for me when I was setting up all
these registers manually and simply using GICv2 as my only interrupt
controller.
I do see the unmask() ops being called for all my stacked irqchips, so
I don't understand how I'm missing so many interrupts.

Any help would be truly appreciated.

Thanks,
ltanica