Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8;
 format=flowed
Content-Transfer-Encoding: 8bit
Date:   Fri, 03 Apr 2020 11:18:43 +0100
From:   Marc Zyngier <maz@kernel.org>
To:     =?UTF-8?Q?Lu=C3=ADs_Matallui?= <matallui@gmail.com>
Cc:     linux-kernel@vger.kernel.org
Subject: Re: Help with IRQ-MSI-IRQ bridges
In-Reply-To: <CAE3Pknn88NZNqiaHahM4HVvTU82hdMcbhGWWb4AyXXYMbcGuaQ@mail.gmail.com>
References: <CAE3Pknn88NZNqiaHahM4HVvTU82hdMcbhGWWb4AyXXYMbcGuaQ@mail.gmail.com>
Message-ID: <3ebbe879471898218f34918b1d009887@kernel.org>
User-Agent: Roundcube Webmail/1.3.10
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

Hi Luis,

On 2020-04-03 02:35, Luís Matallui wrote:
> Hi,
> 
> I've got this SoC which uses IRQ-MSI and MSI-IRQ bridges in order to
> get interrupts from devices external to the ARM subsystem.
> I already got some pointers from Maz and have been able to create the
> drivers with the stacked domains and can now see the mappings working
> fine across domains.
> 
> Maz pointed me to the Marvell mvebu-gicp (for my MSI controller, a.k.a
> MSI-IRQ bridge) and to mvebu-icu for the MSI client (IRQ-MSI bridge).
> 
> I now have the interrupts working, but it seems like I'm missing a
> bunch of them. And therefore my device doesn't work properly.
> The main difference between my HW and Marvell's is that my IRQs are
> not level-triggered and the MSIs don't support the two messages for
> level-triggered interrupts.

Which is probably a very good thing, as long as all your devices 
generate
only edge-triggered interrupts.

> 
> To illustrate my system:
> 
> DEV --line--> IRQ-MSI Bridge (MSIC) --msi--> MSI-IRQ Bridge (GICP)
> --line--> GICv2
> 
> For MSIC, all I can do is configure the address and data for the MSI,
> and I believe on every rising edge of the Device IRQ, an MSI is sent.
> For GICP, all I have is a doorbell and a way to enable/disable it, and
> whenever the doorbell is enabled and has a value != 0, the IRQ line to
> GICv2 gets asserted.
> 
> The first thing I noticed is that when I get an interrupt, the IRQ
> flow goes like:
> 
>   handle_irq();
>   irq_eoi();
> 
> So, I guess my first question here is, how can I guarantee that I
> don't get another MSI whilst in handle_irq()?

At the GIC level, once the interrupt is Ack'd, anything that is signed
after this ack is a separate interrupt. It will be made pending and will
fire once the GIC driver EOIs the first one.

> If I do, then I will clear the doorbell on irq_eoi() (because that's
> my only choice) and will lose the queued IRQs.

Why do you need to do anything at the doorbell level? This is just a 
write,
so there should be nothing to clear. If you do need to clear anything,
then your MSI-IRQ bridge isn't stateless as it should, and you'll need 
to
give much more details about the HW. Do you have a pointer to the TRM
for your HW?

> It also seems that I'm missing IRQs in the beginning after probing the
> device, and before it was working for me when I was setting up all
> these registers manually and simply using GICv2 as my only interrupt
> controller.

Well, setting all of this in firmware is always the preferred option
if you don't expect things to change dynamically.

> I do see the unmask() ops being called for all my stacked irqchips, so
> I don't understand how I'm missing so many interrupts.

unmask is just a static configuration to enable the interrupt. There 
shouldn't
be that many calls to that later on unless an endpoint driver 
disables/enables
interrupts by hand.

Please give us a bit more details to understand the context, as there is 
only
so much I can do with so little HW information.

Thanks,

         M.
-- 
Jazz is not dead. It just smells funny...