Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <b8b62753ad5235e065b4cb0856a7a7c33438dfbb.camel@svanheule.net>
Subject: Re: [PATCH v1 3/6] gpio: realtek-otto: Support per-cpu interrupts
From:   Sander Vanheule <sander@svanheule.net>
To:     Marc Zyngier <maz@kernel.org>,
        Linus Walleij <linus.walleij@linaro.org>
Cc:     linux-gpio@vger.kernel.org, devicetree@vger.kernel.org,
        Bartosz Golaszewski <brgl@bgdev.pl>,
        Rob Herring <robh+dt@kernel.org>,
        Krzysztof Kozlowski <krzk+dt@kernel.org>,
        Bert Vermeulen <bert@biot.com>, linux-kernel@vger.kernel.org
Date:   Fri, 22 Apr 2022 09:04:20 +0200
In-Reply-To: <87h76mahsl.wl-maz@kernel.org>
References: <cover.1649533972.git.sander@svanheule.net>
         <8d4e0848f233c2c1b98aa141741c61d95cd3843f.1649533972.git.sander@svanheule.net>
         <CACRpkdbSdDAKiFAsHBosdVDpBhWW-Keoq+t8GJ5LsyWjOZwp_g@mail.gmail.com>
         <87h76mahsl.wl-maz@kernel.org>
Content-Type: text/plain; charset="UTF-8"
User-Agent: Evolution 3.42.4 (3.42.4-2.fc35) 
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk

Hi Linus, Marc,

On Thu, 2022-04-21 at 10:48 +0100, Marc Zyngier wrote:
> On Thu, 21 Apr 2022 00:04:16 +0100,
> Linus Walleij <linus.walleij@linaro.org> wrote:
> > 
> > On Sat, Apr 9, 2022 at 9:56 PM Sander Vanheule <sander@svanheule.net> wrote:
> > 
> > > On SoCs with multiple cores, it is possible that the GPIO interrupt
> > > controller supports assigning specific pins to one or more cores.
> > > 
> > > IRQ balancing can be performed on a line-by-line basis if the parent
> > > interrupt is routed to all available cores, which is the default upon
> > > initialisation.
> > > 
> > > Signed-off-by: Sander Vanheule <sander@svanheule.net>
> > 
> > That sounds complicated.
> > 
> > Sounds like something the IRQ maintainer (Marc Z) should
> > have a quick look at.
> 
> This is pretty odd indeed. There seem to be a direct mapping between
> the GPIOs and the CPU it interrupts (or at least that's what the code
> seem to express). However, I don't see a direct relation between the
> CPUs and the chained interrupt. It isn't even clear if this interrupt
> itself is per-CPU.
> 
> So this begs a few questions:
> 
> - is the affinity actually affecting the target CPU? or is it
>   affecting the target mux?
> 
> - how is the affinity of the mux interrupt actually enforced?

There are three interrupt controllers at play here:
   1. MIPS CPU interrupt controller: drivers/irqchip/irq-mips-cpu.c
      One interrupt controller per VPE, so in this case there are two. Provides
      per-CPU interrupts.
   2. SoC interrupt controller: drivers/irqchip/irq-realtek-rtl.c
      Also one interrupt controller per VPE. I suppose these will also be per-
      CPU, although this isn't implemented in the driver yet, and I don't think
      I yet fully understand how should work in the kernel.
   3. GPIO interrupt controller: drivers/gpio/gpio-realtek-otto.c
      One interrupt controller for the entire GPIO bank, with optional
      configurable affinity (this patch) for the different VPEs.

For the RTL839x series of SoCs, this results in the following:

GPIO LINES SOC IRQ MIPS
+--------+ +-----------+ HW IRQ +--------+
--->| GPIO | | SOC IRQ | LINES | IRQ |
--->| BANK |-----o-->| VPE0 CTRL |=========>| VPE0 |
. | | | +-----------+ +--------+
. +--------+ | 
. |
| +-----------+ +--------+
\-->| SOC IRQ | | IRQ |
| VPE1 CTRL |=========>| VPE1 |
+-----------+ +--------+


For RTL930x, where GPIO IRQ affinity is configurable:

GPIO LINES SOC IRQ MIPS
+--------+ +-----------+ HW IRQ +--------+
--->| GPIO |-------->| SOC IRQ | LINES | IRQ |
--->| BANK | | VPE0 CTRL |=========>| VPE0 |
. | |-----\ +-----------+ +--------+
. +--------+ | 
. |
| +-----------+ +--------+
\-->| SOC IRQ | | IRQ |
| VPE1 CTRL |=========>| VPE1 |
+-----------+ +--------+

The interrupt for the GPIO controller can be muxed to any of the MIPS HW
interrupts on any (or all) of the VPEs, and these muxes (SoC IRQ controllers)
can be configured independently per CPU. The SoC IRQ line index is fixed, and
consistent for both VPEs.
Only in the second diagram can individual GPIO interrupts be muxed to any of the
VPEs, but there is still only one IRQ line per VPE for all selected GPIO lines.

I hopes this helps to clarify the situation. We don't have any real
documentation, so this is basically derived from registers descriptions in SDK
headers and testing the interrupt behaviour.

Best,
Sander