Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Mon, 30 Aug 2021 21:20:17 +0100
Message-ID: <87o89ed6ri.wl-maz@kernel.org>
From:   Marc Zyngier <maz@kernel.org>
To:     Rob Herring <robh+dt@kernel.org>
Cc:     Mark Kettenis <mark.kettenis@xs4all.nl>,
        devicetree@vger.kernel.org,
        Alyssa Rosenzweig <alyssa@rosenzweig.io>,
        Mark Kettenis <kettenis@openbsd.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Hector Martin <marcan@marcan.st>,
        Bjorn Helgaas <bhelgaas@google.com>,
        Nicolas Saenz Julienne <nsaenz@kernel.org>,
        Jim Quinlan <jim2101024@gmail.com>,
        Florian Fainelli <f.fainelli@gmail.com>,
        "maintainer:BROADCOM BCM7XXX ARM ARCHITECTURE" 
        <bcm-kernel-feedback-list@broadcom.com>,
        Daire McNamara <daire.mcnamara@microchip.com>,
        Saenz Julienne <nsaenzjulienne@suse.de>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        linux-arm-kernel <linux-arm-kernel@lists.infradead.org>,
        PCI <linux-pci@vger.kernel.org>,
        "moderated list:BROADCOM BCM2835 ARM ARCHITECTURE" 
        <linux-rpi-kernel@lists.infradead.org>
Subject: Re: [PATCH v4 4/4] arm64: apple: Add PCIe node
In-Reply-To: <CAL_JsqJC+FxiynFkkcB0amp3s4agsio5ggCrYiPbqoXroAJV4Q@mail.gmail.com>
References: <20210827171534.62380-1-mark.kettenis@xs4all.nl>
        <20210827171534.62380-5-mark.kettenis@xs4all.nl>
        <87pmtvcgec.wl-maz@kernel.org>
        <CAL_JsqJC+FxiynFkkcB0amp3s4agsio5ggCrYiPbqoXroAJV4Q@mail.gmail.com>
User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue)
 FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1
 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO)
MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue")
Content-Type: text/plain; charset=US-ASCII
Precedence: bulk

On Mon, 30 Aug 2021 16:57:59 +0100,
Rob Herring <robh+dt@kernel.org> wrote:
> 
> On Mon, Aug 30, 2021 at 6:37 AM Marc Zyngier <maz@kernel.org> wrote:
> >
> > I have now implemented the MSI change on the Linux driver side, and it
> > works nicely. So thumbs up from me on this front.
> >
> > I am now looking at the interrupts provided by each port:
> > (1) a bunch of port-private interrupts (link up/down...)
> > (2) INTx interrupts
> 
> So each port has an independent INTx space?

Yes.

> Is that even something PCI defines or comprehends?

Can't see why not. That's no different from having several PCI busses.
I don't think anything enforces that INTx interrupts have to be
unique across the system. As long as they are unique across a PCI
hierarchy, we should be OK.

> 
> > Given that the programming is per-port, I've implemented this as a
> > per-port interrupt controller.
> >
> > (1) is dead easy to implement, and doesn't require any DT description.
> > (2) is unfortunately exposing the limits of my DT knowledge, and I'm
> > not clear how to model it. I came up with the following:
> >
> >         port00: pci@0,0 {
> >                 device_type = "pci";
> >                 reg = <0x0 0x0 0x0 0x0 0x0>;
> >                 reset-gpios = <&pinctrl_ap 152 0>;
> >                 max-link-speed = <2>;
> >
> >                 #address-cells = <3>;
> >                 #size-cells = <2>;
> >                 ranges;
> >
> >                 interrupt-controller;
> >                 #interrupt-cells = <1>;
> >                 interrupt-parent = <&port00>;
> >                 interrupt-map-mask = <0 0 0 7>;
> >                 interrupt-map = <0 0 0 1 &port00 0>,
> >                                 <0 0 0 2 &port00 1>,
> >                                 <0 0 0 3 &port00 2>,
> >                                 <0 0 0 4 &port00 3>;
> 
> IIRC, I don't think the DT IRQ code handles a node having both
> 'interrupt-controller' and 'interrupt-map' properties.

Indeed, and that actually explains why the damned INTx interrupts
insist on being 1-based instead of 0-based as the above mapping
attempts to describe it. Turns out I can rip the interrupt-map out and
it isn't worse.

> I think that's why some PCI host bridge nodes have child
> interrupt-controller nodes.  I don't really like that work-around,
> so if the above can be made to work, I'd be happy to see it. But the
> DT IRQ code is some ancient code for ancient platforms (PowerMacs
> being one of them).

That'd probably need some massaging. I'll have a look. I checked that
if I add something like:

		interrupts-extended = <&port02 2>;

to each port, I get the PME interrupt correctly assigned should I pass
pcie_pme=nomsi. Given that this IP is pretty limited in terms of MSIs,
every bit that can free a MSI is welcome.

I guess that it would make sense to expand this support to also match
for an interrupt-map.

> 
> >         };
> >
> > which vaguely seem to do the right thing for the devices behind root
> > ports, but doesn't seem to work for INTx generated by the root ports
> > themselves. Any clue? Alternatively, I could move it to something
> > global to the whole PCIe controller, but that doesn't seem completely
> > right.

I've investigated this one further, and it looks like the DT IRQ code
insists on trying to find the interrupt in the main pcie node instead
of in the root port itself. But of course it doesn't want to parse an
interrupt-map at that level either.

I guess that's related to the above.

> >
> > It also begs the question whether the per-port interrupt to the AIC
> > should be moved into each root port, should my per-port approach hold
> > any water.
> 
> I tend to think per-port is the right thing to do. However, the child
> nodes are PCI devices, so that creates some restrictions. Such as the
> per port registers are in the host address space, not the PCI address
> space, so we can't move the registers into the child nodes. The
> interrupts may be okay. Certainly, being an 'interrupt-controller'
> without having an 'interrupts' property for an non root interrupt
> controller is odd.

That was my own impression as well.

I guess there is no real canonical way to handle this particular
system and to fully support it, we'll have to amend the current
infrastructure. The question is: what is the least ugly way to express
this that will work reasonably across implementations (OpenBSD, Linux,
u-boot)?

	M.

-- 
Without deviation from the norm, progress is not possible.