Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Tue, 13 Apr 2021 10:23:12 +0100
Message-ID: <871rbeo7wf.wl-maz@kernel.org>
From:   Marc Zyngier <maz@kernel.org>
To:     Peter Geis <pgwipeout@gmail.com>
Cc:     Thomas Gleixner <tglx@linutronix.de>,
        "open list:ARM/Rockchip SoC..." <linux-rockchip@lists.infradead.org>,
        linux-kernel@vger.kernel.org
Subject: Re: [RFC] ITS fails to allocate on rk3568/rk3566
In-Reply-To: <CAMdYzYrQ5f-mv_VmTq_CRf9tR=j3mwRpKHNLmPFgCF9whsGFRw@mail.gmail.com>
References: <CAMdYzYrQ5f-mv_VmTq_CRf9tR=j3mwRpKHNLmPFgCF9whsGFRw@mail.gmail.com>
User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue)
 FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1
 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO)
MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue")
Content-Type: text/plain; charset=US-ASCII
Precedence: bulk

Hi Peter,

On Mon, 12 Apr 2021 21:49:59 +0100,
Peter Geis <pgwipeout@gmail.com> wrote:
> 
> Good Afternoon,
> 
> I am assisting with early bringup of the rk3566 based quartz64
> development board for mainline linux.
> I've encountered a few issues with allocating ITS on their version of
> the GIC-V3.
> The first issue is the ITS controller can only use 32bit addresses.
> This leads to the following error:
> [    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
> [    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
> [    0.000000] GICv3: 320 SPIs implemented
> [    0.000000] GICv3: 0 Extended SPIs implemented
> [    0.000000] GICv3: Distributor has no Range Selector support
> [    0.000000] Root IRQ handler: gic_handle_irq
> [    0.000000] GICv3: 16 PPIs implemented
> [    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000fd460000
> [    0.000000] ITS [mem 0xfd440000-0xfd45ffff]
> [    0.000000] ITS@0x00000000fd440000: Devices doesn't stick:
> f907000100190600 f907000000190600

Ouch. That looks pretty bad. Bit 32 of the register doesn't stick, and
that's right in the middle of the address. The register should be
fully writable as far as the address field is concerned.

Please dump the distributor and ITS IIDR registers so that I can find
the TRM for the exact IP.

> [    0.000000] ITS@0x00000000fd440000: failed probing (-6)
> [    0.000000] ITS: No ITS available, not enabling LPIs
> 
> Downstream fixed this by adding the GFP_DMA32 flag to the memory
> allocations.

Urgh... this really looks like broken silicon to me.

> They also force clear the GITS_BASER_SHAREABILITY_MASK.

Why? Does this also apply to the command queue? Are they forcing cache
flushing?

> Unfortunately while this allowed ITS to allocate on downstream, as
> soon as MSIs attempted to use it all interrupts would time out.
> 
> On upstream, we observe this during allocation:
> [    0.000000] ITS [mem 0xfd440000-0xfd45ffff]
> [    0.000000] ITS@0x00000000fd440000: allocated 8192 Devices @3810000
> (indirect, esz 8, psz 64K, shr 1)
> [    0.000000] ITS@0x00000000fd440000: allocated 32768 Interrupt
> Collections @3820000 (flat, esz 2, psz 64K, shr 1)
> [    0.000000] GICv3: using LPI property table @0x0000000003830000
> [    0.000000] GICv3: CPU0: using allocated LPI pending table
> @0x0000000003840000
> [    0.000000] ITS queue timeout (64 1)
> [    0.000000] ITS cmd its_build_mapc_cmd failed
> [    0.000000] ITS queue timeout (96 1)
> [    0.000000] ITS cmd its_build_invall_cmd failed
> <snip>

So the command queue is not making forward progress. Either because
the ITS cannot access the commands, or because it cannot use the
memory it has been allocated. Please dump GITS_CBASER (and the value
that has been written to it), just in case it shows the same
brokenness as the GITS_BASER registers...

[...]

> Any assistance you can provide would be greatly appreciated.

I'm not sure there is much we can do without a lot more details about
the HW. We need to know the exact GIC implementation they are using
(ARM has two versions of the GICv3 IP), and we also need to find out
*how* this has been integrated. Only Rockchip can tell you that.

Once we know which version they are using, and how it is wired, we can
start looking at some IMPDEF debug registers.

The Linux driver assumes that the ITS is able to access the whole
memory. If there are restrictions on what memory ranges can be used,
it is going to be a pain to support :-(.

	M.

-- 
Without deviation from the norm, progress is not possible.