Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
Date:   Fri, 22 Apr 2022 11:27:00 +0200
From:   Pali =?utf-8?B?Um9ow6Fy?= <pali@kernel.org>
To:     Chris Packham <Chris.Packham@alliedtelesis.co.nz>
Cc:     "jason@lakedaemon.net" <jason@lakedaemon.net>,
        Joshua Scott <Joshua.Scott@alliedtelesis.co.nz>,
        "gregory.clement@bootlin.com" <gregory.clement@bootlin.com>,
        "andrew@lunn.ch" <andrew@lunn.ch>,
        "sebastian.hesselbarth@gmail.com" <sebastian.hesselbarth@gmail.com>,
        "linux-arm-kernel@lists.infradead.org" 
        <linux-arm-kernel@lists.infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH] ARM: mvebu: Enable MBUS error propagation
Message-ID: <20220422092700.3j7boazeubccdhcr@pali>
References: <20191124093529.32399-1-chris.packham@alliedtelesis.co.nz>
 <8736cqb63d.fsf@FE-laptop>
 <0285a09b2b1b7ae2ccc268a37e4357c95d270ac9.camel@alliedtelesis.co.nz>
 <20210603125525.nkswvixbabkgq5or@pali>
 <e75b74fa-b393-fede-1a19-fd43dbb7aba4@alliedtelesis.co.nz>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <e75b74fa-b393-fede-1a19-fd43dbb7aba4@alliedtelesis.co.nz>
User-Agent: NeoMutt/20180716
Precedence: bulk

On Thursday 03 June 2021 21:03:55 Chris Packham wrote:
> On 4/06/21 12:55 am, Pali Rohár wrote:
> > On Wednesday 08 January 2020 19:42:12 Chris Packham wrote:
> >> Hi Gregory,
> >>
> >> On Wed, 2020-01-08 at 11:22 +0100, Gregory CLEMENT wrote:
> >>> Hello Chris,
> >>>
> >>>> U-boot disables MBUS error propagation for Armada-385. The effect of
> >>>> this on the kernel is that any access to a mapped but inaccessible
> >>>> address causes the system to hang.
> >>>>
> >>>> By enabling MBUS error propagation the kernel can raise a Bus Error and
> >>>> panic to restart the system.
> >>> Unless I miss it, it seems that nobody comment this patch: sorry for the
> >>> delay.
> >>>
> >> Thanks for the response.
> >>
> >>>> Signed-off-by: Chris Packham <chris.packham@alliedtelesis.co.nz>
> >>>> ---
> >>>>
> >>>> Notes:
> >>>>      We've encountered an issue where rogue accesses to PCI-e space cause an
> >>>>      Armada-385 system to lockup. We've found that enabling MBUS error
> >>>>      propagation lets us get a bus error which at least gives us a panic to
> >>>>      help identify what was accessed.
> >>>>      
> >>>>      U-boot clears the IO Err Prop Enable Bit[1] but so far no-one seems to
> >>>>      know why.
> >>>>      
> >>>>      I wasn't sure where to put this code. There is similar code for kirwood
> >>>>      in the equivalent dt_init function. On Armada-XP the register is part of
> >>>>      the Core Coherency Fabric block (for A385 it's documented as part of the
> >>>>      CCF block).
> >>> What about adding a new set of register to the mvebu mbus driver?
> >>>
> >> After more testing we found that some previously "good" boards started
> >> throwing up panics with this change. I think that this might require
> >> handling some of the PCI-e interrupts (for correctable errors) via the
> >> EDAC subsystem.
> >>
> >> We're still working with Marvell to track down exactly why this is
> >> happening on our system.
> > Hello Chris! Have you somehow solved this issue? Or do you have some
> > contacts in Marvell for A385 PCIe subsystem?
> 
> The problem seemed to be a specific CPU Erratum for the A385 which I 
> brought into upstream u-boot[1].

Hello Chris, could you point me to this CPU Erratum? What is its number
or some other identifier? Because I'm unable to find any such erratum in
the list for A385 erratums.

> But even then we had to update our base 
> u-boot version from 2016.11 to 2019.10 so there may have been more going 
> on than just that one change.
> 
> [1] - 
> https://source.denx.de/u-boot/u-boot/-/commit/ad91fdfff0bd6ea471afe838e0f6d58ed898694e
> 
> >>> In this case it will be called even earlier allowing to see bus error
> >>> earlier.
> >>>
> >>> In any case, you should separate the device tree change from the code
> >>> change and at least provide 2 patches.
> >> Agreed. If/when something solid eventuates we'll do it as a proper
> >> series.
> >>
> >>> Gregory
> >>>
> >>>>      
> >>>>      --
> >>>>      [1] - https://scanmail.trustwave.com/?c=20988&d=yNG44EMQR-PfAPR99zig1LzptmqzHo11i7uAj4AGuQ&u=https%3a%2f%2fgitlab%2edenx%2ede%2fu-boot%2fu-boot%2fblob%2fmaster%2farch%2farm%2fmach-mvebu%2fcpu%2ec%23L489
> ^^^ sorry, as a result of a reasonably high profile cyber attack on a 
> local institution URLs in our incoming mail are intercepted.
> >>>>
> >>>>   arch/arm/boot/dts/armada-38x.dtsi |  5 +++++
> >>>>   arch/arm/mach-mvebu/board-v7.c    | 27 +++++++++++++++++++++++++++
> >>>>   2 files changed, 32 insertions(+)
> >>>>
> >>>> diff --git a/arch/arm/boot/dts/armada-38x.dtsi b/arch/arm/boot/dts/armada-38x.dtsi
> >>>> index 3f4bb44d85f0..3214c67433eb 100644
> >>>> --- a/arch/arm/boot/dts/armada-38x.dtsi
> >>>> +++ b/arch/arm/boot/dts/armada-38x.dtsi
> >>>> @@ -386,6 +386,11 @@
> >>>>   				      <0x20250 0x8>;
> >>>>   			};
> >>>>   
> >>>> +			ioerrc: io-err-control@20200 {
> >>>> +				compatible = "marvell,io-err-control";
> >>>> +				reg = <0x20200 0x4>;
> >>>> +			};
> >>>> +
> >>>>   			mpic: interrupt-controller@20a00 {
> >>>>   				compatible = "marvell,mpic";
> >>>>   				reg = <0x20a00 0x2d0>, <0x21070 0x58>;
> >>>> diff --git a/arch/arm/mach-mvebu/board-v7.c b/arch/arm/mach-mvebu/board-v7.c
> >>>> index d2df5ef9382b..fb7718386ef9 100644
> >>>> --- a/arch/arm/mach-mvebu/board-v7.c
> >>>> +++ b/arch/arm/mach-mvebu/board-v7.c
> >>>> @@ -138,10 +138,36 @@ static void __init i2c_quirk(void)
> >>>>   	}
> >>>>   }
> >>>>   
> >>>> +#define MBUS_ERR_PROP_EN BIT(8)
> >>>> +
> >>>> +/*
> >>>> + * U-boot disables MBUS error propagation. Re-enable it so we
> >>>> + * can handle them as Bus Errors.
> >>>> + */
> >>>> +static void __init enable_mbus_error_propagation(void)
> >>>> +{
> >>>> +	struct device_node *np =
> >>>> +		of_find_compatible_node(NULL, NULL, "marvell,io-err-control");
> >>>> +
> >>>> +	if (np) {
> >>>> +		void __iomem *reg;
> >>>> +
> >>>> +		reg = of_iomap(np, 0);
> >>>> +		if (reg) {
> >>>> +			u32 val;
> >>>> +
> >>>> +			val = readl_relaxed(reg);
> >>>> +			writel_relaxed(val | MBUS_ERR_PROP_EN, reg);
> >>>> +		}
> >>>> +		of_node_put(np);
> >>>> +	}
> >>>> +}
> >>>> +
> >>>>   static void __init mvebu_dt_init(void)
> >>>>   {
> >>>>   	if (of_machine_is_compatible("marvell,armadaxp"))
> >>>>   		i2c_quirk();
> >>>> +	enable_mbus_error_propagation();
> >>>>   }
> >>>>   
> >>>>   static void __init armada_370_xp_dt_fixup(void)
> >>>> @@ -191,6 +217,7 @@ DT_MACHINE_START(ARMADA_38X_DT, "Marvell Armada 380/385 (Device Tree)")
> >>>>   	.l2c_aux_val	= 0,
> >>>>   	.l2c_aux_mask	= ~0,
> >>>>   	.init_irq       = mvebu_init_irq,
> >>>> +	.init_machine	= mvebu_dt_init,
> >>>>   	.restart	= mvebu_restart,
> >>>>   	.dt_compat	= armada_38x_dt_compat,
> >>>>   MACHINE_END
> >>>> -- 
> >>>> 2.24.0
> >>>>
> >>>