The series is meant to support SMMU for AP806 and a workaround
for accessing ARM SMMU 64bit registers is the gist of it.
For the record, AP-806 can't access SMMU registers with 64bit width.
This patches split the readq/writeq into two 32bit accesses instead
and update DT bindings.
The series was successfully tested on a vanilla v5.8-rc3 kernel and
Intel e1000e PCIe NIC. The same for platform devices like SATA and USB.
For reference, previous versions are listed below:
V1: https://lkml.org/lkml/2018/10/15/373
V2: https://lkml.org/lkml/2019/7/11/426
V3: https://lkml.org/lkml/2020/7/2/1114
v3 -> v4
- call cfg_probe() impl hook a bit earlier which simplifies errata handling
- use hi_lo_readq_relaxed() and hi_lo_writeq_relaxed() for register accessors
- keep SMMU status disabled by default and enable where possible (DTS changes)
- commit logs improvements and other minor fixes
Hanna Hawa (1):
iommu/arm-smmu: Workaround for Marvell Armada-AP806 SoC erratum
#582743
Marcin Wojtas (1):
arm64: dts: marvell: add SMMU support
Tomasz Nowicki (2):
iommu/arm-smmu: Call configuration impl hook before consuming features
dt-bindings: arm-smmu: add compatible string for Marvell Armada-AP806
SMMU-500
Documentation/arm64/silicon-errata.rst | 3 ++
.../devicetree/bindings/iommu/arm,smmu.yaml | 4 ++
arch/arm64/boot/dts/marvell/armada-7040.dtsi | 28 ++++++++++++
arch/arm64/boot/dts/marvell/armada-8040.dtsi | 40 +++++++++++++++++
arch/arm64/boot/dts/marvell/armada-ap80x.dtsi | 18 ++++++++
drivers/iommu/arm-smmu-impl.c | 45 +++++++++++++++++++
drivers/iommu/arm-smmu.c | 11 +++--
7 files changed, 145 insertions(+), 4 deletions(-)
--
2.17.1
From: Marcin Wojtas <[email protected]>
Add IOMMU node for Marvell AP806 based SoCs together with platform
and PCI device Stream ID mapping.
Signed-off-by: Marcin Wojtas <[email protected]>
Signed-off-by: Tomasz Nowicki <[email protected]>
---
arch/arm64/boot/dts/marvell/armada-7040.dtsi | 28 +++++++++++++
arch/arm64/boot/dts/marvell/armada-8040.dtsi | 40 +++++++++++++++++++
arch/arm64/boot/dts/marvell/armada-ap80x.dtsi | 18 +++++++++
3 files changed, 86 insertions(+)
diff --git a/arch/arm64/boot/dts/marvell/armada-7040.dtsi b/arch/arm64/boot/dts/marvell/armada-7040.dtsi
index 47247215770d..7a3198cd7a07 100644
--- a/arch/arm64/boot/dts/marvell/armada-7040.dtsi
+++ b/arch/arm64/boot/dts/marvell/armada-7040.dtsi
@@ -14,3 +14,31 @@
compatible = "marvell,armada7040", "marvell,armada-ap806-quad",
"marvell,armada-ap806";
};
+
+&smmu {
+ status = "okay";
+};
+
+&cp0_pcie0 {
+ iommu-map =
+ <0x0 &smmu 0x480 0x20>,
+ <0x100 &smmu 0x4a0 0x20>,
+ <0x200 &smmu 0x4c0 0x20>;
+ iommu-map-mask = <0x031f>;
+};
+
+&cp0_sata0 {
+ iommus = <&smmu 0x444>;
+};
+
+&cp0_sdhci0 {
+ iommus = <&smmu 0x445>;
+};
+
+&cp0_usb3_0 {
+ iommus = <&smmu 0x440>;
+};
+
+&cp0_usb3_1 {
+ iommus = <&smmu 0x441>;
+};
diff --git a/arch/arm64/boot/dts/marvell/armada-8040.dtsi b/arch/arm64/boot/dts/marvell/armada-8040.dtsi
index 7699b19224c2..79e8ce59baa8 100644
--- a/arch/arm64/boot/dts/marvell/armada-8040.dtsi
+++ b/arch/arm64/boot/dts/marvell/armada-8040.dtsi
@@ -15,6 +15,18 @@
"marvell,armada-ap806";
};
+&smmu {
+ status = "okay";
+};
+
+&cp0_pcie0 {
+ iommu-map =
+ <0x0 &smmu 0x480 0x20>,
+ <0x100 &smmu 0x4a0 0x20>,
+ <0x200 &smmu 0x4c0 0x20>;
+ iommu-map-mask = <0x031f>;
+};
+
/* The RTC requires external oscillator. But on Aramda 80x0, the RTC clock
* in CP master is not connected (by package) to the oscillator. So
* disable it. However, the RTC clock in CP slave is connected to the
@@ -23,3 +35,31 @@
&cp0_rtc {
status = "disabled";
};
+
+&cp0_sata0 {
+ iommus = <&smmu 0x444>;
+};
+
+&cp0_sdhci0 {
+ iommus = <&smmu 0x445>;
+};
+
+&cp0_usb3_0 {
+ iommus = <&smmu 0x440>;
+};
+
+&cp0_usb3_1 {
+ iommus = <&smmu 0x441>;
+};
+
+&cp1_sata0 {
+ iommus = <&smmu 0x454>;
+};
+
+&cp1_usb3_0 {
+ iommus = <&smmu 0x450>;
+};
+
+&cp1_usb3_1 {
+ iommus = <&smmu 0x451>;
+};
diff --git a/arch/arm64/boot/dts/marvell/armada-ap80x.dtsi b/arch/arm64/boot/dts/marvell/armada-ap80x.dtsi
index 7f9b9a647717..12e477f1aeb9 100644
--- a/arch/arm64/boot/dts/marvell/armada-ap80x.dtsi
+++ b/arch/arm64/boot/dts/marvell/armada-ap80x.dtsi
@@ -56,6 +56,24 @@
compatible = "simple-bus";
ranges = <0x0 0x0 0xf0000000 0x1000000>;
+ smmu: iommu@5000000 {
+ compatible = "marvell,ap806-smmu-500", "arm,mmu-500";
+ reg = <0x100000 0x100000>;
+ dma-coherent;
+ #iommu-cells = <1>;
+ #global-interrupts = <1>;
+ interrupts = <GIC_SPI 6 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 6 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 6 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 6 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 6 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 6 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 6 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 6 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 6 IRQ_TYPE_LEVEL_HIGH>;
+ status = "disabled";
+ };
+
gic: interrupt-controller@210000 {
compatible = "arm,gic-400";
#interrupt-cells = <3>;
--
2.17.1
On Wed, 15 Jul 2020 09:06:45 +0200, Tomasz Nowicki wrote:
> The series is meant to support SMMU for AP806 and a workaround
> for accessing ARM SMMU 64bit registers is the gist of it.
>
> For the record, AP-806 can't access SMMU registers with 64bit width.
> This patches split the readq/writeq into two 32bit accesses instead
> and update DT bindings.
>
> [...]
Applied to will (for-joerg/arm-smmu/updates), thanks!
[1/3] iommu/arm-smmu: Call configuration impl hook before consuming features
https://git.kernel.org/will/c/6a79a5a3842b
[2/3] iommu/arm-smmu: Workaround for Marvell Armada-AP806 SoC erratum #582743
https://git.kernel.org/will/c/f2d9848aeb9f
[3/3] dt-bindings: arm-smmu: add compatible string for Marvell Armada-AP806 SMMU-500
https://git.kernel.org/will/c/e85e84d19b9d
Cheers,
--
Will
https://fixes.arm64.dev
https://next.arm64.dev
https://will.arm64.dev
On Thu, Jul 16, 2020 at 01:00:43PM +0100, Will Deacon wrote:
> On Wed, 15 Jul 2020 09:06:45 +0200, Tomasz Nowicki wrote:
> > The series is meant to support SMMU for AP806 and a workaround
> > for accessing ARM SMMU 64bit registers is the gist of it.
> >
> > For the record, AP-806 can't access SMMU registers with 64bit width.
> > This patches split the readq/writeq into two 32bit accesses instead
> > and update DT bindings.
> >
> > [...]
>
> Applied to will (for-joerg/arm-smmu/updates), thanks!
>
> [1/3] iommu/arm-smmu: Call configuration impl hook before consuming features
> https://git.kernel.org/will/c/6a79a5a3842b
> [2/3] iommu/arm-smmu: Workaround for Marvell Armada-AP806 SoC erratum #582743
> https://git.kernel.org/will/c/f2d9848aeb9f
> [3/3] dt-bindings: arm-smmu: add compatible string for Marvell Armada-AP806 SMMU-500
> https://git.kernel.org/will/c/e85e84d19b9d
(note that I left patch 4 for arm-soc, as that's just updating .dts files)
Will
czw., 16 lip 2020 o 14:02 Will Deacon <[email protected]> napisał(a):
>
> On Thu, Jul 16, 2020 at 01:00:43PM +0100, Will Deacon wrote:
> > On Wed, 15 Jul 2020 09:06:45 +0200, Tomasz Nowicki wrote:
> > > The series is meant to support SMMU for AP806 and a workaround
> > > for accessing ARM SMMU 64bit registers is the gist of it.
> > >
> > > For the record, AP-806 can't access SMMU registers with 64bit width.
> > > This patches split the readq/writeq into two 32bit accesses instead
> > > and update DT bindings.
> > >
> > > [...]
> >
> > Applied to will (for-joerg/arm-smmu/updates), thanks!
> >
> > [1/3] iommu/arm-smmu: Call configuration impl hook before consuming features
> > https://git.kernel.org/will/c/6a79a5a3842b
> > [2/3] iommu/arm-smmu: Workaround for Marvell Armada-AP806 SoC erratum #582743
> > https://git.kernel.org/will/c/f2d9848aeb9f
> > [3/3] dt-bindings: arm-smmu: add compatible string for Marvell Armada-AP806 SMMU-500
> > https://git.kernel.org/will/c/e85e84d19b9d
>
> (note that I left patch 4 for arm-soc, as that's just updating .dts files)
>
Hi Gregory,
Can you please help with the review/merge of patch #4?
Best regards,
Marcin
Hello,
> czw., 16 lip 2020 o 14:02 Will Deacon <[email protected]> napisał(a):
>>
>> On Thu, Jul 16, 2020 at 01:00:43PM +0100, Will Deacon wrote:
>> > On Wed, 15 Jul 2020 09:06:45 +0200, Tomasz Nowicki wrote:
>> > > The series is meant to support SMMU for AP806 and a workaround
>> > > for accessing ARM SMMU 64bit registers is the gist of it.
>> > >
>> > > For the record, AP-806 can't access SMMU registers with 64bit width.
>> > > This patches split the readq/writeq into two 32bit accesses instead
>> > > and update DT bindings.
>> > >
>> > > [...]
>> >
>> > Applied to will (for-joerg/arm-smmu/updates), thanks!
>> >
>> > [1/3] iommu/arm-smmu: Call configuration impl hook before consuming features
>> > https://git.kernel.org/will/c/6a79a5a3842b
>> > [2/3] iommu/arm-smmu: Workaround for Marvell Armada-AP806 SoC erratum #582743
>> > https://git.kernel.org/will/c/f2d9848aeb9f
>> > [3/3] dt-bindings: arm-smmu: add compatible string for Marvell Armada-AP806 SMMU-500
>> > https://git.kernel.org/will/c/e85e84d19b9d
>>
>> (note that I left patch 4 for arm-soc, as that's just updating .dts files)
>>
>
> Hi Gregory,
>
> Can you please help with the review/merge of patch #4?
Sure!
I've followed the series since the v1 even if I didn't commetn and I am
happy that it finally managed to be merged. I can now remove it from
my TODO list! :)
Gregory
>
> Best regards,
> Marcin
--
Gregory Clement, Bootlin
Embedded Linux and Kernel engineering
http://bootlin.com
Tomasz Nowicki <[email protected]> writes:
> From: Marcin Wojtas <[email protected]>
>
> Add IOMMU node for Marvell AP806 based SoCs together with platform
> and PCI device Stream ID mapping.
>
> Signed-off-by: Marcin Wojtas <[email protected]>
> Signed-off-by: Tomasz Nowicki <[email protected]>
Applied on mvebu/dt64
Thanks,
Gregory
> ---
> arch/arm64/boot/dts/marvell/armada-7040.dtsi | 28 +++++++++++++
> arch/arm64/boot/dts/marvell/armada-8040.dtsi | 40 +++++++++++++++++++
> arch/arm64/boot/dts/marvell/armada-ap80x.dtsi | 18 +++++++++
> 3 files changed, 86 insertions(+)
>
> diff --git a/arch/arm64/boot/dts/marvell/armada-7040.dtsi b/arch/arm64/boot/dts/marvell/armada-7040.dtsi
> index 47247215770d..7a3198cd7a07 100644
> --- a/arch/arm64/boot/dts/marvell/armada-7040.dtsi
> +++ b/arch/arm64/boot/dts/marvell/armada-7040.dtsi
> @@ -14,3 +14,31 @@
> compatible = "marvell,armada7040", "marvell,armada-ap806-quad",
> "marvell,armada-ap806";
> };
> +
> +&smmu {
> + status = "okay";
> +};
> +
> +&cp0_pcie0 {
> + iommu-map =
> + <0x0 &smmu 0x480 0x20>,
> + <0x100 &smmu 0x4a0 0x20>,
> + <0x200 &smmu 0x4c0 0x20>;
> + iommu-map-mask = <0x031f>;
> +};
> +
> +&cp0_sata0 {
> + iommus = <&smmu 0x444>;
> +};
> +
> +&cp0_sdhci0 {
> + iommus = <&smmu 0x445>;
> +};
> +
> +&cp0_usb3_0 {
> + iommus = <&smmu 0x440>;
> +};
> +
> +&cp0_usb3_1 {
> + iommus = <&smmu 0x441>;
> +};
> diff --git a/arch/arm64/boot/dts/marvell/armada-8040.dtsi b/arch/arm64/boot/dts/marvell/armada-8040.dtsi
> index 7699b19224c2..79e8ce59baa8 100644
> --- a/arch/arm64/boot/dts/marvell/armada-8040.dtsi
> +++ b/arch/arm64/boot/dts/marvell/armada-8040.dtsi
> @@ -15,6 +15,18 @@
> "marvell,armada-ap806";
> };
>
> +&smmu {
> + status = "okay";
> +};
> +
> +&cp0_pcie0 {
> + iommu-map =
> + <0x0 &smmu 0x480 0x20>,
> + <0x100 &smmu 0x4a0 0x20>,
> + <0x200 &smmu 0x4c0 0x20>;
> + iommu-map-mask = <0x031f>;
> +};
> +
> /* The RTC requires external oscillator. But on Aramda 80x0, the RTC clock
> * in CP master is not connected (by package) to the oscillator. So
> * disable it. However, the RTC clock in CP slave is connected to the
> @@ -23,3 +35,31 @@
> &cp0_rtc {
> status = "disabled";
> };
> +
> +&cp0_sata0 {
> + iommus = <&smmu 0x444>;
> +};
> +
> +&cp0_sdhci0 {
> + iommus = <&smmu 0x445>;
> +};
> +
> +&cp0_usb3_0 {
> + iommus = <&smmu 0x440>;
> +};
> +
> +&cp0_usb3_1 {
> + iommus = <&smmu 0x441>;
> +};
> +
> +&cp1_sata0 {
> + iommus = <&smmu 0x454>;
> +};
> +
> +&cp1_usb3_0 {
> + iommus = <&smmu 0x450>;
> +};
> +
> +&cp1_usb3_1 {
> + iommus = <&smmu 0x451>;
> +};
> diff --git a/arch/arm64/boot/dts/marvell/armada-ap80x.dtsi b/arch/arm64/boot/dts/marvell/armada-ap80x.dtsi
> index 7f9b9a647717..12e477f1aeb9 100644
> --- a/arch/arm64/boot/dts/marvell/armada-ap80x.dtsi
> +++ b/arch/arm64/boot/dts/marvell/armada-ap80x.dtsi
> @@ -56,6 +56,24 @@
> compatible = "simple-bus";
> ranges = <0x0 0x0 0xf0000000 0x1000000>;
>
> + smmu: iommu@5000000 {
> + compatible = "marvell,ap806-smmu-500", "arm,mmu-500";
> + reg = <0x100000 0x100000>;
> + dma-coherent;
> + #iommu-cells = <1>;
> + #global-interrupts = <1>;
> + interrupts = <GIC_SPI 6 IRQ_TYPE_LEVEL_HIGH>,
> + <GIC_SPI 6 IRQ_TYPE_LEVEL_HIGH>,
> + <GIC_SPI 6 IRQ_TYPE_LEVEL_HIGH>,
> + <GIC_SPI 6 IRQ_TYPE_LEVEL_HIGH>,
> + <GIC_SPI 6 IRQ_TYPE_LEVEL_HIGH>,
> + <GIC_SPI 6 IRQ_TYPE_LEVEL_HIGH>,
> + <GIC_SPI 6 IRQ_TYPE_LEVEL_HIGH>,
> + <GIC_SPI 6 IRQ_TYPE_LEVEL_HIGH>,
> + <GIC_SPI 6 IRQ_TYPE_LEVEL_HIGH>;
> + status = "disabled";
> + };
> +
> gic: interrupt-controller@210000 {
> compatible = "arm,gic-400";
> #interrupt-cells = <3>;
> --
> 2.17.1
>
--
Gregory Clement, Bootlin
Embedded Linux and Kernel engineering
http://bootlin.com
Hi,
> Am 15.07.2020 um 09:06 schrieb Tomasz Nowicki <[email protected]>:
>
> The series is meant to support SMMU for AP806 and a workaround
> for accessing ARM SMMU 64bit registers is the gist of it.
>
> For the record, AP-806 can't access SMMU registers with 64bit width.
> This patches split the readq/writeq into two 32bit accesses instead
> and update DT bindings.
>
> The series was successfully tested on a vanilla v5.8-rc3 kernel and
> Intel e1000e PCIe NIC. The same for platform devices like SATA and USB.
>
> For reference, previous versions are listed below:
> V1: https://lkml.org/lkml/2018/10/15/373
> V2: https://lkml.org/lkml/2019/7/11/426
> V3: https://lkml.org/lkml/2020/7/2/1114
>
1) After enabling SMMU on Armada 8040, and ARM_SMMU_DISABLE_BYPASS_BY_DEFAUL=y by default in kernel since 954a03be033c7cef80ddc232e7cbdb17df735663,
internal eMMC is prevented from being initialised (as there is no iommus property for ap_sdhci0)
Disabling "Disable bypass by default" make it work, but the patch highly suggest doing it properly.
I wasn't able to find correct path for ap_sdhci for iommus in any publicly available documentation,
would be highly appreciated addressed properly, thank you!
2) Second issue I got (btw I have ClearFog GT 8k armada-8040 based board) is mpci ath10k card.
It is found, it is enumerated, it is visible in lspci, but it fails to be initialised. Here is the log:
[ 1.743754] armada8k-pcie f2600000.pcie: host bridge /cp0/pcie@f2600000 ranges:
[ 1.751116] armada8k-pcie f2600000.pcie: MEM 0x00f6000000..0x00f6efffff -> 0x00f6000000
[ 1.964690] armada8k-pcie f2600000.pcie: Link up
[ 1.969379] armada8k-pcie f2600000.pcie: PCI host bridge to bus 0000:00
[ 1.976026] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 1.981537] pci_bus 0000:00: root bus resource [mem 0xf6000000-0xf6efffff]
[ 1.988462] pci 0000:00:00.0: [11ab:0110] type 01 class 0x060400
[ 1.994504] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x000fffff]
[ 2.000843] pci 0000:00:00.0: supports D1 D2
[ 2.005132] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
[ 2.011853] pci 0000:01:00.0: [168c:003c] type 00 class 0x028000
[ 2.018001] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x001fffff 64bit]
[ 2.025002] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
[ 2.032111] pci 0000:01:00.0: supports D1 D2
[ 2.049409] pci 0000:00:00.0: BAR 14: assigned [mem 0xf6000000-0xf61fffff]
[ 2.056322] pci 0000:00:00.0: BAR 0: assigned [mem 0xf6200000-0xf62fffff]
[ 2.063142] pci 0000:00:00.0: BAR 15: assigned [mem 0xf6300000-0xf63fffff pref]
[ 2.070484] pci 0000:01:00.0: BAR 0: assigned [mem 0xf6000000-0xf61fffff 64bit]
[ 2.077880] pci 0000:01:00.0: BAR 6: assigned [mem 0xf6300000-0xf630ffff pref]
[ 2.085135] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[ 2.090384] pci 0000:00:00.0: bridge window [mem 0xf6000000-0xf61fffff]
[ 2.097202] pci 0000:00:00.0: bridge window [mem 0xf6300000-0xf63fffff pref]
[ 2.104539] pcieport 0000:00:00.0: Adding to iommu group 4
[ 2.110232] pcieport 0000:00:00.0: PME: Signaling with IRQ 38
[ 2.116141] pcieport 0000:00:00.0: AER: enabled with IRQ 38
[ 8.131135] ath10k_pci 0000:01:00.0: Adding to iommu group 4
[ 8.131874] ath10k_pci 0000:01:00.0: enabling device (0000 -> 0002)
[ 8.132203] ath10k_pci 0000:01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
up to that point the log is the same as without SMMU enabled, except "Adding to iommu group N" lines, and IRQ being 37
[ 8.221328] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
[ 8.313362] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
[ 8.409373] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
[ 8.553433] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
[ 8.641370] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
[ 8.737979] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
[ 8.807356] ath10k_pci 0000:01:00.0: Failed to get pcie state addr: -16
[ 8.814032] ath10k_pci 0000:01:00.0: failed to setup init config: -16
[ 8.820605] ath10k_pci 0000:01:00.0: could not power on hif bus (-16)
[ 8.827111] ath10k_pci 0000:01:00.0: could not probe fw (-16)
Thank you!
> v3 -> v4
> - call cfg_probe() impl hook a bit earlier which simplifies errata handling
> - use hi_lo_readq_relaxed() and hi_lo_writeq_relaxed() for register accessors
> - keep SMMU status disabled by default and enable where possible (DTS changes)
> - commit logs improvements and other minor fixes
>
> Hanna Hawa (1):
> iommu/arm-smmu: Workaround for Marvell Armada-AP806 SoC erratum
> #582743
>
> Marcin Wojtas (1):
> arm64: dts: marvell: add SMMU support
>
> Tomasz Nowicki (2):
> iommu/arm-smmu: Call configuration impl hook before consuming features
> dt-bindings: arm-smmu: add compatible string for Marvell Armada-AP806
> SMMU-500
>
> Documentation/arm64/silicon-errata.rst | 3 ++
> .../devicetree/bindings/iommu/arm,smmu.yaml | 4 ++
> arch/arm64/boot/dts/marvell/armada-7040.dtsi | 28 ++++++++++++
> arch/arm64/boot/dts/marvell/armada-8040.dtsi | 40 +++++++++++++++++
> arch/arm64/boot/dts/marvell/armada-ap80x.dtsi | 18 ++++++++
> drivers/iommu/arm-smmu-impl.c | 45 +++++++++++++++++++
> drivers/iommu/arm-smmu.c | 11 +++--
> 7 files changed, 145 insertions(+), 4 deletions(-)
>
> --
> 2.17.1
>
> _______________________________________________
> iommu mailing list
> [email protected]
> https://lists.linuxfoundation.org/mailman/listinfo/iommu
>
Hi Denis,
Thank you for your report.
wt., 6 paź 2020 o 17:17 Denis Odintsov <[email protected]> napisał(a):
>
> Hi,
>
> > Am 15.07.2020 um 09:06 schrieb Tomasz Nowicki <[email protected]>:
> >
> > The series is meant to support SMMU for AP806 and a workaround
> > for accessing ARM SMMU 64bit registers is the gist of it.
> >
> > For the record, AP-806 can't access SMMU registers with 64bit width.
> > This patches split the readq/writeq into two 32bit accesses instead
> > and update DT bindings.
> >
> > The series was successfully tested on a vanilla v5.8-rc3 kernel and
> > Intel e1000e PCIe NIC. The same for platform devices like SATA and USB.
> >
> > For reference, previous versions are listed below:
> > V1: https://lkml.org/lkml/2018/10/15/373
> > V2: https://lkml.org/lkml/2019/7/11/426
> > V3: https://lkml.org/lkml/2020/7/2/1114
> >
>
> 1) After enabling SMMU on Armada 8040, and ARM_SMMU_DISABLE_BYPASS_BY_DEFAUL=y by default in kernel since 954a03be033c7cef80ddc232e7cbdb17df735663,
> internal eMMC is prevented from being initialised (as there is no iommus property for ap_sdhci0)
> Disabling "Disable bypass by default" make it work, but the patch highly suggest doing it properly.
> I wasn't able to find correct path for ap_sdhci for iommus in any publicly available documentation,
> would be highly appreciated addressed properly, thank you!
According to my knowledge and the docs AP IO devices cannot be
virtualized, only ones connected via CP110/CP115. We'd need to check
what should be done in such configuration and get back to you.
>
> 2) Second issue I got (btw I have ClearFog GT 8k armada-8040 based board) is mpci ath10k card.
> It is found, it is enumerated, it is visible in lspci, but it fails to be initialised. Here is the log:
>
> [ 1.743754] armada8k-pcie f2600000.pcie: host bridge /cp0/pcie@f2600000 ranges:
> [ 1.751116] armada8k-pcie f2600000.pcie: MEM 0x00f6000000..0x00f6efffff -> 0x00f6000000
> [ 1.964690] armada8k-pcie f2600000.pcie: Link up
> [ 1.969379] armada8k-pcie f2600000.pcie: PCI host bridge to bus 0000:00
> [ 1.976026] pci_bus 0000:00: root bus resource [bus 00-ff]
> [ 1.981537] pci_bus 0000:00: root bus resource [mem 0xf6000000-0xf6efffff]
> [ 1.988462] pci 0000:00:00.0: [11ab:0110] type 01 class 0x060400
> [ 1.994504] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x000fffff]
> [ 2.000843] pci 0000:00:00.0: supports D1 D2
> [ 2.005132] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
> [ 2.011853] pci 0000:01:00.0: [168c:003c] type 00 class 0x028000
> [ 2.018001] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x001fffff 64bit]
> [ 2.025002] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
> [ 2.032111] pci 0000:01:00.0: supports D1 D2
> [ 2.049409] pci 0000:00:00.0: BAR 14: assigned [mem 0xf6000000-0xf61fffff]
> [ 2.056322] pci 0000:00:00.0: BAR 0: assigned [mem 0xf6200000-0xf62fffff]
> [ 2.063142] pci 0000:00:00.0: BAR 15: assigned [mem 0xf6300000-0xf63fffff pref]
> [ 2.070484] pci 0000:01:00.0: BAR 0: assigned [mem 0xf6000000-0xf61fffff 64bit]
> [ 2.077880] pci 0000:01:00.0: BAR 6: assigned [mem 0xf6300000-0xf630ffff pref]
> [ 2.085135] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
> [ 2.090384] pci 0000:00:00.0: bridge window [mem 0xf6000000-0xf61fffff]
> [ 2.097202] pci 0000:00:00.0: bridge window [mem 0xf6300000-0xf63fffff pref]
> [ 2.104539] pcieport 0000:00:00.0: Adding to iommu group 4
> [ 2.110232] pcieport 0000:00:00.0: PME: Signaling with IRQ 38
> [ 2.116141] pcieport 0000:00:00.0: AER: enabled with IRQ 38
> [ 8.131135] ath10k_pci 0000:01:00.0: Adding to iommu group 4
> [ 8.131874] ath10k_pci 0000:01:00.0: enabling device (0000 -> 0002)
> [ 8.132203] ath10k_pci 0000:01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
>
> up to that point the log is the same as without SMMU enabled, except "Adding to iommu group N" lines, and IRQ being 37
>
> [ 8.221328] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
> [ 8.313362] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
> [ 8.409373] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
> [ 8.553433] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
> [ 8.641370] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
> [ 8.737979] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
> [ 8.807356] ath10k_pci 0000:01:00.0: Failed to get pcie state addr: -16
> [ 8.814032] ath10k_pci 0000:01:00.0: failed to setup init config: -16
> [ 8.820605] ath10k_pci 0000:01:00.0: could not power on hif bus (-16)
> [ 8.827111] ath10k_pci 0000:01:00.0: could not probe fw (-16)
>
> Thank you!
The PCIE was validated when booting from edk2 + using pci-host-generic
driver and standard intel NIC. Not sure if it makes any difference vs
the Designware driver ("marvell,armada8k-pcie"), but we need to
double-check that.
Best regards,
Marcin
>
> > v3 -> v4
> > - call cfg_probe() impl hook a bit earlier which simplifies errata handling
> > - use hi_lo_readq_relaxed() and hi_lo_writeq_relaxed() for register accessors
> > - keep SMMU status disabled by default and enable where possible (DTS changes)
> > - commit logs improvements and other minor fixes
> >
> > Hanna Hawa (1):
> > iommu/arm-smmu: Workaround for Marvell Armada-AP806 SoC erratum
> > #582743
> >
> > Marcin Wojtas (1):
> > arm64: dts: marvell: add SMMU support
> >
> > Tomasz Nowicki (2):
> > iommu/arm-smmu: Call configuration impl hook before consuming features
> > dt-bindings: arm-smmu: add compatible string for Marvell Armada-AP806
> > SMMU-500
> >
> > Documentation/arm64/silicon-errata.rst | 3 ++
> > .../devicetree/bindings/iommu/arm,smmu.yaml | 4 ++
> > arch/arm64/boot/dts/marvell/armada-7040.dtsi | 28 ++++++++++++
> > arch/arm64/boot/dts/marvell/armada-8040.dtsi | 40 +++++++++++++++++
> > arch/arm64/boot/dts/marvell/armada-ap80x.dtsi | 18 ++++++++
> > drivers/iommu/arm-smmu-impl.c | 45 +++++++++++++++++++
> > drivers/iommu/arm-smmu.c | 11 +++--
> > 7 files changed, 145 insertions(+), 4 deletions(-)
> >
> > --
> > 2.17.1
> >
> > _______________________________________________
> > iommu mailing list
> > [email protected]
> > https://lists.linuxfoundation.org/mailman/listinfo/iommu
> >
>
On 2020-10-06 16:16, Denis Odintsov wrote:
> Hi,
>
>> Am 15.07.2020 um 09:06 schrieb Tomasz Nowicki <[email protected]>:
>>
>> The series is meant to support SMMU for AP806 and a workaround
>> for accessing ARM SMMU 64bit registers is the gist of it.
>>
>> For the record, AP-806 can't access SMMU registers with 64bit width.
>> This patches split the readq/writeq into two 32bit accesses instead
>> and update DT bindings.
>>
>> The series was successfully tested on a vanilla v5.8-rc3 kernel and
>> Intel e1000e PCIe NIC. The same for platform devices like SATA and USB.
>>
>> For reference, previous versions are listed below:
>> V1: https://lkml.org/lkml/2018/10/15/373
>> V2: https://lkml.org/lkml/2019/7/11/426
>> V3: https://lkml.org/lkml/2020/7/2/1114
>>
>
> 1) After enabling SMMU on Armada 8040, and ARM_SMMU_DISABLE_BYPASS_BY_DEFAUL=y by default in kernel since 954a03be033c7cef80ddc232e7cbdb17df735663,
> internal eMMC is prevented from being initialised (as there is no iommus property for ap_sdhci0)
> Disabling "Disable bypass by default" make it work, but the patch highly suggest doing it properly.
> I wasn't able to find correct path for ap_sdhci for iommus in any publicly available documentation,
> would be highly appreciated addressed properly, thank you!
FWIW the SMMU tells you the offending unmatched Stream ID, so if faults
can reasonably be correlated with a particular device making accesses,
you can effectively discover the Stream ID assignment by trial and
error. Often that can be easier than trying to find formal documentation
anyway ;)
> 2) Second issue I got (btw I have ClearFog GT 8k armada-8040 based board) is mpci ath10k card.
> It is found, it is enumerated, it is visible in lspci, but it fails to be initialised. Here is the log:
>
> [ 1.743754] armada8k-pcie f2600000.pcie: host bridge /cp0/pcie@f2600000 ranges:
> [ 1.751116] armada8k-pcie f2600000.pcie: MEM 0x00f6000000..0x00f6efffff -> 0x00f6000000
> [ 1.964690] armada8k-pcie f2600000.pcie: Link up
> [ 1.969379] armada8k-pcie f2600000.pcie: PCI host bridge to bus 0000:00
> [ 1.976026] pci_bus 0000:00: root bus resource [bus 00-ff]
> [ 1.981537] pci_bus 0000:00: root bus resource [mem 0xf6000000-0xf6efffff]
> [ 1.988462] pci 0000:00:00.0: [11ab:0110] type 01 class 0x060400
> [ 1.994504] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x000fffff]
> [ 2.000843] pci 0000:00:00.0: supports D1 D2
> [ 2.005132] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
> [ 2.011853] pci 0000:01:00.0: [168c:003c] type 00 class 0x028000
> [ 2.018001] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x001fffff 64bit]
> [ 2.025002] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
> [ 2.032111] pci 0000:01:00.0: supports D1 D2
> [ 2.049409] pci 0000:00:00.0: BAR 14: assigned [mem 0xf6000000-0xf61fffff]
> [ 2.056322] pci 0000:00:00.0: BAR 0: assigned [mem 0xf6200000-0xf62fffff]
> [ 2.063142] pci 0000:00:00.0: BAR 15: assigned [mem 0xf6300000-0xf63fffff pref]
> [ 2.070484] pci 0000:01:00.0: BAR 0: assigned [mem 0xf6000000-0xf61fffff 64bit]
> [ 2.077880] pci 0000:01:00.0: BAR 6: assigned [mem 0xf6300000-0xf630ffff pref]
> [ 2.085135] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
> [ 2.090384] pci 0000:00:00.0: bridge window [mem 0xf6000000-0xf61fffff]
> [ 2.097202] pci 0000:00:00.0: bridge window [mem 0xf6300000-0xf63fffff pref]
> [ 2.104539] pcieport 0000:00:00.0: Adding to iommu group 4
> [ 2.110232] pcieport 0000:00:00.0: PME: Signaling with IRQ 38
> [ 2.116141] pcieport 0000:00:00.0: AER: enabled with IRQ 38
> [ 8.131135] ath10k_pci 0000:01:00.0: Adding to iommu group 4
> [ 8.131874] ath10k_pci 0000:01:00.0: enabling device (0000 -> 0002)
> [ 8.132203] ath10k_pci 0000:01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
>
> up to that point the log is the same as without SMMU enabled, except "Adding to iommu group N" lines, and IRQ being 37
Does forcing ath10k to use legacy interrupts rather than MSIs make a
difference?
Judging by the DT it looks like MSIs ought to be targeting the GICv2M
widget, but if things somehow end up trying to use the PCIe controller's
internal MSI doorbell (upstream of SMMU translation) instead, then that
might account for general interrupt-related weirdness.
Robin.
> [ 8.221328] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
> [ 8.313362] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
> [ 8.409373] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
> [ 8.553433] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
> [ 8.641370] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
> [ 8.737979] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
> [ 8.807356] ath10k_pci 0000:01:00.0: Failed to get pcie state addr: -16
> [ 8.814032] ath10k_pci 0000:01:00.0: failed to setup init config: -16
> [ 8.820605] ath10k_pci 0000:01:00.0: could not power on hif bus (-16)
> [ 8.827111] ath10k_pci 0000:01:00.0: could not probe fw (-16)
>
> Thank you!
>
>> v3 -> v4
>> - call cfg_probe() impl hook a bit earlier which simplifies errata handling
>> - use hi_lo_readq_relaxed() and hi_lo_writeq_relaxed() for register accessors
>> - keep SMMU status disabled by default and enable where possible (DTS changes)
>> - commit logs improvements and other minor fixes
>>
>> Hanna Hawa (1):
>> iommu/arm-smmu: Workaround for Marvell Armada-AP806 SoC erratum
>> #582743
>>
>> Marcin Wojtas (1):
>> arm64: dts: marvell: add SMMU support
>>
>> Tomasz Nowicki (2):
>> iommu/arm-smmu: Call configuration impl hook before consuming features
>> dt-bindings: arm-smmu: add compatible string for Marvell Armada-AP806
>> SMMU-500
>>
>> Documentation/arm64/silicon-errata.rst | 3 ++
>> .../devicetree/bindings/iommu/arm,smmu.yaml | 4 ++
>> arch/arm64/boot/dts/marvell/armada-7040.dtsi | 28 ++++++++++++
>> arch/arm64/boot/dts/marvell/armada-8040.dtsi | 40 +++++++++++++++++
>> arch/arm64/boot/dts/marvell/armada-ap80x.dtsi | 18 ++++++++
>> drivers/iommu/arm-smmu-impl.c | 45 +++++++++++++++++++
>> drivers/iommu/arm-smmu.c | 11 +++--
>> 7 files changed, 145 insertions(+), 4 deletions(-)
>>
>> --
>> 2.17.1
>>
>> _______________________________________________
>> iommu mailing list
>> [email protected]
>> https://lists.linuxfoundation.org/mailman/listinfo/iommu
>>
>
> Am 13.10.2020 um 15:08 schrieb Robin Murphy <[email protected]>:
>
> On 2020-10-06 16:16, Denis Odintsov wrote:
>> Hi,
>>> Am 15.07.2020 um 09:06 schrieb Tomasz Nowicki <[email protected]>:
>>>
>>> The series is meant to support SMMU for AP806 and a workaround
>>> for accessing ARM SMMU 64bit registers is the gist of it.
>>>
>>> For the record, AP-806 can't access SMMU registers with 64bit width.
>>> This patches split the readq/writeq into two 32bit accesses instead
>>> and update DT bindings.
>>>
>>> The series was successfully tested on a vanilla v5.8-rc3 kernel and
>>> Intel e1000e PCIe NIC. The same for platform devices like SATA and USB.
>>>
>>> For reference, previous versions are listed below:
>>> V1: https://lkml.org/lkml/2018/10/15/373
>>> V2: https://lkml.org/lkml/2019/7/11/426
>>> V3: https://lkml.org/lkml/2020/7/2/1114
>>>
>> 1) After enabling SMMU on Armada 8040, and ARM_SMMU_DISABLE_BYPASS_BY_DEFAUL=y by default in kernel since 954a03be033c7cef80ddc232e7cbdb17df735663,
>> internal eMMC is prevented from being initialised (as there is no iommus property for ap_sdhci0)
>> Disabling "Disable bypass by default" make it work, but the patch highly suggest doing it properly.
>> I wasn't able to find correct path for ap_sdhci for iommus in any publicly available documentation,
>> would be highly appreciated addressed properly, thank you!
>
> FWIW the SMMU tells you the offending unmatched Stream ID, so if faults can reasonably be correlated with a particular device making accesses, you can effectively discover the Stream ID assignment by trial and error. Often that can be easier than trying to find formal documentation anyway ;)
>
>> 2) Second issue I got (btw I have ClearFog GT 8k armada-8040 based board) is mpci ath10k card.
>> It is found, it is enumerated, it is visible in lspci, but it fails to be initialised. Here is the log:
>> [ 1.743754] armada8k-pcie f2600000.pcie: host bridge /cp0/pcie@f2600000 ranges:
>> [ 1.751116] armada8k-pcie f2600000.pcie: MEM 0x00f6000000..0x00f6efffff -> 0x00f6000000
>> [ 1.964690] armada8k-pcie f2600000.pcie: Link up
>> [ 1.969379] armada8k-pcie f2600000.pcie: PCI host bridge to bus 0000:00
>> [ 1.976026] pci_bus 0000:00: root bus resource [bus 00-ff]
>> [ 1.981537] pci_bus 0000:00: root bus resource [mem 0xf6000000-0xf6efffff]
>> [ 1.988462] pci 0000:00:00.0: [11ab:0110] type 01 class 0x060400
>> [ 1.994504] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x000fffff]
>> [ 2.000843] pci 0000:00:00.0: supports D1 D2
>> [ 2.005132] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
>> [ 2.011853] pci 0000:01:00.0: [168c:003c] type 00 class 0x028000
>> [ 2.018001] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x001fffff 64bit]
>> [ 2.025002] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
>> [ 2.032111] pci 0000:01:00.0: supports D1 D2
>> [ 2.049409] pci 0000:00:00.0: BAR 14: assigned [mem 0xf6000000-0xf61fffff]
>> [ 2.056322] pci 0000:00:00.0: BAR 0: assigned [mem 0xf6200000-0xf62fffff]
>> [ 2.063142] pci 0000:00:00.0: BAR 15: assigned [mem 0xf6300000-0xf63fffff pref]
>> [ 2.070484] pci 0000:01:00.0: BAR 0: assigned [mem 0xf6000000-0xf61fffff 64bit]
>> [ 2.077880] pci 0000:01:00.0: BAR 6: assigned [mem 0xf6300000-0xf630ffff pref]
>> [ 2.085135] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
>> [ 2.090384] pci 0000:00:00.0: bridge window [mem 0xf6000000-0xf61fffff]
>> [ 2.097202] pci 0000:00:00.0: bridge window [mem 0xf6300000-0xf63fffff pref]
>> [ 2.104539] pcieport 0000:00:00.0: Adding to iommu group 4
>> [ 2.110232] pcieport 0000:00:00.0: PME: Signaling with IRQ 38
>> [ 2.116141] pcieport 0000:00:00.0: AER: enabled with IRQ 38
>> [ 8.131135] ath10k_pci 0000:01:00.0: Adding to iommu group 4
>> [ 8.131874] ath10k_pci 0000:01:00.0: enabling device (0000 -> 0002)
>> [ 8.132203] ath10k_pci 0000:01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
>> up to that point the log is the same as without SMMU enabled, except "Adding to iommu group N" lines, and IRQ being 37
>
> Does forcing ath10k to use legacy interrupts rather than MSIs make a difference?
>
> Judging by the DT it looks like MSIs ought to be targeting the GICv2M widget, but if things somehow end up trying to use the PCIe controller's internal MSI doorbell (upstream of SMMU translation) instead, then that might account for general interrupt-related weirdness.
>
> Robin.
>
>> [ 8.221328] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
>> [ 8.313362] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
>> [ 8.409373] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
>> [ 8.553433] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
>> [ 8.641370] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
>> [ 8.737979] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
>> [ 8.807356] ath10k_pci 0000:01:00.0: Failed to get pcie state addr: -16
>> [ 8.814032] ath10k_pci 0000:01:00.0: failed to setup init config: -16
>> [ 8.820605] ath10k_pci 0000:01:00.0: could not power on hif bus (-16)
>> [ 8.827111] ath10k_pci 0000:01:00.0: could not probe fw (-16)
>> Thank you!
>>> v3 -> v4
>>> - call cfg_probe() impl hook a bit earlier which simplifies errata handling
>>> - use hi_lo_readq_relaxed() and hi_lo_writeq_relaxed() for register accessors
>>> - keep SMMU status disabled by default and enable where possible (DTS changes)
>>> - commit logs improvements and other minor fixes
>>>
>>> Hanna Hawa (1):
>>> iommu/arm-smmu: Workaround for Marvell Armada-AP806 SoC erratum
>>> #582743
>>>
>>> Marcin Wojtas (1):
>>> arm64: dts: marvell: add SMMU support
>>>
>>> Tomasz Nowicki (2):
>>> iommu/arm-smmu: Call configuration impl hook before consuming features
>>> dt-bindings: arm-smmu: add compatible string for Marvell Armada-AP806
>>> SMMU-500
>>>
>>> Documentation/arm64/silicon-errata.rst | 3 ++
>>> .../devicetree/bindings/iommu/arm,smmu.yaml | 4 ++
>>> arch/arm64/boot/dts/marvell/armada-7040.dtsi | 28 ++++++++++++
>>> arch/arm64/boot/dts/marvell/armada-8040.dtsi | 40 +++++++++++++++++
>>> arch/arm64/boot/dts/marvell/armada-ap80x.dtsi | 18 ++++++++
>>> drivers/iommu/arm-smmu-impl.c | 45 +++++++++++++++++++
>>> drivers/iommu/arm-smmu.c | 11 +++--
>>> 7 files changed, 145 insertions(+), 4 deletions(-)
>>>
>>> --
>>> 2.17.1
>>>
>>> _______________________________________________
>>> iommu mailing list
>>> [email protected]
>>> https://lists.linuxfoundation.org/mailman/listinfo/iommu
Hi Denis,
Sorry for late response, we had to check few things. Please see comments
inline.
On 10/6/20 3:16 PM, Denis Odintsov wrote:
> Hi,
>
>> Am 15.07.2020 um 09:06 schrieb Tomasz Nowicki <[email protected]>:
>>
>> The series is meant to support SMMU for AP806 and a workaround
>> for accessing ARM SMMU 64bit registers is the gist of it.
>>
>> For the record, AP-806 can't access SMMU registers with 64bit width.
>> This patches split the readq/writeq into two 32bit accesses instead
>> and update DT bindings.
>>
>> The series was successfully tested on a vanilla v5.8-rc3 kernel and
>> Intel e1000e PCIe NIC. The same for platform devices like SATA and USB.
>>
>> For reference, previous versions are listed below:
>> V1: https://lkml.org/lkml/2018/10/15/373
>> V2: https://lkml.org/lkml/2019/7/11/426
>> V3: https://lkml.org/lkml/2020/7/2/1114
>>
>
> 1) After enabling SMMU on Armada 8040, and ARM_SMMU_DISABLE_BYPASS_BY_DEFAUL=y by default in kernel since 954a03be033c7cef80ddc232e7cbdb17df735663,
> internal eMMC is prevented from being initialised (as there is no iommus property for ap_sdhci0)
> Disabling "Disable bypass by default" make it work, but the patch highly suggest doing it properly.
> I wasn't able to find correct path for ap_sdhci for iommus in any publicly available documentation,
> would be highly appreciated addressed properly, thank you!
>
> 2) Second issue I got (btw I have ClearFog GT 8k armada-8040 based board) is mpci ath10k card.
> It is found, it is enumerated, it is visible in lspci, but it fails to be initialised. Here is the log:
Firmware has to configure and assign device StreamIDs. Most of the
devices are configured properly and supported in public FW. However, for
both these cases (ap_sdhci0 and PCIe) some extra (u-boot/UEFI/ATF)
patches are required which are not available yet. Sorry we let that happen.
Since we have dependency on custom FW and we cannot enforce people to
patch their FW we will send the follow up fix patch (v5.9+) and revert
respective DTS changes.
The most important Armada-806 SMMU driver enhancements were merged so
people who still willing to use SMMU need to provide proper DTB and use
ARM_SMMU_DISABLE_BYPASS_BY_DEFAUL=n (or via kernel command line) with
extra cautious.
Thanks,
Tomasz
>
> [ 1.743754] armada8k-pcie f2600000.pcie: host bridge /cp0/pcie@f2600000 ranges:
> [ 1.751116] armada8k-pcie f2600000.pcie: MEM 0x00f6000000..0x00f6efffff -> 0x00f6000000
> [ 1.964690] armada8k-pcie f2600000.pcie: Link up
> [ 1.969379] armada8k-pcie f2600000.pcie: PCI host bridge to bus 0000:00
> [ 1.976026] pci_bus 0000:00: root bus resource [bus 00-ff]
> [ 1.981537] pci_bus 0000:00: root bus resource [mem 0xf6000000-0xf6efffff]
> [ 1.988462] pci 0000:00:00.0: [11ab:0110] type 01 class 0x060400
> [ 1.994504] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x000fffff]
> [ 2.000843] pci 0000:00:00.0: supports D1 D2
> [ 2.005132] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
> [ 2.011853] pci 0000:01:00.0: [168c:003c] type 00 class 0x028000
> [ 2.018001] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x001fffff 64bit]
> [ 2.025002] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
> [ 2.032111] pci 0000:01:00.0: supports D1 D2
> [ 2.049409] pci 0000:00:00.0: BAR 14: assigned [mem 0xf6000000-0xf61fffff]
> [ 2.056322] pci 0000:00:00.0: BAR 0: assigned [mem 0xf6200000-0xf62fffff]
> [ 2.063142] pci 0000:00:00.0: BAR 15: assigned [mem 0xf6300000-0xf63fffff pref]
> [ 2.070484] pci 0000:01:00.0: BAR 0: assigned [mem 0xf6000000-0xf61fffff 64bit]
> [ 2.077880] pci 0000:01:00.0: BAR 6: assigned [mem 0xf6300000-0xf630ffff pref]
> [ 2.085135] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
> [ 2.090384] pci 0000:00:00.0: bridge window [mem 0xf6000000-0xf61fffff]
> [ 2.097202] pci 0000:00:00.0: bridge window [mem 0xf6300000-0xf63fffff pref]
> [ 2.104539] pcieport 0000:00:00.0: Adding to iommu group 4
> [ 2.110232] pcieport 0000:00:00.0: PME: Signaling with IRQ 38
> [ 2.116141] pcieport 0000:00:00.0: AER: enabled with IRQ 38
> [ 8.131135] ath10k_pci 0000:01:00.0: Adding to iommu group 4
> [ 8.131874] ath10k_pci 0000:01:00.0: enabling device (0000 -> 0002)
> [ 8.132203] ath10k_pci 0000:01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
>
> up to that point the log is the same as without SMMU enabled, except "Adding to iommu group N" lines, and IRQ being 37
>
> [ 8.221328] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
> [ 8.313362] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
> [ 8.409373] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
> [ 8.553433] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
> [ 8.641370] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
> [ 8.737979] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
> [ 8.807356] ath10k_pci 0000:01:00.0: Failed to get pcie state addr: -16
> [ 8.814032] ath10k_pci 0000:01:00.0: failed to setup init config: -16
> [ 8.820605] ath10k_pci 0000:01:00.0: could not power on hif bus (-16)
> [ 8.827111] ath10k_pci 0000:01:00.0: could not probe fw (-16)
>
> Thank you!
>
>> v3 -> v4
>> - call cfg_probe() impl hook a bit earlier which simplifies errata handling
>> - use hi_lo_readq_relaxed() and hi_lo_writeq_relaxed() for register accessors
>> - keep SMMU status disabled by default and enable where possible (DTS changes)
>> - commit logs improvements and other minor fixes
>>
>> Hanna Hawa (1):
>> iommu/arm-smmu: Workaround for Marvell Armada-AP806 SoC erratum
>> #582743
>>
>> Marcin Wojtas (1):
>> arm64: dts: marvell: add SMMU support
>>
>> Tomasz Nowicki (2):
>> iommu/arm-smmu: Call configuration impl hook before consuming features
>> dt-bindings: arm-smmu: add compatible string for Marvell Armada-AP806
>> SMMU-500
>>
>> Documentation/arm64/silicon-errata.rst | 3 ++
>> .../devicetree/bindings/iommu/arm,smmu.yaml | 4 ++
>> arch/arm64/boot/dts/marvell/armada-7040.dtsi | 28 ++++++++++++
>> arch/arm64/boot/dts/marvell/armada-8040.dtsi | 40 +++++++++++++++++
>> arch/arm64/boot/dts/marvell/armada-ap80x.dtsi | 18 ++++++++
>> drivers/iommu/arm-smmu-impl.c | 45 +++++++++++++++++++
>> drivers/iommu/arm-smmu.c | 11 +++--
>> 7 files changed, 145 insertions(+), 4 deletions(-)
>>
>> --
>> 2.17.1
>>
>> _______________________________________________
>> iommu mailing list
>> [email protected]
>> https://lists.linuxfoundation.org/mailman/listinfo/iommu
>>
>
On 2020-10-23 13:19, Tomasz Nowicki wrote:
> Hi Denis,
>
> Sorry for late response, we had to check few things. Please see comments
> inline.
>
> On 10/6/20 3:16 PM, Denis Odintsov wrote:
>> Hi,
>>
>>> Am 15.07.2020 um 09:06 schrieb Tomasz Nowicki <[email protected]>:
>>>
>>> The series is meant to support SMMU for AP806 and a workaround
>>> for accessing ARM SMMU 64bit registers is the gist of it.
>>>
>>> For the record, AP-806 can't access SMMU registers with 64bit width.
>>> This patches split the readq/writeq into two 32bit accesses instead
>>> and update DT bindings.
>>>
>>> The series was successfully tested on a vanilla v5.8-rc3 kernel and
>>> Intel e1000e PCIe NIC. The same for platform devices like SATA and USB.
>>>
>>> For reference, previous versions are listed below:
>>> V1: https://lkml.org/lkml/2018/10/15/373
>>> V2: https://lkml.org/lkml/2019/7/11/426
>>> V3: https://lkml.org/lkml/2020/7/2/1114
>>>
>>
>> 1) After enabling SMMU on Armada 8040, and
>> ARM_SMMU_DISABLE_BYPASS_BY_DEFAUL=y by default in kernel since
>> 954a03be033c7cef80ddc232e7cbdb17df735663,
>> internal eMMC is prevented from being initialised (as there is no
>> iommus property for ap_sdhci0)
>> Disabling "Disable bypass by default" make it work, but the patch
>> highly suggest doing it properly.
>> I wasn't able to find correct path for ap_sdhci for iommus in any
>> publicly available documentation,
>> would be highly appreciated addressed properly, thank you!
>>
>> 2) Second issue I got (btw I have ClearFog GT 8k armada-8040 based
>> board) is mpci ath10k card.
>> It is found, it is enumerated, it is visible in lspci, but it fails to
>> be initialised. Here is the log:
>
> Firmware has to configure and assign device StreamIDs. Most of the
> devices are configured properly and supported in public FW. However, for
> both these cases (ap_sdhci0 and PCIe) some extra (u-boot/UEFI/ATF)
> patches are required which are not available yet. Sorry we let that happen.
>
> Since we have dependency on custom FW and we cannot enforce people to
> patch their FW we will send the follow up fix patch (v5.9+) and revert
> respective DTS changes.
Note that it should be sufficient to simply keep the SMMU node disabled,
rather than fully revert everything. For example, the PCIe SMMU for Arm
Juno boards has been in that state for a long time - there are reasons
why it isn't (yet) 100% usable for everyone, but it can easily be
enabled locally for development (as I do).
Robin.
> The most important Armada-806 SMMU driver enhancements were merged so
> people who still willing to use SMMU need to provide proper DTB and use
> ARM_SMMU_DISABLE_BYPASS_BY_DEFAUL=n (or via kernel command line) with
> extra cautious.
>
> Thanks,
> Tomasz
>
>>
>> [ 1.743754] armada8k-pcie f2600000.pcie: host bridge
>> /cp0/pcie@f2600000 ranges:
>> [ 1.751116] armada8k-pcie f2600000.pcie: MEM
>> 0x00f6000000..0x00f6efffff -> 0x00f6000000
>> [ 1.964690] armada8k-pcie f2600000.pcie: Link up
>> [ 1.969379] armada8k-pcie f2600000.pcie: PCI host bridge to bus
>> 0000:00
>> [ 1.976026] pci_bus 0000:00: root bus resource [bus 00-ff]
>> [ 1.981537] pci_bus 0000:00: root bus resource [mem
>> 0xf6000000-0xf6efffff]
>> [ 1.988462] pci 0000:00:00.0: [11ab:0110] type 01 class 0x060400
>> [ 1.994504] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x000fffff]
>> [ 2.000843] pci 0000:00:00.0: supports D1 D2
>> [ 2.005132] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
>> [ 2.011853] pci 0000:01:00.0: [168c:003c] type 00 class 0x028000
>> [ 2.018001] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x001fffff
>> 64bit]
>> [ 2.025002] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0000ffff
>> pref]
>> [ 2.032111] pci 0000:01:00.0: supports D1 D2
>> [ 2.049409] pci 0000:00:00.0: BAR 14: assigned [mem
>> 0xf6000000-0xf61fffff]
>> [ 2.056322] pci 0000:00:00.0: BAR 0: assigned [mem
>> 0xf6200000-0xf62fffff]
>> [ 2.063142] pci 0000:00:00.0: BAR 15: assigned [mem
>> 0xf6300000-0xf63fffff pref]
>> [ 2.070484] pci 0000:01:00.0: BAR 0: assigned [mem
>> 0xf6000000-0xf61fffff 64bit]
>> [ 2.077880] pci 0000:01:00.0: BAR 6: assigned [mem
>> 0xf6300000-0xf630ffff pref]
>> [ 2.085135] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
>> [ 2.090384] pci 0000:00:00.0: bridge window [mem
>> 0xf6000000-0xf61fffff]
>> [ 2.097202] pci 0000:00:00.0: bridge window [mem
>> 0xf6300000-0xf63fffff pref]
>> [ 2.104539] pcieport 0000:00:00.0: Adding to iommu group 4
>> [ 2.110232] pcieport 0000:00:00.0: PME: Signaling with IRQ 38
>> [ 2.116141] pcieport 0000:00:00.0: AER: enabled with IRQ 38
>> [ 8.131135] ath10k_pci 0000:01:00.0: Adding to iommu group 4
>> [ 8.131874] ath10k_pci 0000:01:00.0: enabling device (0000 -> 0002)
>> [ 8.132203] ath10k_pci 0000:01:00.0: pci irq msi oper_irq_mode 2
>> irq_mode 0 reset_mode 0
>>
>> up to that point the log is the same as without SMMU enabled, except
>> "Adding to iommu group N" lines, and IRQ being 37
>>
>> [ 8.221328] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
>> [ 8.313362] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
>> [ 8.409373] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
>> [ 8.553433] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
>> [ 8.641370] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
>> [ 8.737979] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
>> [ 8.807356] ath10k_pci 0000:01:00.0: Failed to get pcie state addr:
>> -16
>> [ 8.814032] ath10k_pci 0000:01:00.0: failed to setup init config: -16
>> [ 8.820605] ath10k_pci 0000:01:00.0: could not power on hif bus (-16)
>> [ 8.827111] ath10k_pci 0000:01:00.0: could not probe fw (-16)
>>
>> Thank you!
>>
>>> v3 -> v4
>>> - call cfg_probe() impl hook a bit earlier which simplifies errata
>>> handling
>>> - use hi_lo_readq_relaxed() and hi_lo_writeq_relaxed() for register
>>> accessors
>>> - keep SMMU status disabled by default and enable where possible (DTS
>>> changes)
>>> - commit logs improvements and other minor fixes
>>>
>>> Hanna Hawa (1):
>>> iommu/arm-smmu: Workaround for Marvell Armada-AP806 SoC erratum
>>> #582743
>>>
>>> Marcin Wojtas (1):
>>> arm64: dts: marvell: add SMMU support
>>>
>>> Tomasz Nowicki (2):
>>> iommu/arm-smmu: Call configuration impl hook before consuming features
>>> dt-bindings: arm-smmu: add compatible string for Marvell Armada-AP806
>>> SMMU-500
>>>
>>> Documentation/arm64/silicon-errata.rst | 3 ++
>>> .../devicetree/bindings/iommu/arm,smmu.yaml | 4 ++
>>> arch/arm64/boot/dts/marvell/armada-7040.dtsi | 28 ++++++++++++
>>> arch/arm64/boot/dts/marvell/armada-8040.dtsi | 40 +++++++++++++++++
>>> arch/arm64/boot/dts/marvell/armada-ap80x.dtsi | 18 ++++++++
>>> drivers/iommu/arm-smmu-impl.c | 45 +++++++++++++++++++
>>> drivers/iommu/arm-smmu.c | 11 +++--
>>> 7 files changed, 145 insertions(+), 4 deletions(-)
>>>
>>> --
>>> 2.17.1
>>>
>>> _______________________________________________
>>> iommu mailing list
>>> [email protected]
>>> https://lists.linuxfoundation.org/mailman/listinfo/iommu
>>>
>>
On 10/23/20 12:33 PM, Robin Murphy wrote:
> On 2020-10-23 13:19, Tomasz Nowicki wrote:
>> Hi Denis,
>>
>> Sorry for late response, we had to check few things. Please see
>> comments inline.
>>
>> On 10/6/20 3:16 PM, Denis Odintsov wrote:
>>> Hi,
>>>
>>>> Am 15.07.2020 um 09:06 schrieb Tomasz Nowicki <[email protected]>:
>>>>
>>>> The series is meant to support SMMU for AP806 and a workaround
>>>> for accessing ARM SMMU 64bit registers is the gist of it.
>>>>
>>>> For the record, AP-806 can't access SMMU registers with 64bit width.
>>>> This patches split the readq/writeq into two 32bit accesses instead
>>>> and update DT bindings.
>>>>
>>>> The series was successfully tested on a vanilla v5.8-rc3 kernel and
>>>> Intel e1000e PCIe NIC. The same for platform devices like SATA and USB.
>>>>
>>>> For reference, previous versions are listed below:
>>>> V1: https://lkml.org/lkml/2018/10/15/373
>>>> V2: https://lkml.org/lkml/2019/7/11/426
>>>> V3: https://lkml.org/lkml/2020/7/2/1114
>>>>
>>>
>>> 1) After enabling SMMU on Armada 8040, and
>>> ARM_SMMU_DISABLE_BYPASS_BY_DEFAUL=y by default in kernel since
>>> 954a03be033c7cef80ddc232e7cbdb17df735663,
>>> internal eMMC is prevented from being initialised (as there is no
>>> iommus property for ap_sdhci0)
>>> Disabling "Disable bypass by default" make it work, but the patch
>>> highly suggest doing it properly.
>>> I wasn't able to find correct path for ap_sdhci for iommus in any
>>> publicly available documentation,
>>> would be highly appreciated addressed properly, thank you!
>>>
>>> 2) Second issue I got (btw I have ClearFog GT 8k armada-8040 based
>>> board) is mpci ath10k card.
>>> It is found, it is enumerated, it is visible in lspci, but it fails
>>> to be initialised. Here is the log:
>>
>> Firmware has to configure and assign device StreamIDs. Most of the
>> devices are configured properly and supported in public FW. However,
>> for both these cases (ap_sdhci0 and PCIe) some extra (u-boot/UEFI/ATF)
>> patches are required which are not available yet. Sorry we let that
>> happen.
>>
>> Since we have dependency on custom FW and we cannot enforce people to
>> patch their FW we will send the follow up fix patch (v5.9+) and revert
>> respective DTS changes.
>
> Note that it should be sufficient to simply keep the SMMU node disabled,
> rather than fully revert everything. For example, the PCIe SMMU for Arm
> Juno boards has been in that state for a long time - there are reasons
> why it isn't (yet) 100% usable for everyone, but it can easily be
> enabled locally for development (as I do).
>
Actually that was our plan :) but then we decided to keep DTS clean if
something is not used. Your reasoning, however, does make sense and we
will go for it.
Thanks,
Tomasz
>
>> The most important Armada-806 SMMU driver enhancements were merged so
>> people who still willing to use SMMU need to provide proper DTB and
>> use ARM_SMMU_DISABLE_BYPASS_BY_DEFAUL=n (or via kernel command line)
>> with extra cautious.
>>
>> Thanks,
>> Tomasz
>>
>>>
>>> [ 1.743754] armada8k-pcie f2600000.pcie: host bridge
>>> /cp0/pcie@f2600000 ranges:
>>> [ 1.751116] armada8k-pcie f2600000.pcie: MEM
>>> 0x00f6000000..0x00f6efffff -> 0x00f6000000
>>> [ 1.964690] armada8k-pcie f2600000.pcie: Link up
>>> [ 1.969379] armada8k-pcie f2600000.pcie: PCI host bridge to bus
>>> 0000:00
>>> [ 1.976026] pci_bus 0000:00: root bus resource [bus 00-ff]
>>> [ 1.981537] pci_bus 0000:00: root bus resource [mem
>>> 0xf6000000-0xf6efffff]
>>> [ 1.988462] pci 0000:00:00.0: [11ab:0110] type 01 class 0x060400
>>> [ 1.994504] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x000fffff]
>>> [ 2.000843] pci 0000:00:00.0: supports D1 D2
>>> [ 2.005132] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
>>> [ 2.011853] pci 0000:01:00.0: [168c:003c] type 00 class 0x028000
>>> [ 2.018001] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x001fffff
>>> 64bit]
>>> [ 2.025002] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0000ffff
>>> pref]
>>> [ 2.032111] pci 0000:01:00.0: supports D1 D2
>>> [ 2.049409] pci 0000:00:00.0: BAR 14: assigned [mem
>>> 0xf6000000-0xf61fffff]
>>> [ 2.056322] pci 0000:00:00.0: BAR 0: assigned [mem
>>> 0xf6200000-0xf62fffff]
>>> [ 2.063142] pci 0000:00:00.0: BAR 15: assigned [mem
>>> 0xf6300000-0xf63fffff pref]
>>> [ 2.070484] pci 0000:01:00.0: BAR 0: assigned [mem
>>> 0xf6000000-0xf61fffff 64bit]
>>> [ 2.077880] pci 0000:01:00.0: BAR 6: assigned [mem
>>> 0xf6300000-0xf630ffff pref]
>>> [ 2.085135] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
>>> [ 2.090384] pci 0000:00:00.0: bridge window [mem
>>> 0xf6000000-0xf61fffff]
>>> [ 2.097202] pci 0000:00:00.0: bridge window [mem
>>> 0xf6300000-0xf63fffff pref]
>>> [ 2.104539] pcieport 0000:00:00.0: Adding to iommu group 4
>>> [ 2.110232] pcieport 0000:00:00.0: PME: Signaling with IRQ 38
>>> [ 2.116141] pcieport 0000:00:00.0: AER: enabled with IRQ 38
>>> [ 8.131135] ath10k_pci 0000:01:00.0: Adding to iommu group 4
>>> [ 8.131874] ath10k_pci 0000:01:00.0: enabling device (0000 -> 0002)
>>> [ 8.132203] ath10k_pci 0000:01:00.0: pci irq msi oper_irq_mode 2
>>> irq_mode 0 reset_mode 0
>>>
>>> up to that point the log is the same as without SMMU enabled, except
>>> "Adding to iommu group N" lines, and IRQ being 37
>>>
>>> [ 8.221328] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
>>> [ 8.313362] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
>>> [ 8.409373] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
>>> [ 8.553433] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
>>> [ 8.641370] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
>>> [ 8.737979] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
>>> [ 8.807356] ath10k_pci 0000:01:00.0: Failed to get pcie state
>>> addr: -16
>>> [ 8.814032] ath10k_pci 0000:01:00.0: failed to setup init config: -16
>>> [ 8.820605] ath10k_pci 0000:01:00.0: could not power on hif bus (-16)
>>> [ 8.827111] ath10k_pci 0000:01:00.0: could not probe fw (-16)
>>>
>>> Thank you!
>>>
>>>> v3 -> v4
>>>> - call cfg_probe() impl hook a bit earlier which simplifies errata
>>>> handling
>>>> - use hi_lo_readq_relaxed() and hi_lo_writeq_relaxed() for register
>>>> accessors
>>>> - keep SMMU status disabled by default and enable where possible
>>>> (DTS changes)
>>>> - commit logs improvements and other minor fixes
>>>>
>>>> Hanna Hawa (1):
>>>> iommu/arm-smmu: Workaround for Marvell Armada-AP806 SoC erratum
>>>> #582743
>>>>
>>>> Marcin Wojtas (1):
>>>> arm64: dts: marvell: add SMMU support
>>>>
>>>> Tomasz Nowicki (2):
>>>> iommu/arm-smmu: Call configuration impl hook before consuming
>>>> features
>>>> dt-bindings: arm-smmu: add compatible string for Marvell Armada-AP806
>>>> SMMU-500
>>>>
>>>> Documentation/arm64/silicon-errata.rst | 3 ++
>>>> .../devicetree/bindings/iommu/arm,smmu.yaml | 4 ++
>>>> arch/arm64/boot/dts/marvell/armada-7040.dtsi | 28 ++++++++++++
>>>> arch/arm64/boot/dts/marvell/armada-8040.dtsi | 40 +++++++++++++++++
>>>> arch/arm64/boot/dts/marvell/armada-ap80x.dtsi | 18 ++++++++
>>>> drivers/iommu/arm-smmu-impl.c | 45 +++++++++++++++++++
>>>> drivers/iommu/arm-smmu.c | 11 +++--
>>>> 7 files changed, 145 insertions(+), 4 deletions(-)
>>>>
>>>> --
>>>> 2.17.1
>>>>
>>>> _______________________________________________
>>>> iommu mailing list
>>>> [email protected]
>>>> https://lists.linuxfoundation.org/mailman/listinfo/iommu
>>>>
>>>
Hi,
> Am 23.10.2020 um 14:19 schrieb Tomasz Nowicki <[email protected]>:
>
> Hi Denis,
>
> Sorry for late response, we had to check few things. Please see comments inline.
>
> On 10/6/20 3:16 PM, Denis Odintsov wrote:
>> Hi,
>>> Am 15.07.2020 um 09:06 schrieb Tomasz Nowicki <[email protected]>:
>>>
>>> The series is meant to support SMMU for AP806 and a workaround
>>> for accessing ARM SMMU 64bit registers is the gist of it.
>>>
>>> For the record, AP-806 can't access SMMU registers with 64bit width.
>>> This patches split the readq/writeq into two 32bit accesses instead
>>> and update DT bindings.
>>>
>>> The series was successfully tested on a vanilla v5.8-rc3 kernel and
>>> Intel e1000e PCIe NIC. The same for platform devices like SATA and USB.
>>>
>>> For reference, previous versions are listed below:
>>> V1: https://lkml.org/lkml/2018/10/15/373
>>> V2: https://lkml.org/lkml/2019/7/11/426
>>> V3: https://lkml.org/lkml/2020/7/2/1114
>>>
>> 1) After enabling SMMU on Armada 8040, and ARM_SMMU_DISABLE_BYPASS_BY_DEFAUL=y by default in kernel since 954a03be033c7cef80ddc232e7cbdb17df735663,
>> internal eMMC is prevented from being initialised (as there is no iommus property for ap_sdhci0)
>> Disabling "Disable bypass by default" make it work, but the patch highly suggest doing it properly.
>> I wasn't able to find correct path for ap_sdhci for iommus in any publicly available documentation,
>> would be highly appreciated addressed properly, thank you!
>> 2) Second issue I got (btw I have ClearFog GT 8k armada-8040 based board) is mpci ath10k card.
>> It is found, it is enumerated, it is visible in lspci, but it fails to be initialised. Here is the log:
>
> Firmware has to configure and assign device StreamIDs. Most of the devices are configured properly and supported in public FW. However, for both these cases (ap_sdhci0 and PCIe) some extra (u-boot/UEFI/ATF) patches are required which are not available yet. Sorry we let that happen.
>
> Since we have dependency on custom FW and we cannot enforce people to patch their FW we will send the follow up fix patch (v5.9+) and revert respective DTS changes.
>
> The most important Armada-806 SMMU driver enhancements were merged so people who still willing to use SMMU need to provide proper DTB and use ARM_SMMU_DISABLE_BYPASS_BY_DEFAUL=n (or via kernel command line) with extra cautious.
>
Thank you very much for the reply, I'm a mere computer enthusiast who like to progress with the technology and keep up with the software.
There was no harm at all with all those changes, and I see how they all are planned for a good reason.
I'm looking forward for that patches and further progress, and thank you for your work.
Denis.
> Thanks,
> Tomasz
>
>> [ 1.743754] armada8k-pcie f2600000.pcie: host bridge /cp0/pcie@f2600000 ranges:
>> [ 1.751116] armada8k-pcie f2600000.pcie: MEM 0x00f6000000..0x00f6efffff -> 0x00f6000000
>> [ 1.964690] armada8k-pcie f2600000.pcie: Link up
>> [ 1.969379] armada8k-pcie f2600000.pcie: PCI host bridge to bus 0000:00
>> [ 1.976026] pci_bus 0000:00: root bus resource [bus 00-ff]
>> [ 1.981537] pci_bus 0000:00: root bus resource [mem 0xf6000000-0xf6efffff]
>> [ 1.988462] pci 0000:00:00.0: [11ab:0110] type 01 class 0x060400
>> [ 1.994504] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x000fffff]
>> [ 2.000843] pci 0000:00:00.0: supports D1 D2
>> [ 2.005132] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
>> [ 2.011853] pci 0000:01:00.0: [168c:003c] type 00 class 0x028000
>> [ 2.018001] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x001fffff 64bit]
>> [ 2.025002] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
>> [ 2.032111] pci 0000:01:00.0: supports D1 D2
>> [ 2.049409] pci 0000:00:00.0: BAR 14: assigned [mem 0xf6000000-0xf61fffff]
>> [ 2.056322] pci 0000:00:00.0: BAR 0: assigned [mem 0xf6200000-0xf62fffff]
>> [ 2.063142] pci 0000:00:00.0: BAR 15: assigned [mem 0xf6300000-0xf63fffff pref]
>> [ 2.070484] pci 0000:01:00.0: BAR 0: assigned [mem 0xf6000000-0xf61fffff 64bit]
>> [ 2.077880] pci 0000:01:00.0: BAR 6: assigned [mem 0xf6300000-0xf630ffff pref]
>> [ 2.085135] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
>> [ 2.090384] pci 0000:00:00.0: bridge window [mem 0xf6000000-0xf61fffff]
>> [ 2.097202] pci 0000:00:00.0: bridge window [mem 0xf6300000-0xf63fffff pref]
>> [ 2.104539] pcieport 0000:00:00.0: Adding to iommu group 4
>> [ 2.110232] pcieport 0000:00:00.0: PME: Signaling with IRQ 38
>> [ 2.116141] pcieport 0000:00:00.0: AER: enabled with IRQ 38
>> [ 8.131135] ath10k_pci 0000:01:00.0: Adding to iommu group 4
>> [ 8.131874] ath10k_pci 0000:01:00.0: enabling device (0000 -> 0002)
>> [ 8.132203] ath10k_pci 0000:01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
>> up to that point the log is the same as without SMMU enabled, except "Adding to iommu group N" lines, and IRQ being 37
>> [ 8.221328] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
>> [ 8.313362] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
>> [ 8.409373] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
>> [ 8.553433] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
>> [ 8.641370] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
>> [ 8.737979] ath10k_pci 0000:01:00.0: failed to poke copy engine: -16
>> [ 8.807356] ath10k_pci 0000:01:00.0: Failed to get pcie state addr: -16
>> [ 8.814032] ath10k_pci 0000:01:00.0: failed to setup init config: -16
>> [ 8.820605] ath10k_pci 0000:01:00.0: could not power on hif bus (-16)
>> [ 8.827111] ath10k_pci 0000:01:00.0: could not probe fw (-16)
>> Thank you!
>>> v3 -> v4
>>> - call cfg_probe() impl hook a bit earlier which simplifies errata handling
>>> - use hi_lo_readq_relaxed() and hi_lo_writeq_relaxed() for register accessors
>>> - keep SMMU status disabled by default and enable where possible (DTS changes)
>>> - commit logs improvements and other minor fixes
>>>
>>> Hanna Hawa (1):
>>> iommu/arm-smmu: Workaround for Marvell Armada-AP806 SoC erratum
>>> #582743
>>>
>>> Marcin Wojtas (1):
>>> arm64: dts: marvell: add SMMU support
>>>
>>> Tomasz Nowicki (2):
>>> iommu/arm-smmu: Call configuration impl hook before consuming features
>>> dt-bindings: arm-smmu: add compatible string for Marvell Armada-AP806
>>> SMMU-500
>>>
>>> Documentation/arm64/silicon-errata.rst | 3 ++
>>> .../devicetree/bindings/iommu/arm,smmu.yaml | 4 ++
>>> arch/arm64/boot/dts/marvell/armada-7040.dtsi | 28 ++++++++++++
>>> arch/arm64/boot/dts/marvell/armada-8040.dtsi | 40 +++++++++++++++++
>>> arch/arm64/boot/dts/marvell/armada-ap80x.dtsi | 18 ++++++++
>>> drivers/iommu/arm-smmu-impl.c | 45 +++++++++++++++++++
>>> drivers/iommu/arm-smmu.c | 11 +++--
>>> 7 files changed, 145 insertions(+), 4 deletions(-)
>>>
>>> --
>>> 2.17.1
>>>
>>> _______________________________________________
>>> iommu mailing list
>>> [email protected]
>>> https://lists.linuxfoundation.org/mailman/listinfo/iommu