Hi,
This series enables Hantro support for RK3588. It is based on these two
previous series from Emmanuel Gil Peyrot and Jianfeng Liu, which looked
stall to me. Considering the full driver is already upstream, I think
this low hanging fruit should be enabled in 6.11:
* https://lore.kernel.org/all/[email protected]/
* https://lore.kernel.org/linux-rockchip/[email protected]/
Their series got some feedback from Nicolas Dufresne, that there should be a
plan how multi-core processing will be handled once it is supported in the
kernel. I had a look (and internal discussion with Nicolas) and came up with a
patch, which allows describing all the Hantro IP in DT. The driver will only
probe for the first instance. This involves dropping the RK3568 compatible
for the VEPU121, so that only kernels with the driver change will try to
handle these IP. Once the kernel is capable of multi-core support, the same
technique to disable cores 1-3 can be used to combine them all into one
cluster.
We also discussed, if they should be described as a cluster (e.g. by creating
some kind of virtual bus for the 4 encoders in DT). Apparently the VSI doc
describes the grouping of up to 4 instances. But there is no obvious reason
why only these groups can be used as a cluster. It seems that even the 5th
encoder from the combo VPU121 could be used together with the other clustered
cores in theory. In practice this is probably a bad idea because of the shared
cache of that encoder. Since that is handled with a different compatible, this
can be thought about at a later point of time and handled in the kernel. Thus
no special cluster description is needed in DT.
Changes since PATCHv5:
* Fix binding for vepu121 (use enum)
* split hantro driver patch (multicore / vepu121 compatible)
* move video-codec@fdb50000 node to correct position
* change "jpeg_enc*" alias to "vepu121_*"
* change "vpu_*" alias to "vpu121_*" (to be consistent)
Changes since PATCHv3 (VEPU121) / PATCHv4 (VPU121)
* combine both patchsets, since there is some overleap
* add patch to disable multi-core handling in the hantro driver
* drop the RK3568 fallback compatible for VEPU (see above for the reason)
* describe all RK3588 VEPU cores (possible because of driver change)
Greetings,
-- Sebastian
Emmanuel Gil Peyrot (2):
media: dt-bindings: rk3568-vepu: Add RK3588 VEPU121
arm64: dts: rockchip: Add VEPU121 to RK3588
Jianfeng Liu (2):
media: dt-bindings: rockchip-vpu: Add RK3588 VPU121
arm64: dts: rockchip: Add VPU121 support for RK3588
Sebastian Reichel (2):
media: hantro: Disable multicore support
media: hantro: Add RK3588 VEPU121
.../bindings/media/rockchip,rk3568-vepu.yaml | 1 +
.../bindings/media/rockchip-vpu.yaml | 3 +
arch/arm64/boot/dts/rockchip/rk3588s.dtsi | 101 ++++++++++++++++++
.../media/platform/verisilicon/hantro_drv.c | 38 +++++++
4 files changed, 143 insertions(+)
--
2.43.0
From: Emmanuel Gil Peyrot <[email protected]>
This encoder-only device is present four times on this SoC, and should
support everything the rk3568 vepu supports (so JPEG, H.264 and VP8
encoding). No fallback compatible has been added, since the operating
systems might already support RK3568 VEPU and want to avoid registering
four of them separately considering they can be used as a cluster.
Signed-off-by: Emmanuel Gil Peyrot <[email protected]>
Signed-off-by: Sebastian Reichel <[email protected]>
---
.../devicetree/bindings/media/rockchip,rk3568-vepu.yaml | 1 +
1 file changed, 1 insertion(+)
diff --git a/Documentation/devicetree/bindings/media/rockchip,rk3568-vepu.yaml b/Documentation/devicetree/bindings/media/rockchip,rk3568-vepu.yaml
index 9d90d8d0565a..947ad699cc5e 100644
--- a/Documentation/devicetree/bindings/media/rockchip,rk3568-vepu.yaml
+++ b/Documentation/devicetree/bindings/media/rockchip,rk3568-vepu.yaml
@@ -17,6 +17,7 @@ properties:
compatible:
enum:
- rockchip,rk3568-vepu
+ - rockchip,rk3588-vepu121
reg:
maxItems: 1
--
2.43.0
From: Jianfeng Liu <[email protected]>
Enable Hantro G1 video decoder in RK3588's devicetree.
Tested with FFmpeg v4l2_request code taken from [1]
with MPEG2, H.264 and VP8 samples.
[1] https://github.com/LibreELEC/LibreELEC.tv/blob/master/packages/multimedia/ffmpeg/patches/v4l2-request/ffmpeg-001-v4l2-request.patch
Signed-off-by: Jianfeng Liu <[email protected]>
Tested-by: Hugh Cole-Baker <[email protected]>
Signed-off-by: Sebastian Reichel <[email protected]>
---
arch/arm64/boot/dts/rockchip/rk3588s.dtsi | 21 +++++++++++++++++++++
1 file changed, 21 insertions(+)
diff --git a/arch/arm64/boot/dts/rockchip/rk3588s.dtsi b/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
index dd85d4e55922..c0466982646f 100644
--- a/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
+++ b/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
@@ -1159,6 +1159,27 @@ power-domain@RK3588_PD_SDMMC {
};
};
+ vpu121: video-codec@fdb50000 {
+ compatible = "rockchip,rk3588-vpu121", "rockchip,rk3568-vpu";
+ reg = <0x0 0xfdb50000 0x0 0x800>;
+ interrupts = <GIC_SPI 119 IRQ_TYPE_LEVEL_HIGH 0>;
+ interrupt-names = "vdpu";
+ clocks = <&cru ACLK_VPU>, <&cru HCLK_VPU>;
+ clock-names = "aclk", "hclk";
+ iommus = <&vpu121_mmu>;
+ power-domains = <&power RK3588_PD_VDPU>;
+ };
+
+ vpu121_mmu: iommu@fdb50800 {
+ compatible = "rockchip,rk3588-iommu", "rockchip,rk3568-iommu";
+ reg = <0x0 0xfdb50800 0x0 0x40>;
+ interrupts = <GIC_SPI 118 IRQ_TYPE_LEVEL_HIGH 0>;
+ clock-names = "aclk", "iface";
+ clocks = <&cru ACLK_VPU>, <&cru HCLK_VPU>;
+ power-domains = <&power RK3588_PD_VDPU>;
+ #iommu-cells = <0>;
+ };
+
vepu121_0: video-codec@fdba0000 {
compatible = "rockchip,rk3588-vepu121";
reg = <0x0 0xfdba0000 0x0 0x800>;
--
2.43.0
From: Emmanuel Gil Peyrot <[email protected]>
RK3588 has 4 Hantro G1 encoder-only cores. They are all independent IP,
but can be used as a cluster (i.e. sharing work between the cores).
These cores are called VEPU121 in the TRM. The TRM describes one more
VEPU121, but that is combined with a Hantro H1. That one will be handled
using the VPU binding instead.
Signed-off-by: Emmanuel Gil Peyrot <[email protected]>
Signed-off-by: Sebastian Reichel <[email protected]>
---
arch/arm64/boot/dts/rockchip/rk3588s.dtsi | 80 +++++++++++++++++++++++
1 file changed, 80 insertions(+)
diff --git a/arch/arm64/boot/dts/rockchip/rk3588s.dtsi b/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
index 6ac5ac8b48ab..dd85d4e55922 100644
--- a/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
+++ b/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
@@ -1159,6 +1159,86 @@ power-domain@RK3588_PD_SDMMC {
};
};
+ vepu121_0: video-codec@fdba0000 {
+ compatible = "rockchip,rk3588-vepu121";
+ reg = <0x0 0xfdba0000 0x0 0x800>;
+ interrupts = <GIC_SPI 122 IRQ_TYPE_LEVEL_HIGH 0>;
+ clocks = <&cru ACLK_JPEG_ENCODER0>, <&cru HCLK_JPEG_ENCODER0>;
+ clock-names = "aclk", "hclk";
+ iommus = <&vepu121_0_mmu>;
+ power-domains = <&power RK3588_PD_VDPU>;
+ };
+
+ vepu121_0_mmu: iommu@fdba0800 {
+ compatible = "rockchip,rk3588-iommu", "rockchip,rk3568-iommu";
+ reg = <0x0 0xfdba0800 0x0 0x40>;
+ interrupts = <GIC_SPI 121 IRQ_TYPE_LEVEL_HIGH 0>;
+ clocks = <&cru ACLK_JPEG_ENCODER0>, <&cru HCLK_JPEG_ENCODER0>;
+ clock-names = "aclk", "iface";
+ power-domains = <&power RK3588_PD_VDPU>;
+ #iommu-cells = <0>;
+ };
+
+ vepu121_1: video-codec@fdba4000 {
+ compatible = "rockchip,rk3588-vepu121";
+ reg = <0x0 0xfdba4000 0x0 0x800>;
+ interrupts = <GIC_SPI 124 IRQ_TYPE_LEVEL_HIGH 0>;
+ clocks = <&cru ACLK_JPEG_ENCODER1>, <&cru HCLK_JPEG_ENCODER1>;
+ clock-names = "aclk", "hclk";
+ iommus = <&vepu121_1_mmu>;
+ power-domains = <&power RK3588_PD_VDPU>;
+ };
+
+ vepu121_1_mmu: iommu@fdba4800 {
+ compatible = "rockchip,rk3588-iommu", "rockchip,rk3568-iommu";
+ reg = <0x0 0xfdba4800 0x0 0x40>;
+ interrupts = <GIC_SPI 123 IRQ_TYPE_LEVEL_HIGH 0>;
+ clocks = <&cru ACLK_JPEG_ENCODER1>, <&cru HCLK_JPEG_ENCODER1>;
+ clock-names = "aclk", "iface";
+ power-domains = <&power RK3588_PD_VDPU>;
+ #iommu-cells = <0>;
+ };
+
+ vepu121_2: video-codec@fdba8000 {
+ compatible = "rockchip,rk3588-vepu121";
+ reg = <0x0 0xfdba8000 0x0 0x800>;
+ interrupts = <GIC_SPI 126 IRQ_TYPE_LEVEL_HIGH 0>;
+ clocks = <&cru ACLK_JPEG_ENCODER2>, <&cru HCLK_JPEG_ENCODER2>;
+ clock-names = "aclk", "hclk";
+ iommus = <&vepu121_2_mmu>;
+ power-domains = <&power RK3588_PD_VDPU>;
+ };
+
+ vepu121_2_mmu: iommu@fdba8800 {
+ compatible = "rockchip,rk3588-iommu", "rockchip,rk3568-iommu";
+ reg = <0x0 0xfdba8800 0x0 0x40>;
+ interrupts = <GIC_SPI 125 IRQ_TYPE_LEVEL_HIGH 0>;
+ clocks = <&cru ACLK_JPEG_ENCODER2>, <&cru HCLK_JPEG_ENCODER2>;
+ clock-names = "aclk", "iface";
+ power-domains = <&power RK3588_PD_VDPU>;
+ #iommu-cells = <0>;
+ };
+
+ vepu121_3: video-codec@fdbac000 {
+ compatible = "rockchip,rk3588-vepu121";
+ reg = <0x0 0xfdbac000 0x0 0x800>;
+ interrupts = <GIC_SPI 128 IRQ_TYPE_LEVEL_HIGH 0>;
+ clocks = <&cru ACLK_JPEG_ENCODER3>, <&cru HCLK_JPEG_ENCODER3>;
+ clock-names = "aclk", "hclk";
+ iommus = <&vepu121_3_mmu>;
+ power-domains = <&power RK3588_PD_VDPU>;
+ };
+
+ vepu121_3_mmu: iommu@fdbac800 {
+ compatible = "rockchip,rk3588-iommu", "rockchip,rk3568-iommu";
+ reg = <0x0 0xfdbac800 0x0 0x40>;
+ interrupts = <GIC_SPI 127 IRQ_TYPE_LEVEL_HIGH 0>;
+ clocks = <&cru ACLK_JPEG_ENCODER3>, <&cru HCLK_JPEG_ENCODER3>;
+ clock-names = "aclk", "iface";
+ power-domains = <&power RK3588_PD_VDPU>;
+ #iommu-cells = <0>;
+ };
+
av1d: video-codec@fdc70000 {
compatible = "rockchip,rk3588-av1-vpu";
reg = <0x0 0xfdc70000 0x0 0x800>;
--
2.43.0
On Thu, Jun 13, 2024 at 03:48:42PM +0200, Sebastian Reichel wrote:
> From: Emmanuel Gil Peyrot <[email protected]>
>
> This encoder-only device is present four times on this SoC, and should
> support everything the rk3568 vepu supports (so JPEG, H.264 and VP8
> encoding). No fallback compatible has been added, since the operating
> systems might already support RK3568 VEPU and want to avoid registering
> four of them separately considering they can be used as a cluster.
>
> Signed-off-by: Emmanuel Gil Peyrot <[email protected]>
> Signed-off-by: Sebastian Reichel <[email protected]>
Acked-by: Conor Dooley <[email protected]>
Le jeudi 13 juin 2024 à 15:48 +0200, Sebastian Reichel a écrit :
> From: Emmanuel Gil Peyrot <[email protected]>
>
> RK3588 has 4 Hantro G1 encoder-only cores. They are all independent IP,
Hantro H1
H1 is the encoder core, G1 is the decoder core, and this exists as a combo on
this platform (vpu121).
> but can be used as a cluster (i.e. sharing work between the cores).
> These cores are called VEPU121 in the TRM. The TRM describes one more
> VEPU121, but that is combined with a Hantro H1. That one will be handled
> using the VPU binding instead.
>
> Signed-off-by: Emmanuel Gil Peyrot <[email protected]>
> Signed-off-by: Sebastian Reichel <[email protected]>
Acked-by: Nicolas Dufresne <[email protected]>
> ---
> arch/arm64/boot/dts/rockchip/rk3588s.dtsi | 80 +++++++++++++++++++++++
> 1 file changed, 80 insertions(+)
>
> diff --git a/arch/arm64/boot/dts/rockchip/rk3588s.dtsi b/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
> index 6ac5ac8b48ab..dd85d4e55922 100644
> --- a/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
> +++ b/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
> @@ -1159,6 +1159,86 @@ power-domain@RK3588_PD_SDMMC {
> };
> };
>
> + vepu121_0: video-codec@fdba0000 {
> + compatible = "rockchip,rk3588-vepu121";
> + reg = <0x0 0xfdba0000 0x0 0x800>;
> + interrupts = <GIC_SPI 122 IRQ_TYPE_LEVEL_HIGH 0>;
> + clocks = <&cru ACLK_JPEG_ENCODER0>, <&cru HCLK_JPEG_ENCODER0>;
> + clock-names = "aclk", "hclk";
> + iommus = <&vepu121_0_mmu>;
> + power-domains = <&power RK3588_PD_VDPU>;
> + };
> +
> + vepu121_0_mmu: iommu@fdba0800 {
> + compatible = "rockchip,rk3588-iommu", "rockchip,rk3568-iommu";
> + reg = <0x0 0xfdba0800 0x0 0x40>;
> + interrupts = <GIC_SPI 121 IRQ_TYPE_LEVEL_HIGH 0>;
> + clocks = <&cru ACLK_JPEG_ENCODER0>, <&cru HCLK_JPEG_ENCODER0>;
> + clock-names = "aclk", "iface";
> + power-domains = <&power RK3588_PD_VDPU>;
> + #iommu-cells = <0>;
> + };
> +
> + vepu121_1: video-codec@fdba4000 {
> + compatible = "rockchip,rk3588-vepu121";
> + reg = <0x0 0xfdba4000 0x0 0x800>;
> + interrupts = <GIC_SPI 124 IRQ_TYPE_LEVEL_HIGH 0>;
> + clocks = <&cru ACLK_JPEG_ENCODER1>, <&cru HCLK_JPEG_ENCODER1>;
> + clock-names = "aclk", "hclk";
> + iommus = <&vepu121_1_mmu>;
> + power-domains = <&power RK3588_PD_VDPU>;
> + };
> +
> + vepu121_1_mmu: iommu@fdba4800 {
> + compatible = "rockchip,rk3588-iommu", "rockchip,rk3568-iommu";
> + reg = <0x0 0xfdba4800 0x0 0x40>;
> + interrupts = <GIC_SPI 123 IRQ_TYPE_LEVEL_HIGH 0>;
> + clocks = <&cru ACLK_JPEG_ENCODER1>, <&cru HCLK_JPEG_ENCODER1>;
> + clock-names = "aclk", "iface";
> + power-domains = <&power RK3588_PD_VDPU>;
> + #iommu-cells = <0>;
> + };
> +
> + vepu121_2: video-codec@fdba8000 {
> + compatible = "rockchip,rk3588-vepu121";
> + reg = <0x0 0xfdba8000 0x0 0x800>;
> + interrupts = <GIC_SPI 126 IRQ_TYPE_LEVEL_HIGH 0>;
> + clocks = <&cru ACLK_JPEG_ENCODER2>, <&cru HCLK_JPEG_ENCODER2>;
> + clock-names = "aclk", "hclk";
> + iommus = <&vepu121_2_mmu>;
> + power-domains = <&power RK3588_PD_VDPU>;
> + };
> +
> + vepu121_2_mmu: iommu@fdba8800 {
> + compatible = "rockchip,rk3588-iommu", "rockchip,rk3568-iommu";
> + reg = <0x0 0xfdba8800 0x0 0x40>;
> + interrupts = <GIC_SPI 125 IRQ_TYPE_LEVEL_HIGH 0>;
> + clocks = <&cru ACLK_JPEG_ENCODER2>, <&cru HCLK_JPEG_ENCODER2>;
> + clock-names = "aclk", "iface";
> + power-domains = <&power RK3588_PD_VDPU>;
> + #iommu-cells = <0>;
> + };
> +
> + vepu121_3: video-codec@fdbac000 {
> + compatible = "rockchip,rk3588-vepu121";
> + reg = <0x0 0xfdbac000 0x0 0x800>;
> + interrupts = <GIC_SPI 128 IRQ_TYPE_LEVEL_HIGH 0>;
> + clocks = <&cru ACLK_JPEG_ENCODER3>, <&cru HCLK_JPEG_ENCODER3>;
> + clock-names = "aclk", "hclk";
> + iommus = <&vepu121_3_mmu>;
> + power-domains = <&power RK3588_PD_VDPU>;
> + };
> +
> + vepu121_3_mmu: iommu@fdbac800 {
> + compatible = "rockchip,rk3588-iommu", "rockchip,rk3568-iommu";
> + reg = <0x0 0xfdbac800 0x0 0x40>;
> + interrupts = <GIC_SPI 127 IRQ_TYPE_LEVEL_HIGH 0>;
> + clocks = <&cru ACLK_JPEG_ENCODER3>, <&cru HCLK_JPEG_ENCODER3>;
> + clock-names = "aclk", "iface";
> + power-domains = <&power RK3588_PD_VDPU>;
> + #iommu-cells = <0>;
> + };
> +
> av1d: video-codec@fdc70000 {
> compatible = "rockchip,rk3588-av1-vpu";
> reg = <0x0 0xfdc70000 0x0 0x800>;