2022-01-26 03:00:34

by Adam Ford

[permalink] [raw]
Subject: [PATCH V4 00/11] media: hantro: imx8mq/imx8mm: Let VPU decoders get controlled by vpu-blk-ctrl

Both the i.MX8MQ and i.MX8MM have G1 and G2 decoders.
The two decoders are similar, but the imx8mm lacks the
post-processor, so they will have distinct compatible flags.

From what I can tell, the G2 decoder wasn't working, so splitting
the i.MX8MQ VPU into G1 and G2 makes it easier to control them
independently since the TRM of both the i.MX8MQ and
i.MX8MM list them as distinct IP blocks. This also allowed G2 to
become available.

With them being split, the power-domain can shift to the
vpu-blk-ctrl which is available on both i.MX8MQ and i.MX8MM,
but some of bits are different, so they'll have separate bindings.

Lastly, with the G1 and G2 operational, enable the i.MX8MM.
On the i.MX8MM, the clock speed of 600MHz was chosen to match
the default of the kernel repo from NXP and can be overwritten
by board files for anyone who under/over volts the power rail.

There seems to be some disagreement between the TRM and the Datasheet
for the imx8mq as to whether the speed should be 300MHz (TRM) or
600MHz (datasheet), so feedback from NXP would be very much
appreciated.

The repo used as the starting point was:
5.17-rc1

Fluster was run on both i.MX8MM and i.MX8MQ

At 600 MHz, the i.MX8MM had the following:

./fluster.py run -d GStreamer-VP8-V4L2SL-Gst1.0
Ran 55/61 tests successfully in 8.299 secs

./fluster.py run -dGStreamer-H.264-V4L2SL-Gst1.0
Ran 90/135 tests successfully in 71.200 secs

./fluster.py run -d GStreamer-VP9-V4L2SL-Gst1.0
Ran 139/303 tests successfully in 218.079 secs

The i.MX8MQ had the following:

./fluster.py run -d GStreamer-VP8-V4L2SL-Gst1.0
Ran 55/61 tests successfully in 7.732 secs

./fluster.py run -dGStreamer-H.264-V4L2SL-Gst1.0
Ran 90/135 tests successfully in 58.558 secs

./fluster.py run -d GStreamer-VP9-V4L2SL-Gst1.0
Ran 144/303 tests successfully in 271.373 secs

V4: Add a comment and a notice based on feedback from Ezequiel Garcia
if the older compatible flag is used.
Add new patch which removes a reference to vpu from imx8mq-tqma8mq
because the VPU is enabled by default and it's not needed. Without
this change, its device tree would fail to build.

V3: Rebase on 5.17-RC1.
Remove imx8mm-vpu-g2 since it's identical to imx8mq-vpu-g2
Remove unnecessary examples in YAML files.
Remove some unused variables.

V2: Remove references to legacy dt-binding from YAML, but keep
it in the driver so older device trees can still be used.
Fix typos in YAML
Remove reg-names, interrupt-names, and clock-names from YAML,
since each node will only have one of each, they're not necessary
Add Fluster scores to cover letter for i.MX8MQ



Adam Ford (8):
arm64: dts: imx8mq-tqma8mq: Remove redundant vpu reference
dt-bindings: media: nxp, imx8mq-vpu: Split G1 and G2 nodes
media: hantro: Allow i.MX8MQ G1 and G2 to run independently
arm64: dts: imx8mq: Enable both G1 and G2 VPU's with vpu-blk-ctrl
arm64: dts: imx8mm: Fix VPU Hanging
dt-bindings: media: nxp, imx8mq-vpu: Add support for G1 on imx8mm
media: hantro: Add support for i.MX8MM Hantro-G1
arm64: dts: imx8mm: Enable Hantro G1 and G2 video decoders

Lucas Stach (3):
dt-bindings: power: imx8mq: add defines for VPU blk-ctrl domains
dt-bindings: soc: add binding for i.MX8MQ VPU blk-ctrl
soc: imx: imx8m-blk-ctrl: add i.MX8MQ VPU blk-ctrl

.../bindings/media/nxp,imx8mq-vpu.yaml | 68 ++++++++----------
.../soc/imx/fsl,imx8mq-vpu-blk-ctrl.yaml | 71 +++++++++++++++++++
arch/arm64/boot/dts/freescale/imx8mm.dtsi | 23 +++++-
.../boot/dts/freescale/imx8mq-tqma8mq.dtsi | 4 --
arch/arm64/boot/dts/freescale/imx8mq.dtsi | 63 +++++++++-------
drivers/soc/imx/imx8m-blk-ctrl.c | 66 +++++++++++++++++
drivers/staging/media/hantro/hantro_drv.c | 11 +++
drivers/staging/media/hantro/hantro_hw.h | 2 +
drivers/staging/media/hantro/imx8m_vpu_hw.c | 62 ++++++++++++----
include/dt-bindings/power/imx8mq-power.h | 3 +
10 files changed, 292 insertions(+), 81 deletions(-)
create mode 100644 Documentation/devicetree/bindings/soc/imx/fsl,imx8mq-vpu-blk-ctrl.yaml


base-commit: e783362eb54cd99b2cac8b3a9aeac942e6f6ac07
--
2.32.0


2022-01-26 03:00:43

by Adam Ford

[permalink] [raw]
Subject: [PATCH V4 11/11] arm64: dts: imx8mm: Enable Hantro G1 and G2 video decoders

There are two decoders on the i.MX8M Mini controlled by the
vpu-blk-ctrl. The G1 supports H264 and VP8 while the
G2 support HEVC and VP9.

Signed-off-by: Adam Ford <[email protected]>
Reviewed-by: Ezequiel Garcia <[email protected]>

diff --git a/arch/arm64/boot/dts/freescale/imx8mm.dtsi b/arch/arm64/boot/dts/freescale/imx8mm.dtsi
index 0c7a72c51a31..98aec4421713 100644
--- a/arch/arm64/boot/dts/freescale/imx8mm.dtsi
+++ b/arch/arm64/boot/dts/freescale/imx8mm.dtsi
@@ -1272,6 +1272,22 @@ gpu_2d: gpu@38008000 {
power-domains = <&pgc_gpu>;
};

+ vpu_g1: video-codec@38300000 {
+ compatible = "nxp,imx8mm-vpu-g1";
+ reg = <0x38300000 0x10000>;
+ interrupts = <GIC_SPI 7 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&clk IMX8MM_CLK_VPU_G1_ROOT>;
+ power-domains = <&vpu_blk_ctrl IMX8MM_VPUBLK_PD_G1>;
+ };
+
+ vpu_g2: video-codec@38310000 {
+ compatible = "nxp,imx8mq-vpu-g2";
+ reg = <0x38310000 0x10000>;
+ interrupts = <GIC_SPI 8 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&clk IMX8MM_CLK_VPU_G2_ROOT>;
+ power-domains = <&vpu_blk_ctrl IMX8MM_VPUBLK_PD_G2>;
+ };
+
vpu_blk_ctrl: blk-ctrl@38330000 {
compatible = "fsl,imx8mm-vpu-blk-ctrl", "syscon";
reg = <0x38330000 0x100>;
@@ -1282,6 +1298,12 @@ vpu_blk_ctrl: blk-ctrl@38330000 {
<&clk IMX8MM_CLK_VPU_G2_ROOT>,
<&clk IMX8MM_CLK_VPU_H1_ROOT>;
clock-names = "g1", "g2", "h1";
+ assigned-clocks = <&clk IMX8MM_CLK_VPU_G1>,
+ <&clk IMX8MM_CLK_VPU_G2>;
+ assigned-clock-parents = <&clk IMX8MM_VPU_PLL_OUT>,
+ <&clk IMX8MM_VPU_PLL_OUT>;
+ assigned-clock-rates = <600000000>,
+ <600000000>;
#power-domain-cells = <1>;
};

--
2.32.0

2022-01-26 03:01:12

by Adam Ford

[permalink] [raw]
Subject: [PATCH V4 09/11] dt-bindings: media: nxp, imx8mq-vpu: Add support for G1 on imx8mm

The i.MX8M mini appears to have a similar G1 decoder but the
post-processing isn't present, so different compatible flag is required.
Since all the other parameters are the same with imx8mq, just add
the new compatible flag to nxp,imx8mq-vpu.yaml.

Signed-off-by: Adam Ford <[email protected]>
Reviewed-by: Ezequiel Garcia <[email protected]>

diff --git a/Documentation/devicetree/bindings/media/nxp,imx8mq-vpu.yaml b/Documentation/devicetree/bindings/media/nxp,imx8mq-vpu.yaml
index 9c28d562112b..7dc13a4b1805 100644
--- a/Documentation/devicetree/bindings/media/nxp,imx8mq-vpu.yaml
+++ b/Documentation/devicetree/bindings/media/nxp,imx8mq-vpu.yaml
@@ -5,7 +5,7 @@
$id: "http://devicetree.org/schemas/media/nxp,imx8mq-vpu.yaml#"
$schema: "http://devicetree.org/meta-schemas/core.yaml#"

-title: Hantro G1/G2 VPU codecs implemented on i.MX8MQ SoCs
+title: Hantro G1/G2 VPU codecs implemented on i.MX8M SoCs

maintainers:
- Philipp Zabel <[email protected]>
@@ -20,6 +20,7 @@ properties:
deprecated: true
- const: nxp,imx8mq-vpu-g1
- const: nxp,imx8mq-vpu-g2
+ - const: nxp,imx8mm-vpu-g1

reg:
maxItems: 1
--
2.32.0

2022-01-26 03:01:23

by Adam Ford

[permalink] [raw]
Subject: [PATCH V4 10/11] media: hantro: Add support for i.MX8MM Hantro-G1

The i.MX8MM has a Hantro G1 video decoder similar to the
imx8mq but lacks the post-processor present in the imx8mq.
Add support in the driver for it with the post-processing
removed.

Signed-off-by: Adam Ford <[email protected]>
Reviewed-by: Ezequiel Garcia <[email protected]>

diff --git a/drivers/staging/media/hantro/hantro_drv.c b/drivers/staging/media/hantro/hantro_drv.c
index f56e8b3efada..8f020ba15194 100644
--- a/drivers/staging/media/hantro/hantro_drv.c
+++ b/drivers/staging/media/hantro/hantro_drv.c
@@ -615,6 +615,7 @@ static const struct of_device_id of_hantro_match[] = {
{ .compatible = "rockchip,rk3399-vpu", .data = &rk3399_vpu_variant, },
#endif
#ifdef CONFIG_VIDEO_HANTRO_IMX8M
+ { .compatible = "nxp,imx8mm-vpu-g1", .data = &imx8mm_vpu_g1_variant, },
{ .compatible = "nxp,imx8mq-vpu", .data = &imx8mq_vpu_variant, },
{ .compatible = "nxp,imx8mq-vpu-g1", .data = &imx8mq_vpu_g1_variant },
{ .compatible = "nxp,imx8mq-vpu-g2", .data = &imx8mq_vpu_g2_variant },
diff --git a/drivers/staging/media/hantro/hantro_hw.h b/drivers/staging/media/hantro/hantro_hw.h
index f0bd2ffe290b..c00b46e06055 100644
--- a/drivers/staging/media/hantro/hantro_hw.h
+++ b/drivers/staging/media/hantro/hantro_hw.h
@@ -299,6 +299,7 @@ enum hantro_enc_fmt {
ROCKCHIP_VPU_ENC_FMT_UYVY422 = 3,
};

+extern const struct hantro_variant imx8mm_vpu_g1_variant;
extern const struct hantro_variant imx8mq_vpu_g1_variant;
extern const struct hantro_variant imx8mq_vpu_g2_variant;
extern const struct hantro_variant imx8mq_vpu_variant;
diff --git a/drivers/staging/media/hantro/imx8m_vpu_hw.c b/drivers/staging/media/hantro/imx8m_vpu_hw.c
index 849ea7122d47..9802508bade2 100644
--- a/drivers/staging/media/hantro/imx8m_vpu_hw.c
+++ b/drivers/staging/media/hantro/imx8m_vpu_hw.c
@@ -327,3 +327,15 @@ const struct hantro_variant imx8mq_vpu_g2_variant = {
.clk_names = imx8mq_g2_clk_names,
.num_clocks = ARRAY_SIZE(imx8mq_g2_clk_names),
};
+
+const struct hantro_variant imx8mm_vpu_g1_variant = {
+ .dec_fmts = imx8m_vpu_dec_fmts,
+ .num_dec_fmts = ARRAY_SIZE(imx8m_vpu_dec_fmts),
+ .codec = HANTRO_MPEG2_DECODER | HANTRO_VP8_DECODER |
+ HANTRO_H264_DECODER,
+ .codec_ops = imx8mq_vpu_g1_codec_ops,
+ .irqs = imx8mq_irqs,
+ .num_irqs = ARRAY_SIZE(imx8mq_irqs),
+ .clk_names = imx8mq_g1_clk_names,
+ .num_clocks = ARRAY_SIZE(imx8mq_g1_clk_names),
+};
--
2.32.0

2022-01-26 03:29:52

by Lucas Stach

[permalink] [raw]
Subject: Re: [PATCH V4 11/11] arm64: dts: imx8mm: Enable Hantro G1 and G2 video decoders

Am Dienstag, dem 25.01.2022 um 11:11 -0600 schrieb Adam Ford:
> There are two decoders on the i.MX8M Mini controlled by the
> vpu-blk-ctrl. The G1 supports H264 and VP8 while the
> G2 support HEVC and VP9.
>
> Signed-off-by: Adam Ford <[email protected]>
> Reviewed-by: Ezequiel Garcia <[email protected]>

Reviewed-by: Lucas Stach <[email protected]>

>
> diff --git a/arch/arm64/boot/dts/freescale/imx8mm.dtsi b/arch/arm64/boot/dts/freescale/imx8mm.dtsi
> index 0c7a72c51a31..98aec4421713 100644
> --- a/arch/arm64/boot/dts/freescale/imx8mm.dtsi
> +++ b/arch/arm64/boot/dts/freescale/imx8mm.dtsi
> @@ -1272,6 +1272,22 @@ gpu_2d: gpu@38008000 {
> power-domains = <&pgc_gpu>;
> };
>
> + vpu_g1: video-codec@38300000 {
> + compatible = "nxp,imx8mm-vpu-g1";
> + reg = <0x38300000 0x10000>;
> + interrupts = <GIC_SPI 7 IRQ_TYPE_LEVEL_HIGH>;
> + clocks = <&clk IMX8MM_CLK_VPU_G1_ROOT>;
> + power-domains = <&vpu_blk_ctrl IMX8MM_VPUBLK_PD_G1>;
> + };
> +
> + vpu_g2: video-codec@38310000 {
> + compatible = "nxp,imx8mq-vpu-g2";
> + reg = <0x38310000 0x10000>;
> + interrupts = <GIC_SPI 8 IRQ_TYPE_LEVEL_HIGH>;
> + clocks = <&clk IMX8MM_CLK_VPU_G2_ROOT>;
> + power-domains = <&vpu_blk_ctrl IMX8MM_VPUBLK_PD_G2>;
> + };
> +
> vpu_blk_ctrl: blk-ctrl@38330000 {
> compatible = "fsl,imx8mm-vpu-blk-ctrl", "syscon";
> reg = <0x38330000 0x100>;
> @@ -1282,6 +1298,12 @@ vpu_blk_ctrl: blk-ctrl@38330000 {
> <&clk IMX8MM_CLK_VPU_G2_ROOT>,
> <&clk IMX8MM_CLK_VPU_H1_ROOT>;
> clock-names = "g1", "g2", "h1";
> + assigned-clocks = <&clk IMX8MM_CLK_VPU_G1>,
> + <&clk IMX8MM_CLK_VPU_G2>;
> + assigned-clock-parents = <&clk IMX8MM_VPU_PLL_OUT>,
> + <&clk IMX8MM_VPU_PLL_OUT>;
> + assigned-clock-rates = <600000000>,
> + <600000000>;
> #power-domain-cells = <1>;
> };
>


2022-01-26 20:09:13

by Alexander Stein

[permalink] [raw]
Subject: Re: (EXT) [PATCH V4 09/11] dt-bindings: media: nxp, imx8mq-vpu: Add support for G1 on imx8mm

Am Dienstag, 25. Januar 2022, 18:11:26 CET schrieb Adam Ford:
> The i.MX8M mini appears to have a similar G1 decoder but the
> post-processing isn't present, so different compatible flag is required.
> Since all the other parameters are the same with imx8mq, just add
> the new compatible flag to nxp,imx8mq-vpu.yaml.
>
> Signed-off-by: Adam Ford <[email protected]>
> Reviewed-by: Ezequiel Garcia <[email protected]>
>
> diff --git a/Documentation/devicetree/bindings/media/nxp,imx8mq-vpu.yaml
> b/Documentation/devicetree/bindings/media/nxp,imx8mq-vpu.yaml index
> 9c28d562112b..7dc13a4b1805 100644
> --- a/Documentation/devicetree/bindings/media/nxp,imx8mq-vpu.yaml
> +++ b/Documentation/devicetree/bindings/media/nxp,imx8mq-vpu.yaml
> @@ -5,7 +5,7 @@
> $id: "http://devicetree.org/schemas/media/nxp,imx8mq-vpu.yaml#"
> $schema: "http://devicetree.org/meta-schemas/core.yaml#"
>
> -title: Hantro G1/G2 VPU codecs implemented on i.MX8MQ SoCs
> +title: Hantro G1/G2 VPU codecs implemented on i.MX8M SoCs
>
> maintainers:
> - Philipp Zabel <[email protected]>
> @@ -20,6 +20,7 @@ properties:
> deprecated: true
> - const: nxp,imx8mq-vpu-g1
> - const: nxp,imx8mq-vpu-g2
> + - const: nxp,imx8mm-vpu-g1
>
> reg:
> maxItems: 1

Is there no rule that items are sorted alphabetically?

Regards,
Alexander




2022-02-07 06:21:40

by Rob Herring (Arm)

[permalink] [raw]
Subject: Re: [PATCH V4 09/11] dt-bindings: media: nxp, imx8mq-vpu: Add support for G1 on imx8mm

On Tue, 25 Jan 2022 11:11:26 -0600, Adam Ford wrote:
> The i.MX8M mini appears to have a similar G1 decoder but the
> post-processing isn't present, so different compatible flag is required.
> Since all the other parameters are the same with imx8mq, just add
> the new compatible flag to nxp,imx8mq-vpu.yaml.
>
> Signed-off-by: Adam Ford <[email protected]>
> Reviewed-by: Ezequiel Garcia <[email protected]>
>

Acked-by: Rob Herring <[email protected]>

2022-02-09 10:29:45

by Shawn Guo

[permalink] [raw]
Subject: Re: [PATCH V4 11/11] arm64: dts: imx8mm: Enable Hantro G1 and G2 video decoders

On Tue, Jan 25, 2022 at 11:11:28AM -0600, Adam Ford wrote:
> There are two decoders on the i.MX8M Mini controlled by the
> vpu-blk-ctrl. The G1 supports H264 and VP8 while the
> G2 support HEVC and VP9.
>
> Signed-off-by: Adam Ford <[email protected]>
> Reviewed-by: Ezequiel Garcia <[email protected]>

Applied, thanks!