LinuxLists.cc - [PATCH 00/14] staging: vc04_services: bcm2835-isp support

[permalink] [raw]

Subject: [PATCH 02/14] staging: vchiq_arm: Register vcsm-cma as a platform driver

From: Dave Stevenson <[email protected]>

Following the same pattern as bcm2835-camera and bcm2835-audio,
register the vcsm-cma driver as a platform driver.

Signed-off-by: Dave Stevenson <[email protected]>
Signed-off-by: Umang Jain <[email protected]>
---
drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c | 3 +++
1 file changed, 3 insertions(+)

diff --git a/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c b/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
index dc33490ba7fb..642fdbc0d654 100644
--- a/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
+++ b/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
@@ -67,6 +67,7 @@ struct vchiq_state g_state;

static struct platform_device *bcm2835_camera;
static struct platform_device *bcm2835_audio;
+static struct platform_device *vcsm_cma;

struct vchiq_drvdata {
const unsigned int cache_line_size;
@@ -1832,6 +1833,7 @@ static int vchiq_probe(struct platform_device *pdev)
goto error_exit;
}

+ vcsm_cma = vchiq_register_child(pdev, "vcsm-cma");
bcm2835_camera = vchiq_register_child(pdev, "bcm2835-camera");
bcm2835_audio = vchiq_register_child(pdev, "bcm2835_audio");

@@ -1847,6 +1849,7 @@ static int vchiq_remove(struct platform_device *pdev)
{
platform_device_unregister(bcm2835_audio);
platform_device_unregister(bcm2835_camera);
+ platform_device_unregister(vcsm_cma);
vchiq_debugfs_deinit();
vchiq_deregister_chrdev();

--
2.37.3

2022-11-21 22:10:32

[permalink] [raw]

Subject: [PATCH 08/14] uapi: bcm2835-isp: Add bcm2835-isp uapi header file

From: Dave Stevenson <[email protected]>

This file defines the userland interface to the bcm2835-isp driver
that will follow in a separate commit.

Signed-off-by: Naushir Patuck <[email protected]>
Signed-off-by: Dave Stevenson <[email protected]>
Signed-off-by: Umang Jain <[email protected]>
---
include/uapi/linux/bcm2835-isp.h | 320 +++++++++++++++++++++++++++++
include/uapi/linux/v4l2-controls.h | 6 +
2 files changed, 326 insertions(+)
create mode 100644 include/uapi/linux/bcm2835-isp.h

diff --git a/include/uapi/linux/bcm2835-isp.h b/include/uapi/linux/bcm2835-isp.h
new file mode 100644
index 000000000000..cf8c0437f159
--- /dev/null
+++ b/include/uapi/linux/bcm2835-isp.h
@@ -0,0 +1,320 @@
+/* SPDX-License-Identifier: ((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) */
+/*
+ * bcm2835-isp.h
+ *
+ * BCM2835 ISP driver - user space header file.
+ *
+ * Copyright © 2019-2020 Raspberry Pi (Trading) Ltd.
+ *
+ * Author: Naushir Patuck ([email protected])
+ *
+ */
+
+#ifndef __BCM2835_ISP_H_
+#define __BCM2835_ISP_H_
+
+#include <linux/v4l2-controls.h>
+
+#define V4L2_CID_USER_BCM2835_ISP_CC_MATRIX \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0001)
+#define V4L2_CID_USER_BCM2835_ISP_LENS_SHADING \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0002)
+#define V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0003)
+#define V4L2_CID_USER_BCM2835_ISP_GEQ \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0004)
+#define V4L2_CID_USER_BCM2835_ISP_GAMMA \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0005)
+#define V4L2_CID_USER_BCM2835_ISP_DENOISE \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0006)
+#define V4L2_CID_USER_BCM2835_ISP_SHARPEN \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0007)
+#define V4L2_CID_USER_BCM2835_ISP_DPC \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0008)
+
+/*
+ * All structs below are directly mapped onto the equivalent structs in
+ * drivers/staging/vc04_services/vchiq-mmal/mmal-parameters.h
+ * for convenience.
+ */
+
+/**
+ * struct bcm2835_isp_rational - Rational value type.
+ *
+ * @num: Numerator.
+ * @den: Denominator.
+ */
+struct bcm2835_isp_rational {
+ __s32 num;
+ __u32 den;
+};
+
+/**
+ * struct bcm2835_isp_ccm - Colour correction matrix.
+ *
+ * @ccm: 3x3 correction matrix coefficients.
+ * @offsets: 1x3 correction offsets.
+ */
+struct bcm2835_isp_ccm {
+ struct bcm2835_isp_rational ccm[3][3];
+ __s32 offsets[3];
+};
+
+/**
+ * struct bcm2835_isp_custom_ccm - Custom CCM applied with the
+ * V4L2_CID_USER_BCM2835_ISP_CC_MATRIX ctrl.
+ *
+ * @enabled: Enable custom CCM.
+ * @ccm: Custom CCM coefficients and offsets.
+ */
+struct bcm2835_isp_custom_ccm {
+ __u32 enabled;
+ struct bcm2835_isp_ccm ccm;
+};
+
+/**
+ * enum bcm2835_isp_gain_format - format of the gains in the lens shading
+ * tables used with the
+ * V4L2_CID_USER_BCM2835_ISP_LENS_SHADING ctrl.
+ *
+ * @GAIN_FORMAT_U0P8_1: Gains are u0.8 format, starting at 1.0
+ * @GAIN_FORMAT_U1P7_0: Gains are u1.7 format, starting at 0.0
+ * @GAIN_FORMAT_U1P7_1: Gains are u1.7 format, starting at 1.0
+ * @GAIN_FORMAT_U2P6_0: Gains are u2.6 format, starting at 0.0
+ * @GAIN_FORMAT_U2P6_1: Gains are u2.6 format, starting at 1.0
+ * @GAIN_FORMAT_U3P5_0: Gains are u3.5 format, starting at 0.0
+ * @GAIN_FORMAT_U3P5_1: Gains are u3.5 format, starting at 1.0
+ * @GAIN_FORMAT_U4P10: Gains are u4.10 format, starting at 0.0
+ */
+enum bcm2835_isp_gain_format {
+ GAIN_FORMAT_U0P8_1 = 0,
+ GAIN_FORMAT_U1P7_0 = 1,
+ GAIN_FORMAT_U1P7_1 = 2,
+ GAIN_FORMAT_U2P6_0 = 3,
+ GAIN_FORMAT_U2P6_1 = 4,
+ GAIN_FORMAT_U3P5_0 = 5,
+ GAIN_FORMAT_U3P5_1 = 6,
+ GAIN_FORMAT_U4P10 = 7,
+};
+
+/**
+ * struct bcm2835_isp_lens_shading - Lens shading tables supplied with the
+ * V4L2_CID_USER_BCM2835_ISP_LENS_SHADING
+ * ctrl.
+ *
+ * @enabled: Enable lens shading.
+ * @grid_cell_size: Size of grid cells in samples (16, 32, 64, 128 or 256).
+ * @grid_width: Width of lens shading tables in grid cells.
+ * @grid_stride: Row to row distance (in grid cells) between grid cells
+ * in the same horizontal location.
+ * @grid_height: Height of lens shading tables in grid cells.
+ * @dmabuf: dmabuf file handle containing the table.
+ * @ref_transform: Reference transform - unsupported, please pass zero.
+ * @corner_sampled: Whether the gains are sampled at the corner points
+ * of the grid cells or in the cell centres.
+ * @gain_format: Format of the gains (see enum &bcm2835_isp_gain_format).
+ */
+struct bcm2835_isp_lens_shading {
+ __u32 enabled;
+ __u32 grid_cell_size;
+ __u32 grid_width;
+ __u32 grid_stride;
+ __u32 grid_height;
+ __s32 dmabuf;
+ __u32 ref_transform;
+ __u32 corner_sampled;
+ __u32 gain_format;
+};
+
+/**
+ * struct bcm2835_isp_black_level - Sensor black level set with the
+ * V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL ctrl.
+ *
+ * @enabled: Enable black level.
+ * @black_level_r: Black level for red channel.
+ * @black_level_g: Black level for green channels.
+ * @black_level_b: Black level for blue channel.
+ */
+struct bcm2835_isp_black_level {
+ __u32 enabled;
+ __u16 black_level_r;
+ __u16 black_level_g;
+ __u16 black_level_b;
+ __u8 padding[2]; /* Unused */
+};
+
+/**
+ * struct bcm2835_isp_geq - Green equalisation parameters set with the
+ * V4L2_CID_USER_BCM2835_ISP_GEQ ctrl.
+ *
+ * @enabled: Enable green equalisation.
+ * @offset: Fixed offset of the green equalisation threshold.
+ * @slope: Slope of the green equalisation threshold.
+ */
+struct bcm2835_isp_geq {
+ __u32 enabled;
+ __u32 offset;
+ struct bcm2835_isp_rational slope;
+};
+
+#define BCM2835_NUM_GAMMA_PTS 33
+
+/**
+ * struct bcm2835_isp_gamma - Gamma parameters set with the
+ * V4L2_CID_USER_BCM2835_ISP_GAMMA ctrl.
+ *
+ * @enabled: Enable gamma adjustment.
+ * @X: X values of the points defining the gamma curve.
+ * Values should be scaled to 16 bits.
+ * @Y: Y values of the points defining the gamma curve.
+ * Values should be scaled to 16 bits.
+ */
+struct bcm2835_isp_gamma {
+ __u32 enabled;
+ __u16 x[BCM2835_NUM_GAMMA_PTS];
+ __u16 y[BCM2835_NUM_GAMMA_PTS];
+};
+
+/**
+ * struct bcm2835_isp_denoise - Denoise parameters set with the
+ * V4L2_CID_USER_BCM2835_ISP_DENOISE ctrl.
+ *
+ * @enabled: Enable denoise.
+ * @constant: Fixed offset of the noise threshold.
+ * @slope: Slope of the noise threshold.
+ * @strength: Denoise strength between 0.0 (off) and 1.0 (maximum).
+ */
+struct bcm2835_isp_denoise {
+ __u32 enabled;
+ __u32 constant;
+ struct bcm2835_isp_rational slope;
+ struct bcm2835_isp_rational strength;
+};
+
+/**
+ * struct bcm2835_isp_sharpen - Sharpen parameters set with the
+ * V4L2_CID_USER_BCM2835_ISP_SHARPEN ctrl.
+ *
+ * @enabled: Enable sharpening.
+ * @threshold: Threshold at which to start sharpening pixels.
+ * @strength: Strength with which pixel sharpening increases.
+ * @limit: Limit to the amount of sharpening applied.
+ */
+struct bcm2835_isp_sharpen {
+ __u32 enabled;
+ struct bcm2835_isp_rational threshold;
+ struct bcm2835_isp_rational strength;
+ struct bcm2835_isp_rational limit;
+};
+
+/**
+ * enum bcm2835_isp_dpc_mode - defective pixel correction (DPC) strength.
+ *
+ * @DPC_MODE_OFF: No DPC.
+ * @DPC_MODE_NORMAL: Normal DPC.
+ * @DPC_MODE_STRONG: Strong DPC.
+ */
+enum bcm2835_isp_dpc_mode {
+ DPC_MODE_OFF = 0,
+ DPC_MODE_NORMAL = 1,
+ DPC_MODE_STRONG = 2,
+};
+
+/**
+ * struct bcm2835_isp_dpc - Defective pixel correction (DPC) parameters set
+ * with the V4L2_CID_USER_BCM2835_ISP_DPC ctrl.
+ *
+ * @enabled: Enable DPC.
+ * @strength: DPC strength (see enum &bcm2835_isp_dpc_mode).
+ */
+struct bcm2835_isp_dpc {
+ __u32 enabled;
+ __u32 strength;
+};
+
+/*
+ * ISP statistics structures.
+ *
+ * The bcm2835_isp_stats structure is generated at the output of the
+ * statistics node. Note that this does not directly map onto the statistics
+ * output of the ISP HW. Instead, the MMAL firmware code maps the HW statistics
+ * to the bcm2835_isp_stats structure.
+ */
+#define DEFAULT_AWB_REGIONS_X 16
+#define DEFAULT_AWB_REGIONS_Y 12
+
+#define NUM_HISTOGRAMS 2
+#define NUM_HISTOGRAM_BINS 128
+#define AWB_REGIONS (DEFAULT_AWB_REGIONS_X * DEFAULT_AWB_REGIONS_Y)
+#define FLOATING_REGIONS 16
+#define AGC_REGIONS 16
+#define FOCUS_REGIONS 12
+
+/**
+ * struct bcm2835_isp_stats_hist - Histogram statistics
+ *
+ * @r_hist: Red channel histogram.
+ * @g_hist: Combined green channel histogram.
+ * @b_hist: Blue channel histogram.
+ */
+struct bcm2835_isp_stats_hist {
+ __u32 r_hist[NUM_HISTOGRAM_BINS];
+ __u32 g_hist[NUM_HISTOGRAM_BINS];
+ __u32 b_hist[NUM_HISTOGRAM_BINS];
+};
+
+/**
+ * struct bcm2835_isp_stats_region - Region sums.
+ *
+ * @counted: The number of 2x2 bayer tiles accumulated.
+ * @notcounted: The number of 2x2 bayer tiles not accumulated.
+ * @r_sum: Total sum of counted pixels in the red channel for a region.
+ * @g_sum: Total sum of counted pixels in the green channel for a region.
+ * @b_sum: Total sum of counted pixels in the blue channel for a region.
+ */
+struct bcm2835_isp_stats_region {
+ __u32 counted;
+ __u32 notcounted;
+ __u64 r_sum;
+ __u64 g_sum;
+ __u64 b_sum;
+};
+
+/**
+ * struct bcm2835_isp_stats_focus - Focus statistics.
+ *
+ * @contrast_val: Focus measure - accumulated output of the focus filter.
+ * In the first dimension, index [0] counts pixels below a
+ * preset threshold, and index [1] counts pixels above the
+ * threshold. In the second dimension, index [0] uses the
+ * first predefined filter, and index [1] uses the second
+ * predefined filter.
+ * @contrast_val_num: The number of counted pixels in the above accumulation.
+ */
+struct bcm2835_isp_stats_focus {
+ __u64 contrast_val[2][2];
+ __u32 contrast_val_num[2][2];
+};
+
+/**
+ * struct bcm2835_isp_stats - ISP statistics.
+ *
+ * @version: Version of the bcm2835_isp_stats structure.
+ * @size: Size of the bcm2835_isp_stats structure.
+ * @hist: Histogram statistics for the entire image.
+ * @awb_stats: Statistics for the regions defined for AWB calculations.
+ * @floating_stats: Statistics for arbitrarily placed (floating) regions.
+ * @agc_stats: Statistics for the regions defined for AGC calculations.
+ * @focus_stats: Focus filter statistics for the focus regions.
+ */
+struct bcm2835_isp_stats {
+ __u32 version;
+ __u32 size;
+ struct bcm2835_isp_stats_hist hist[NUM_HISTOGRAMS];
+ struct bcm2835_isp_stats_region awb_stats[AWB_REGIONS];
+ struct bcm2835_isp_stats_region floating_stats[FLOATING_REGIONS];
+ struct bcm2835_isp_stats_region agc_stats[AGC_REGIONS];
+ struct bcm2835_isp_stats_focus focus_stats[FOCUS_REGIONS];
+};
+
+#endif /* __BCM2835_ISP_H_ */
diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h
index b5e7d082b8ad..31606d42ec58 100644
--- a/include/uapi/linux/v4l2-controls.h
+++ b/include/uapi/linux/v4l2-controls.h
@@ -217,6 +217,7 @@ enum v4l2_colorfx {
* The base for Allegro driver controls.
* We reserve 16 controls for this driver.
*/
+
#define V4L2_CID_USER_ALLEGRO_BASE (V4L2_CID_USER_BASE + 0x1170)

/*
@@ -231,6 +232,11 @@ enum v4l2_colorfx {
*/
#define V4L2_CID_USER_DW100_BASE (V4L2_CID_USER_BASE + 0x1190)

+/* The base for the bcm2835-isp driver controls.
+ * We reserve 16 controls for this driver.
+ */
+#define V4L2_CID_USER_BCM2835_ISP_BASE (V4L2_CID_USER_BASE + 0x11a0)
+
/* MPEG-class control IDs */
/* The MPEG controls are applicable to all codec controls
* and the 'MPEG' part of the define is historical */
--
2.37.3

2022-11-21 22:30:54

[permalink] [raw]

Subject: [PATCH 09/14] staging: vc04_services: bcm2835-isp: Add a more complex ISP processing component

From: Naushir Patuck <[email protected]>

Driver for the BCM2835 ISP hardware block. This driver uses the MMAL
component to program the ISP hardware through the VC firmware.

The ISP component can produce two video stream outputs, and Bayer
image statistics. This can't be encompassed in a simple V4L2
M2M device, so create a new device that registers 4 video nodes.

Signed-off-by: Naushir Patuck <[email protected]>
Signed-off-by: Umang Jain <[email protected]>
---
MAINTAINERS | 9 +
drivers/staging/vc04_services/Kconfig | 2 +
drivers/staging/vc04_services/Makefile | 1 +
.../staging/vc04_services/bcm2835-isp/Kconfig | 14 +
.../vc04_services/bcm2835-isp/Makefile | 8 +
.../bcm2835-isp/bcm2835-isp-ctrls.h | 72 +
.../bcm2835-isp/bcm2835-isp-fmts.h | 481 +++++
.../bcm2835-isp/bcm2835-v4l2-isp.c | 1708 +++++++++++++++++
.../vc04_services/vchiq-mmal/mmal-encodings.h | 4 +
.../vchiq-mmal/mmal-parameters.h | 165 +-
include/uapi/linux/bcm2835-isp.h | 27 +
11 files changed, 2490 insertions(+), 1 deletion(-)
create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Kconfig
create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Makefile
create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h
create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 6466dd5ef1fa..14e162ca0b58 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4037,6 +4037,15 @@ S: Maintained
F: drivers/media/platform/bcm2835/
F: Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml

+BROADCOM BCM2835 ISP DRIVER
+M: Raspberry Pi Kernel Maintenance <[email protected]>
+L: [email protected]
+S: Maintained
+F: Documentation/media/uapi/v4l/pixfmt-meta-bcm2835-isp-stats.rst
+F: Documentation/media/v4l-drivers/bcm2835-isp.rst
+F: drivers/staging/vc04_services/bcm2835-isp
+F: include/uapi/linux/bcm2835-isp.h
+
BROADCOM BCM47XX MIPS ARCHITECTURE
M: Hauke Mehrtens <[email protected]>
M: Rafał Miłecki <[email protected]>
diff --git a/drivers/staging/vc04_services/Kconfig b/drivers/staging/vc04_services/Kconfig
index 6c0e77d64376..e71e5a8b820a 100644
--- a/drivers/staging/vc04_services/Kconfig
+++ b/drivers/staging/vc04_services/Kconfig
@@ -44,6 +44,8 @@ source "drivers/staging/vc04_services/bcm2835-audio/Kconfig"

source "drivers/staging/vc04_services/bcm2835-camera/Kconfig"

+source "drivers/staging/vc04_services/bcm2835-isp/Kconfig"
+
source "drivers/staging/vc04_services/vchiq-mmal/Kconfig"

source "drivers/staging/vc04_services/vc-sm-cma/Kconfig"
diff --git a/drivers/staging/vc04_services/Makefile b/drivers/staging/vc04_services/Makefile
index 01089f369fb4..055cf28fc19c 100644
--- a/drivers/staging/vc04_services/Makefile
+++ b/drivers/staging/vc04_services/Makefile
@@ -15,6 +15,7 @@ obj-$(CONFIG_SND_BCM2835) += bcm2835-audio/
obj-$(CONFIG_VIDEO_BCM2835) += bcm2835-camera/
obj-$(CONFIG_BCM2835_VCHIQ_MMAL) += vchiq-mmal/
obj-$(CONFIG_BCM_VC_SM_CMA) += vc-sm-cma/
+obj-$(CONFIG_VIDEO_ISP_BCM2835) += bcm2835-isp/

ccflags-y += -I $(srctree)/$(src)/include

diff --git a/drivers/staging/vc04_services/bcm2835-isp/Kconfig b/drivers/staging/vc04_services/bcm2835-isp/Kconfig
new file mode 100644
index 000000000000..6222799ebe16
--- /dev/null
+++ b/drivers/staging/vc04_services/bcm2835-isp/Kconfig
@@ -0,0 +1,14 @@
+config VIDEO_ISP_BCM2835
+ tristate "BCM2835 ISP support"
+ depends on MEDIA_SUPPORT
+ depends on VIDEO_DEV && (ARCH_BCM2835 || COMPILE_TEST)
+ depends on MEDIA_CONTROLLER
+ select BCM2835_VCHIQ_MMAL
+ select VIDEOBUF2_DMA_CONTIG
+ help
+ This is the V4L2 driver for the Broadcom BCM2835 ISP hardware.
+ This operates over the VCHIQ interface to a service running on
+ VideoCore.
+
+ To compile this driver as a module, choose M here: the module
+ will be called bcm2835-isp.
diff --git a/drivers/staging/vc04_services/bcm2835-isp/Makefile b/drivers/staging/vc04_services/bcm2835-isp/Makefile
new file mode 100644
index 000000000000..42d3081d342f
--- /dev/null
+++ b/drivers/staging/vc04_services/bcm2835-isp/Makefile
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0
+bcm2835-isp-objs := bcm2835-v4l2-isp.o
+
+obj-$(CONFIG_VIDEO_ISP_BCM2835) += bcm2835-isp.o
+
+ccflags-y += \
+ -I$(srctree)/drivers/staging/vc04_services \
+ -D__VCCOREVER__=0x04000000
diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h
new file mode 100644
index 000000000000..172605718cdf
--- /dev/null
+++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Broadcom BCM2835 ISP driver
+ *
+ * Copyright © 2019-2020 Raspberry Pi (Trading) Ltd.
+ *
+ * Author: Naushir Patuck ([email protected])
+ *
+ */
+
+#ifndef BCM2835_ISP_CTRLS
+#define BCM2835_ISP_CTRLS
+
+#include <linux/bcm2835-isp.h>
+
+struct bcm2835_isp_custom_ctrl {
+ const char *name;
+ u32 id;
+ u32 size;
+ u32 flags;
+};
+
+static const struct bcm2835_isp_custom_ctrl custom_ctrls[] = {
+ {
+ .name = "Colour Correction Matrix",
+ .id = V4L2_CID_USER_BCM2835_ISP_CC_MATRIX,
+ .size = sizeof(struct bcm2835_isp_custom_ccm),
+ .flags = 0
+ }, {
+ .name = "Lens Shading",
+ .id = V4L2_CID_USER_BCM2835_ISP_LENS_SHADING,
+ .size = sizeof(struct bcm2835_isp_lens_shading),
+ .flags = V4L2_CTRL_FLAG_EXECUTE_ON_WRITE
+ }, {
+ .name = "Black Level",
+ .id = V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL,
+ .size = sizeof(struct bcm2835_isp_black_level),
+ .flags = 0
+ }, {
+ .name = "Green Equalisation",
+ .id = V4L2_CID_USER_BCM2835_ISP_GEQ,
+ .size = sizeof(struct bcm2835_isp_geq),
+ .flags = 0
+ }, {
+ .name = "Gamma",
+ .id = V4L2_CID_USER_BCM2835_ISP_GAMMA,
+ .size = sizeof(struct bcm2835_isp_gamma),
+ .flags = 0
+ }, {
+ .name = "Sharpen",
+ .id = V4L2_CID_USER_BCM2835_ISP_SHARPEN,
+ .size = sizeof(struct bcm2835_isp_sharpen),
+ .flags = 0
+ }, {
+ .name = "Denoise",
+ .id = V4L2_CID_USER_BCM2835_ISP_DENOISE,
+ .size = sizeof(struct bcm2835_isp_denoise),
+ .flags = 0
+ }, {
+ .name = "Colour Denoise",
+ .id = V4L2_CID_USER_BCM2835_ISP_CDN,
+ .size = sizeof(struct bcm2835_isp_cdn),
+ .flags = 0
+ }, {
+ .name = "Defective Pixel Correction",
+ .id = V4L2_CID_USER_BCM2835_ISP_DPC,
+ .size = sizeof(struct bcm2835_isp_dpc),
+ .flags = 0
+ }
+};
+
+#endif
diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
new file mode 100644
index 000000000000..65913a4ff203
--- /dev/null
+++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
@@ -0,0 +1,481 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Broadcom BCM2835 ISP driver
+ *
+ * Copyright © 2019-2020 Raspberry Pi (Trading) Ltd.
+ *
+ * Author: Naushir Patuck ([email protected])
+ *
+ */
+
+#ifndef BCM2835_ISP_FMTS
+#define BCM2835_ISP_FMTS
+
+#include <linux/videodev2.h>
+#include "vchiq-mmal/mmal-encodings.h"
+
+struct bcm2835_isp_fmt {
+ u32 fourcc;
+ int depth;
+ int bytesperline_align;
+ u32 mmal_fmt;
+ int size_multiplier_x2;
+ enum v4l2_colorspace colorspace;
+ unsigned int step_size;
+};
+
+static const struct bcm2835_isp_fmt supported_formats[] = {
+ {
+ /* YUV formats */
+ .fourcc = V4L2_PIX_FMT_YUV420,
+ .depth = 8,
+ .bytesperline_align = 64,
+ .mmal_fmt = MMAL_ENCODING_I420,
+ .size_multiplier_x2 = 3,
+ .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_YVU420,
+ .depth = 8,
+ .bytesperline_align = 64,
+ .mmal_fmt = MMAL_ENCODING_YV12,
+ .size_multiplier_x2 = 3,
+ .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_NV12,
+ .depth = 8,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_NV12,
+ .size_multiplier_x2 = 3,
+ .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_NV21,
+ .depth = 8,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_NV21,
+ .size_multiplier_x2 = 3,
+ .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_YUYV,
+ .depth = 16,
+ .bytesperline_align = 64,
+ .mmal_fmt = MMAL_ENCODING_YUYV,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_UYVY,
+ .depth = 16,
+ .bytesperline_align = 64,
+ .mmal_fmt = MMAL_ENCODING_UYVY,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_YVYU,
+ .depth = 16,
+ .bytesperline_align = 64,
+ .mmal_fmt = MMAL_ENCODING_YVYU,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_VYUY,
+ .depth = 16,
+ .bytesperline_align = 64,
+ .mmal_fmt = MMAL_ENCODING_VYUY,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .step_size = 2,
+ }, {
+ /* RGB formats */
+ .fourcc = V4L2_PIX_FMT_RGB24,
+ .depth = 24,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_RGB24,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SRGB,
+ .step_size = 1,
+ }, {
+ .fourcc = V4L2_PIX_FMT_RGB565,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_RGB16,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SRGB,
+ .step_size = 1,
+ }, {
+ .fourcc = V4L2_PIX_FMT_BGR24,
+ .depth = 24,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BGR24,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SRGB,
+ .step_size = 1,
+ }, {
+ .fourcc = V4L2_PIX_FMT_XBGR32,
+ .depth = 32,
+ .bytesperline_align = 64,
+ .mmal_fmt = MMAL_ENCODING_BGRA,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SRGB,
+ .step_size = 1,
+ }, {
+ .fourcc = V4L2_PIX_FMT_RGBX32,
+ .depth = 32,
+ .bytesperline_align = 64,
+ .mmal_fmt = MMAL_ENCODING_RGBA,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SRGB,
+ .step_size = 1,
+ }, {
+ /* Bayer formats */
+ /* 8 bit */
+ .fourcc = V4L2_PIX_FMT_SRGGB8,
+ .depth = 8,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB8,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR8,
+ .depth = 8,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR8,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG8,
+ .depth = 8,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG8,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG8,
+ .depth = 8,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG8,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 10 bit */
+ .fourcc = V4L2_PIX_FMT_SRGGB10P,
+ .depth = 10,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB10P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR10P,
+ .depth = 10,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR10P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG10P,
+ .depth = 10,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG10P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG10P,
+ .depth = 10,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG10P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 12 bit */
+ .fourcc = V4L2_PIX_FMT_SRGGB12P,
+ .depth = 12,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB12P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR12P,
+ .depth = 12,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR12P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG12P,
+ .depth = 12,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG12P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG12P,
+ .depth = 12,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG12P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 14 bit */
+ .fourcc = V4L2_PIX_FMT_SRGGB14P,
+ .depth = 14,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB14P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR14P,
+ .depth = 14,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR14P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG14P,
+ .depth = 14,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG14P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG14P,
+ .depth = 14,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG14P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 16 bit */
+ .fourcc = V4L2_PIX_FMT_SRGGB16,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB16,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR16,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR16,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG16,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG16,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG16,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG16,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* Bayer formats unpacked to 16bpp */
+ /* 10 bit */
+ .fourcc = V4L2_PIX_FMT_SRGGB10,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB10,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR10,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR10,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG10,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG10,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG10,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG10,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 12 bit */
+ .fourcc = V4L2_PIX_FMT_SRGGB12,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB12,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR12,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR12,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG12,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG12,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG12,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG12,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 14 bit */
+ .fourcc = V4L2_PIX_FMT_SRGGB14,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB14,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR14,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR14,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG14,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG14,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG14,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG14,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* Monochrome MIPI formats */
+ /* 8 bit */
+ .fourcc = V4L2_PIX_FMT_GREY,
+ .depth = 8,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_GREY,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 10 bit */
+ .fourcc = V4L2_PIX_FMT_Y10P,
+ .depth = 10,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_Y10P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 12 bit */
+ .fourcc = V4L2_PIX_FMT_Y12P,
+ .depth = 12,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_Y12P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 14 bit */
+ .fourcc = V4L2_PIX_FMT_Y14P,
+ .depth = 14,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_Y14P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 16 bit */
+ .fourcc = V4L2_PIX_FMT_Y16,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_Y16,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 10 bit as 16bpp */
+ .fourcc = V4L2_PIX_FMT_Y10,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_Y10,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 12 bit as 16bpp */
+ .fourcc = V4L2_PIX_FMT_Y12,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_Y12,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 14 bit as 16bpp */
+ .fourcc = V4L2_PIX_FMT_Y14,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_Y14,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_META_FMT_BCM2835_ISP_STATS,
+ .depth = 8,
+ .mmal_fmt = MMAL_ENCODING_BRCM_STATS,
+ /* The rest are not valid fields for stats. */
+ }
+};
+
+#endif
diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
new file mode 100644
index 000000000000..eb5f076fc6dc
--- /dev/null
+++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
@@ -0,0 +1,1708 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Broadcom BCM2835 ISP driver
+ *
+ * Copyright © 2019-2020 Raspberry Pi (Trading) Ltd.
+ *
+ * Author: Naushir Patuck ([email protected])
+ *
+ */
+
+#include <linux/module.h>
+#include <linux/platform_device.h>
+
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-event.h>
+#include <media/v4l2-ioctl.h>
+#include <media/videobuf2-dma-contig.h>
+
+#include "vchiq-mmal/mmal-msg.h"
+#include "vchiq-mmal/mmal-parameters.h"
+#include "vchiq-mmal/mmal-vchiq.h"
+
+#include "vc-sm-cma/vc_sm_knl.h"
+
+#include "bcm2835-isp-ctrls.h"
+#include "bcm2835-isp-fmts.h"
+
+MODULE_IMPORT_NS(DMA_BUF);
+
+static unsigned int debug;
+module_param(debug, uint, 0644);
+MODULE_PARM_DESC(debug, "activates debug info");
+
+static unsigned int video_nr = 13;
+module_param(video_nr, uint, 0644);
+MODULE_PARM_DESC(video_nr, "base video device number");
+
+#define BCM2835_ISP_NAME "bcm2835-isp"
+#define BCM2835_ISP_ENTITY_NAME_LEN 32
+
+#define BCM2835_ISP_NUM_OUTPUTS 1
+#define BCM2835_ISP_NUM_CAPTURES 2
+#define BCM2835_ISP_NUM_METADATA 1
+
+#define BCM2835_ISP_NUM_NODES \
+ (BCM2835_ISP_NUM_OUTPUTS + BCM2835_ISP_NUM_CAPTURES + \
+ BCM2835_ISP_NUM_METADATA)
+
+/* Default frame dimension of 1280 pixels. */
+#define DEFAULT_DIM 1280U
+/*
+ * Maximum frame dimension of 16384 pixels. Even though the ISP runs in tiles,
+ * have a sensible limit so that we do not create an excessive number of tiles
+ * to process.
+ */
+#define MAX_DIM 16384U
+/*
+ * Minimum frame dimension of 64 pixels. Anything lower, and the tiling
+ * algorithm may not be able to cope when applying filter context.
+ */
+#define MIN_DIM 64U
+
+/* Timeout for stop_streaming to allow all buffers to return */
+#define COMPLETE_TIMEOUT (2 * HZ)
+
+/* Per-queue, driver-specific private data */
+struct bcm2835_isp_q_data {
+ /*
+ * These parameters should be treated as gospel, with everything else
+ * being determined from them.
+ */
+ unsigned int bytesperline;
+ unsigned int width;
+ unsigned int height;
+ unsigned int sizeimage;
+ const struct bcm2835_isp_fmt *fmt;
+};
+
+/*
+ * Structure to describe a single node /dev/video<N> which represents a single
+ * input or output queue to the ISP device.
+ */
+struct bcm2835_isp_node {
+ int vfl_dir;
+ unsigned int id;
+ const char *name;
+ struct vchiq_mmal_port *port;
+ struct video_device vfd;
+ struct media_pad pad;
+ struct media_intf_devnode *intf_devnode;
+ struct media_link *intf_link;
+ struct mutex lock; /* top level device node lock */
+ struct mutex queue_lock;
+
+ struct vb2_queue queue;
+ unsigned int sequence;
+
+ /* The list of formats supported on the node. */
+ struct bcm2835_isp_fmt const **supported_fmts;
+ unsigned int num_supported_fmts;
+
+ struct bcm2835_isp_q_data q_data;
+
+ /* Parent device structure */
+ struct bcm2835_isp_dev *dev;
+
+ bool registered;
+ bool media_node_registered;
+};
+
+/*
+ * Structure representing the entire ISP device, comprising several input and
+ * output nodes /dev/video<N>.
+ */
+struct bcm2835_isp_dev {
+ struct v4l2_device v4l2_dev;
+ struct device *dev;
+ struct v4l2_ctrl_handler ctrl_handler;
+ struct media_device mdev;
+ struct media_entity entity;
+ bool media_device_registered;
+ bool media_entity_registered;
+ struct vchiq_mmal_instance *mmal_instance;
+ struct vchiq_mmal_component *component;
+ struct completion frame_cmplt;
+
+ struct bcm2835_isp_node node[BCM2835_ISP_NUM_NODES];
+ struct media_pad pad[BCM2835_ISP_NUM_NODES];
+ atomic_t num_streaming;
+
+ /* Image pipeline controls. */
+ int r_gain;
+ int b_gain;
+};
+
+struct bcm2835_isp_buffer {
+ struct vb2_v4l2_buffer vb;
+ struct mmal_buffer mmal;
+};
+
+static
+inline struct bcm2835_isp_dev *node_get_dev(struct bcm2835_isp_node *node)
+{
+ return node->dev;
+}
+
+static inline bool node_is_output(struct bcm2835_isp_node *node)
+{
+ return node->queue.type == V4L2_BUF_TYPE_VIDEO_OUTPUT;
+}
+
+static inline bool node_is_capture(struct bcm2835_isp_node *node)
+{
+ return node->queue.type == V4L2_BUF_TYPE_VIDEO_CAPTURE;
+}
+
+static inline bool node_is_stats(struct bcm2835_isp_node *node)
+{
+ return node->queue.type == V4L2_BUF_TYPE_META_CAPTURE;
+}
+
+static inline enum v4l2_buf_type index_to_queue_type(int index)
+{
+ if (index < BCM2835_ISP_NUM_OUTPUTS)
+ return V4L2_BUF_TYPE_VIDEO_OUTPUT;
+ else if (index < BCM2835_ISP_NUM_OUTPUTS + BCM2835_ISP_NUM_CAPTURES)
+ return V4L2_BUF_TYPE_VIDEO_CAPTURE;
+ else
+ return V4L2_BUF_TYPE_META_CAPTURE;
+}
+
+static int set_isp_param(struct bcm2835_isp_node *node, u32 parameter,
+ void *value, u32 value_size)
+{
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+
+ return vchiq_mmal_port_parameter_set(dev->mmal_instance, node->port,
+ parameter, value, value_size);
+}
+
+static int set_wb_gains(struct bcm2835_isp_node *node)
+{
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ struct mmal_parameter_awbgains gains = {
+ .r_gain = { dev->r_gain, 1000 },
+ .b_gain = { dev->b_gain, 1000 }
+ };
+
+ return set_isp_param(node, MMAL_PARAMETER_CUSTOM_AWB_GAINS,
+ &gains, sizeof(gains));
+}
+
+static int set_digital_gain(struct bcm2835_isp_node *node, uint32_t gain)
+{
+ struct s32_fract digital_gain = {
+ .numerator = gain,
+ .denominator = 1000
+ };
+
+ return set_isp_param(node, MMAL_PARAMETER_DIGITAL_GAIN,
+ &digital_gain, sizeof(digital_gain));
+}
+
+static const struct bcm2835_isp_fmt *get_fmt(u32 mmal_fmt)
+{
+ unsigned int i;
+
+ for (i = 0; i < ARRAY_SIZE(supported_formats); i++) {
+ if (supported_formats[i].mmal_fmt == mmal_fmt)
+ return &supported_formats[i];
+ }
+ return NULL;
+}
+
+static const
+struct bcm2835_isp_fmt *find_format_by_fourcc(unsigned int fourcc,
+ struct bcm2835_isp_node *node)
+{
+ const struct bcm2835_isp_fmt *fmt;
+ unsigned int i;
+
+ for (i = 0; i < node->num_supported_fmts; i++) {
+ fmt = node->supported_fmts[i];
+ if (fmt->fourcc == fourcc)
+ return fmt;
+ }
+
+ return NULL;
+}
+
+static const
+struct bcm2835_isp_fmt *find_format(struct v4l2_format *f,
+ struct bcm2835_isp_node *node)
+{
+ return find_format_by_fourcc(node_is_stats(node) ?
+ f->fmt.meta.dataformat :
+ f->fmt.pix.pixelformat,
+ node);
+}
+
+/* vb2_to_mmal_buffer() - converts vb2 buffer header to MMAL
+ *
+ * Copies all the required fields from a VB2 buffer to the MMAL buffer header,
+ * ready for sending to the VPU.
+ */
+static void vb2_to_mmal_buffer(struct mmal_buffer *buf,
+ struct vb2_v4l2_buffer *vb2)
+{
+ u64 pts;
+
+ buf->mmal_flags = 0;
+ if (vb2->flags & V4L2_BUF_FLAG_KEYFRAME)
+ buf->mmal_flags |= MMAL_BUFFER_HEADER_FLAG_KEYFRAME;
+
+ /* Data must be framed correctly as one frame per buffer. */
+ buf->mmal_flags |= MMAL_BUFFER_HEADER_FLAG_FRAME_END;
+
+ buf->length = vb2->vb2_buf.planes[0].bytesused;
+ /*
+ * Minor ambiguity in the V4L2 spec as to whether passing in a 0 length
+ * buffer, or one with V4L2_BUF_FLAG_LAST set denotes end of stream.
+ * Handle either.
+ */
+ if (!buf->length || vb2->flags & V4L2_BUF_FLAG_LAST)
+ buf->mmal_flags |= MMAL_BUFFER_HEADER_FLAG_EOS;
+
+ /* vb2 timestamps in nsecs, mmal in usecs */
+ pts = vb2->vb2_buf.timestamp;
+ do_div(pts, 1000);
+ buf->pts = pts;
+ buf->dts = MMAL_TIME_UNKNOWN;
+}
+
+static void mmal_buffer_cb(struct vchiq_mmal_instance *instance,
+ struct vchiq_mmal_port *port, int status,
+ struct mmal_buffer *mmal_buf)
+{
+ struct bcm2835_isp_buffer *q_buf;
+ struct bcm2835_isp_node *node = port->cb_ctx;
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ struct vb2_v4l2_buffer *vb2;
+
+ q_buf = container_of(mmal_buf, struct bcm2835_isp_buffer, mmal);
+ vb2 = &q_buf->vb;
+ v4l2_dbg(2, debug, &dev->v4l2_dev,
+ "%s: port:%s[%d], status:%d, buf:%p, dmabuf:%p, length:%lu, flags %u, pts %lld\n",
+ __func__, node_is_output(node) ? "input" : "output", node->id,
+ status, mmal_buf, mmal_buf->dma_buf, mmal_buf->length,
+ mmal_buf->mmal_flags, mmal_buf->pts);
+
+ if (mmal_buf->cmd)
+ v4l2_err(&dev->v4l2_dev,
+ "%s: Unexpected event on output callback - %08x\n",
+ __func__, mmal_buf->cmd);
+
+ if (status) {
+ /* error in transfer */
+ if (vb2) {
+ /* there was a buffer with the error so return it */
+ vb2_buffer_done(&vb2->vb2_buf, VB2_BUF_STATE_ERROR);
+ }
+ return;
+ }
+
+ /* vb2 timestamps in nsecs, mmal in usecs */
+ vb2->vb2_buf.timestamp = mmal_buf->pts * 1000;
+ vb2->sequence = node->sequence++;
+ vb2_set_plane_payload(&vb2->vb2_buf, 0, mmal_buf->length);
+ vb2_buffer_done(&vb2->vb2_buf, VB2_BUF_STATE_DONE);
+
+ if (!port->enabled)
+ complete(&dev->frame_cmplt);
+}
+
+static void setup_mmal_port_format(struct bcm2835_isp_node *node,
+ struct vchiq_mmal_port *port)
+{
+ struct bcm2835_isp_q_data *q_data = &node->q_data;
+
+ port->format.encoding = q_data->fmt->mmal_fmt;
+ /* Raw image format - set width/height */
+ port->es.video.width = (q_data->bytesperline << 3) / q_data->fmt->depth;
+ port->es.video.height = q_data->height;
+ port->es.video.crop.width = q_data->width;
+ port->es.video.crop.height = q_data->height;
+ port->es.video.crop.x = 0;
+ port->es.video.crop.y = 0;
+};
+
+static int setup_mmal_port(struct bcm2835_isp_node *node)
+{
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ unsigned int enable = 1;
+ int ret;
+
+ v4l2_dbg(2, debug, &dev->v4l2_dev, "%s: setup %s[%d]\n", __func__,
+ node->name, node->id);
+
+ vchiq_mmal_port_parameter_set(dev->mmal_instance, node->port,
+ MMAL_PARAMETER_ZERO_COPY, &enable,
+ sizeof(enable));
+ setup_mmal_port_format(node, node->port);
+ ret = vchiq_mmal_port_set_format(dev->mmal_instance, node->port);
+ if (ret < 0) {
+ v4l2_dbg(1, debug, &dev->v4l2_dev,
+ "%s: vchiq_mmal_port_set_format failed\n",
+ __func__);
+ return ret;
+ }
+
+ if (node->q_data.sizeimage < node->port->minimum_buffer.size) {
+ v4l2_err(&dev->v4l2_dev,
+ "buffer size mismatch sizeimage %u < min size %u\n",
+ node->q_data.sizeimage,
+ node->port->minimum_buffer.size);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int bcm2835_isp_mmal_buf_cleanup(struct mmal_buffer *mmal_buf)
+{
+ mmal_vchi_buffer_cleanup(mmal_buf);
+
+ if (mmal_buf->dma_buf) {
+ dma_buf_put(mmal_buf->dma_buf);
+ mmal_buf->dma_buf = NULL;
+ }
+
+ return 0;
+}
+
+static int bcm2835_isp_node_queue_setup(struct vb2_queue *q,
+ unsigned int *nbuffers,
+ unsigned int *nplanes,
+ unsigned int sizes[],
+ struct device *alloc_devs[])
+{
+ struct bcm2835_isp_node *node = vb2_get_drv_priv(q);
+ unsigned int size;
+
+ if (setup_mmal_port(node))
+ return -EINVAL;
+
+ size = node->q_data.sizeimage;
+ if (size == 0) {
+ v4l2_info(&node_get_dev(node)->v4l2_dev,
+ "%s: Image size unset in queue_setup for node %s[%d]\n",
+ __func__, node->name, node->id);
+ return -EINVAL;
+ }
+
+ if (*nplanes)
+ return sizes[0] < size ? -EINVAL : 0;
+
+ *nplanes = 1;
+ sizes[0] = size;
+
+ node->port->current_buffer.size = size;
+
+ if (*nbuffers < node->port->minimum_buffer.num)
+ *nbuffers = node->port->minimum_buffer.num;
+
+ node->port->current_buffer.num = *nbuffers;
+
+ v4l2_dbg(2, debug, &node_get_dev(node)->v4l2_dev,
+ "%s: Image size %u, nbuffers %u for node %s[%d]\n",
+ __func__, sizes[0], *nbuffers, node->name, node->id);
+ return 0;
+}
+
+static int bcm2835_isp_buf_init(struct vb2_buffer *vb)
+{
+ struct bcm2835_isp_node *node = vb2_get_drv_priv(vb->vb2_queue);
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ struct vb2_v4l2_buffer *vb2 = to_vb2_v4l2_buffer(vb);
+ struct bcm2835_isp_buffer *buf =
+ container_of(vb2, struct bcm2835_isp_buffer, vb);
+
+ v4l2_dbg(3, debug, &dev->v4l2_dev, "%s: vb %p\n", __func__, vb);
+
+ buf->mmal.buffer = vb2_plane_vaddr(&buf->vb.vb2_buf, 0);
+ buf->mmal.buffer_size = vb2_plane_size(&buf->vb.vb2_buf, 0);
+ mmal_vchi_buffer_init(dev->mmal_instance, &buf->mmal);
+ return 0;
+}
+
+static int bcm2835_isp_buf_prepare(struct vb2_buffer *vb)
+{
+ struct bcm2835_isp_node *node = vb2_get_drv_priv(vb->vb2_queue);
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ struct vb2_v4l2_buffer *vb2 = to_vb2_v4l2_buffer(vb);
+ struct bcm2835_isp_buffer *buf =
+ container_of(vb2, struct bcm2835_isp_buffer, vb);
+ struct dma_buf *dma_buf;
+ int ret;
+
+ v4l2_dbg(3, debug, &dev->v4l2_dev, "%s: type: %d ptr %p\n",
+ __func__, vb->vb2_queue->type, vb);
+
+ if (V4L2_TYPE_IS_OUTPUT(vb->vb2_queue->type)) {
+ if (vb2->field == V4L2_FIELD_ANY)
+ vb2->field = V4L2_FIELD_NONE;
+ if (vb2->field != V4L2_FIELD_NONE) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s field isn't supported\n", __func__);
+ return -EINVAL;
+ }
+ }
+
+ if (vb2_plane_size(vb, 0) < node->q_data.sizeimage) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s data will not fit into plane (%lu < %lu)\n",
+ __func__, vb2_plane_size(vb, 0),
+ (long)node->q_data.sizeimage);
+ return -EINVAL;
+ }
+
+ if (!V4L2_TYPE_IS_OUTPUT(vb->vb2_queue->type))
+ vb2_set_plane_payload(vb, 0, node->q_data.sizeimage);
+
+ switch (vb->memory) {
+ case VB2_MEMORY_DMABUF:
+ dma_buf = dma_buf_get(vb->planes[0].m.fd);
+
+ if (dma_buf != buf->mmal.dma_buf) {
+ /*
+ * dmabuf either hasn't already been mapped, or it has
+ * changed.
+ */
+ if (buf->mmal.dma_buf) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s Buffer changed - why did the core not call cleanup?\n",
+ __func__);
+ bcm2835_isp_mmal_buf_cleanup(&buf->mmal);
+ }
+
+ buf->mmal.dma_buf = dma_buf;
+ } else {
+ /*
+ * Already have a reference to the buffer, so release it
+ * here.
+ */
+ dma_buf_put(dma_buf);
+ }
+ ret = 0;
+ break;
+ case VB2_MEMORY_MMAP:
+ /*
+ * We want to do this at init, but vb2_core_expbuf checks that
+ * the index < q->num_buffers, and q->num_buffers only gets
+ * updated once all the buffers are allocated.
+ */
+ if (!buf->mmal.dma_buf) {
+ buf->mmal.dma_buf = vb2_core_expbuf_dmabuf(vb->vb2_queue,
+ vb->vb2_queue->type,
+ vb->index, 0, O_CLOEXEC);
+ v4l2_dbg(3, debug, &dev->v4l2_dev,
+ "%s: exporting ptr %p to dmabuf %p\n",
+ __func__, vb, buf->mmal.dma_buf);
+ if (IS_ERR(buf->mmal.dma_buf)) {
+ ret = PTR_ERR(buf->mmal.dma_buf);
+ v4l2_err(&dev->v4l2_dev,
+ "%s: Failed to expbuf idx %d, ret %d\n",
+ __func__, vb->index, ret);
+ }
+ } else {
+ ret = 0;
+ }
+ break;
+ default:
+ ret = -EINVAL;
+ break;
+ }
+
+ return ret;
+}
+
+static void bcm2835_isp_node_buffer_queue(struct vb2_buffer *buf)
+{
+ struct bcm2835_isp_node *node = vb2_get_drv_priv(buf->vb2_queue);
+ struct vb2_v4l2_buffer *vbuf =
+ container_of(buf, struct vb2_v4l2_buffer, vb2_buf);
+ struct bcm2835_isp_buffer *buffer =
+ container_of(vbuf, struct bcm2835_isp_buffer, vb);
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+
+ v4l2_dbg(3, debug, &dev->v4l2_dev, "%s: node %s[%d], buffer %p\n",
+ __func__, node->name, node->id, buffer);
+
+ vb2_to_mmal_buffer(&buffer->mmal, &buffer->vb);
+ v4l2_dbg(3, debug, &dev->v4l2_dev,
+ "%s: node %s[%d] - submitting mmal dmabuf %p\n", __func__,
+ node->name, node->id, buffer->mmal.dma_buf);
+ vchiq_mmal_submit_buffer(dev->mmal_instance, node->port, &buffer->mmal);
+}
+
+static void bcm2835_isp_buffer_cleanup(struct vb2_buffer *vb)
+{
+ struct vb2_v4l2_buffer *vb2 = to_vb2_v4l2_buffer(vb);
+ struct bcm2835_isp_buffer *buffer =
+ container_of(vb2, struct bcm2835_isp_buffer, vb);
+
+ bcm2835_isp_mmal_buf_cleanup(&buffer->mmal);
+}
+
+static int bcm2835_isp_node_start_streaming(struct vb2_queue *q,
+ unsigned int count)
+{
+ struct bcm2835_isp_node *node = vb2_get_drv_priv(q);
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ int ret;
+
+ v4l2_dbg(1, debug, &dev->v4l2_dev, "%s: node %s[%d] (count %u)\n",
+ __func__, node->name, node->id, count);
+
+ ret = vchiq_mmal_component_enable(dev->mmal_instance, dev->component);
+ if (ret) {
+ v4l2_err(&dev->v4l2_dev, "%s: Failed enabling component, ret %d\n",
+ __func__, ret);
+ return -EIO;
+ }
+
+ node->sequence = 0;
+ node->port->cb_ctx = node;
+ ret = vchiq_mmal_port_enable(dev->mmal_instance, node->port,
+ mmal_buffer_cb);
+ if (!ret)
+ atomic_inc(&dev->num_streaming);
+ else
+ v4l2_err(&dev->v4l2_dev,
+ "%s: Failed enabling port, ret %d\n", __func__, ret);
+
+ return ret;
+}
+
+static void bcm2835_isp_node_stop_streaming(struct vb2_queue *q)
+{
+ struct bcm2835_isp_node *node = vb2_get_drv_priv(q);
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ int ret;
+
+ v4l2_dbg(1, debug, &dev->v4l2_dev, "%s: node %s[%d], mmal port %p\n",
+ __func__, node->name, node->id, node->port);
+
+ init_completion(&dev->frame_cmplt);
+
+ /* Disable MMAL port - this will flush buffers back */
+ ret = vchiq_mmal_port_disable(dev->mmal_instance, node->port);
+ if (ret)
+ v4l2_err(&dev->v4l2_dev,
+ "%s: Failed disabling %s port, ret %d\n", __func__,
+ node_is_output(node) ? "i/p" : "o/p",
+ ret);
+
+ while (atomic_read(&node->port->buffers_with_vpu)) {
+ v4l2_dbg(1, debug, &dev->v4l2_dev,
+ "%s: Waiting for buffers to be returned - %d outstanding\n",
+ __func__, atomic_read(&node->port->buffers_with_vpu));
+ ret = wait_for_completion_timeout(&dev->frame_cmplt,
+ COMPLETE_TIMEOUT);
+ if (ret <= 0) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s: Timeout waiting for buffers to be returned - %d outstanding\n",
+ __func__,
+ atomic_read(&node->port->buffers_with_vpu));
+ break;
+ }
+ }
+
+ atomic_dec(&dev->num_streaming);
+ /* If all ports disabled, then disable the component */
+ if (atomic_read(&dev->num_streaming) == 0) {
+ struct bcm2835_isp_lens_shading ls;
+ /*
+ * The ISP component on the firmware has a reference to the
+ * dmabuf handle for the lens shading table. Pass a null handle
+ * to remove that reference now.
+ */
+ memset(&ls, 0, sizeof(ls));
+ /* Must set a valid grid size for the FW */
+ ls.grid_cell_size = 16;
+ set_isp_param(&dev->node[0],
+ MMAL_PARAMETER_LENS_SHADING_OVERRIDE,
+ &ls, sizeof(ls));
+
+ ret = vchiq_mmal_component_disable(dev->mmal_instance,
+ dev->component);
+ if (ret) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s: Failed disabling component, ret %d\n",
+ __func__, ret);
+ }
+ }
+
+ /*
+ * Simply wait for any vb2 buffers to finish. We could take steps to
+ * make them complete more quickly if we care, or even return them
+ * ourselves.
+ */
+ vb2_wait_for_all_buffers(&node->queue);
+}
+
+static const struct vb2_ops bcm2835_isp_node_queue_ops = {
+ .queue_setup = bcm2835_isp_node_queue_setup,
+ .buf_init = bcm2835_isp_buf_init,
+ .buf_prepare = bcm2835_isp_buf_prepare,
+ .buf_queue = bcm2835_isp_node_buffer_queue,
+ .buf_cleanup = bcm2835_isp_buffer_cleanup,
+ .start_streaming = bcm2835_isp_node_start_streaming,
+ .stop_streaming = bcm2835_isp_node_stop_streaming,
+};
+
+static const
+struct bcm2835_isp_fmt *get_default_format(struct bcm2835_isp_node *node)
+{
+ return node->supported_fmts[0];
+}
+
+static inline unsigned int get_bytesperline(int width,
+ const struct bcm2835_isp_fmt *fmt)
+{
+ /* GPU aligns 24bpp images to a multiple of 32 pixels (not bytes). */
+ if (fmt->depth == 24)
+ return ALIGN(width, 32) * 3;
+ else
+ return ALIGN((width * fmt->depth) >> 3, fmt->bytesperline_align);
+}
+
+static inline unsigned int get_sizeimage(int bpl, int width, int height,
+ const struct bcm2835_isp_fmt *fmt)
+{
+ return (bpl * height * fmt->size_multiplier_x2) >> 1;
+}
+
+static int bcm2835_isp_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+ struct bcm2835_isp_dev *dev =
+ container_of(ctrl->handler, struct bcm2835_isp_dev, ctrl_handler);
+ struct bcm2835_isp_node *node = &dev->node[0];
+ int ret = 0;
+
+ /*
+ * The ISP firmware driver will ensure these settings are applied on
+ * a frame boundary, so we are safe to write them as they come in.
+ *
+ * Note that the bcm2835_isp_* param structures are identical to the
+ * mmal-parameters.h definitions. This avoids the need for unnecessary
+ * field-by-field copying between structures.
+ */
+ switch (ctrl->id) {
+ case V4L2_CID_RED_BALANCE:
+ dev->r_gain = ctrl->val;
+ ret = set_wb_gains(node);
+ break;
+ case V4L2_CID_BLUE_BALANCE:
+ dev->b_gain = ctrl->val;
+ ret = set_wb_gains(node);
+ break;
+ case V4L2_CID_DIGITAL_GAIN:
+ ret = set_digital_gain(node, ctrl->val);
+ break;
+ case V4L2_CID_USER_BCM2835_ISP_CC_MATRIX:
+ ret = set_isp_param(node, MMAL_PARAMETER_CUSTOM_CCM,
+ ctrl->p_new.p_u8,
+ sizeof(struct bcm2835_isp_custom_ccm));
+ break;
+ case V4L2_CID_USER_BCM2835_ISP_LENS_SHADING:
+ {
+ struct bcm2835_isp_lens_shading *v4l2_ls;
+ struct mmal_parameter_lens_shading_v2 ls;
+ struct dma_buf *dmabuf;
+ void *vcsm_handle;
+
+ v4l2_ls = (struct bcm2835_isp_lens_shading *)ctrl->p_new.p_u8;
+ /*
+ * struct bcm2835_isp_lens_shading and struct
+ * mmal_parameter_lens_shading_v2 match so that we can do a
+ * simple memcpy here.
+ * Only the dmabuf to the actual table needs any manipulation.
+ */
+ memcpy(&ls, v4l2_ls, sizeof(ls));
+
+ dmabuf = dma_buf_get(v4l2_ls->dmabuf);
+ if (IS_ERR_OR_NULL(dmabuf))
+ return -EINVAL;
+
+ ret = vc_sm_cma_import_dmabuf(dmabuf, &vcsm_handle);
+ if (ret) {
+ dma_buf_put(dmabuf);
+ return -EINVAL;
+ }
+
+ ls.mem_handle_table = vc_sm_cma_int_handle(vcsm_handle);
+ if (ls.mem_handle_table)
+ /* The VPU will take a reference on the vcsm handle,
+ * which in turn will retain a reference on the dmabuf.
+ * This code can therefore safely release all
+ * references to the buffer.
+ */
+ ret = set_isp_param(node,
+ MMAL_PARAMETER_LENS_SHADING_OVERRIDE,
+ &ls,
+ sizeof(ls));
+ else
+ ret = -EINVAL;
+
+ vc_sm_cma_free(vcsm_handle);
+ dma_buf_put(dmabuf);
+ break;
+ }
+ case V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL:
+ ret = set_isp_param(node, MMAL_PARAMETER_BLACK_LEVEL,
+ ctrl->p_new.p_u8,
+ sizeof(struct bcm2835_isp_black_level));
+ break;
+ case V4L2_CID_USER_BCM2835_ISP_GEQ:
+ ret = set_isp_param(node, MMAL_PARAMETER_GEQ,
+ ctrl->p_new.p_u8,
+ sizeof(struct bcm2835_isp_geq));
+ break;
+ case V4L2_CID_USER_BCM2835_ISP_GAMMA:
+ ret = set_isp_param(node, MMAL_PARAMETER_GAMMA,
+ ctrl->p_new.p_u8,
+ sizeof(struct bcm2835_isp_gamma));
+ break;
+ case V4L2_CID_USER_BCM2835_ISP_DENOISE:
+ ret = set_isp_param(node, MMAL_PARAMETER_DENOISE,
+ ctrl->p_new.p_u8,
+ sizeof(struct bcm2835_isp_denoise));
+ break;
+ case V4L2_CID_USER_BCM2835_ISP_CDN:
+ ret = set_isp_param(node, MMAL_PARAMETER_CDN,
+ ctrl->p_new.p_u8,
+ sizeof(struct bcm2835_isp_cdn));
+ break;
+ case V4L2_CID_USER_BCM2835_ISP_SHARPEN:
+ ret = set_isp_param(node, MMAL_PARAMETER_SHARPEN,
+ ctrl->p_new.p_u8,
+ sizeof(struct bcm2835_isp_sharpen));
+ break;
+ case V4L2_CID_USER_BCM2835_ISP_DPC:
+ ret = set_isp_param(node, MMAL_PARAMETER_DPC,
+ ctrl->p_new.p_u8,
+ sizeof(struct bcm2835_isp_dpc));
+ break;
+ default:
+ v4l2_info(&dev->v4l2_dev, "Unrecognised control\n");
+ ret = -EINVAL;
+ }
+
+ if (ret) {
+ v4l2_err(&dev->v4l2_dev, "%s: Failed setting ctrl \"%s\" (%08x), err %d\n",
+ __func__, ctrl->name, ctrl->id, ret);
+ ret = -EIO;
+ }
+
+ return ret;
+}
+
+static const struct v4l2_ctrl_ops bcm2835_isp_ctrl_ops = {
+ .s_ctrl = bcm2835_isp_s_ctrl,
+};
+
+static const struct v4l2_file_operations bcm2835_isp_fops = {
+ .owner = THIS_MODULE,
+ .open = v4l2_fh_open,
+ .release = vb2_fop_release,
+ .poll = vb2_fop_poll,
+ .unlocked_ioctl = video_ioctl2,
+ .mmap = vb2_fop_mmap
+};
+
+static int populate_qdata_fmt(struct v4l2_format *f,
+ struct bcm2835_isp_node *node)
+{
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ struct bcm2835_isp_q_data *q_data = &node->q_data;
+ int ret;
+
+ if (!node_is_stats(node)) {
+ v4l2_dbg(1, debug, &dev->v4l2_dev,
+ "%s: Setting pix format for type %d, wxh: %ux%u, fmt: %08x, size %u\n",
+ __func__, f->type, f->fmt.pix.width, f->fmt.pix.height,
+ f->fmt.pix.pixelformat, f->fmt.pix.sizeimage);
+
+ q_data->fmt = find_format(f, node);
+ q_data->width = f->fmt.pix.width;
+ q_data->height = f->fmt.pix.height;
+ q_data->height = f->fmt.pix.height;
+
+ /* All parameters should have been set correctly by try_fmt */
+ q_data->bytesperline = f->fmt.pix.bytesperline;
+ q_data->sizeimage = f->fmt.pix.sizeimage;
+ } else {
+ v4l2_dbg(1, debug, &dev->v4l2_dev,
+ "%s: Setting meta format for fmt: %08x, size %u\n",
+ __func__, f->fmt.meta.dataformat,
+ f->fmt.meta.buffersize);
+
+ q_data->fmt = find_format(f, node);
+ q_data->width = 0;
+ q_data->height = 0;
+ q_data->bytesperline = 0;
+ q_data->sizeimage = f->fmt.meta.buffersize;
+ }
+
+ v4l2_dbg(1, debug, &dev->v4l2_dev,
+ "%s: Calculated bpl as %u, size %u\n", __func__,
+ q_data->bytesperline, q_data->sizeimage);
+
+ setup_mmal_port_format(node, node->port);
+ ret = vchiq_mmal_port_set_format(dev->mmal_instance, node->port);
+ if (ret) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s: Failed vchiq_mmal_port_set_format on port, ret %d\n",
+ __func__, ret);
+ ret = -EINVAL;
+ }
+
+ if (q_data->sizeimage < node->port->minimum_buffer.size) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s: Current buffer size of %u < min buf size %u - driver mismatch to MMAL\n",
+ __func__,
+ q_data->sizeimage,
+ node->port->minimum_buffer.size);
+ }
+
+ v4l2_dbg(1, debug, &dev->v4l2_dev,
+ "%s: Set format for type %d, wxh: %dx%d, fmt: %08x, size %u\n",
+ __func__, f->type, q_data->width, q_data->height,
+ q_data->fmt->fourcc, q_data->sizeimage);
+
+ return ret;
+}
+
+static int bcm2835_isp_node_querycap(struct file *file, void *priv,
+ struct v4l2_capability *cap)
+{
+ strscpy(cap->driver, BCM2835_ISP_NAME, sizeof(cap->driver));
+ strscpy(cap->card, BCM2835_ISP_NAME, sizeof(cap->card));
+ snprintf(cap->bus_info, sizeof(cap->bus_info), "platform:%s",
+ BCM2835_ISP_NAME);
+
+ return 0;
+}
+
+static int bcm2835_isp_node_g_fmt(struct file *file, void *priv,
+ struct v4l2_format *f)
+{
+ struct bcm2835_isp_node *node = video_drvdata(file);
+
+ if (f->type != node->queue.type)
+ return -EINVAL;
+
+ if (node_is_stats(node)) {
+ f->fmt.meta.dataformat = V4L2_META_FMT_BCM2835_ISP_STATS;
+ f->fmt.meta.buffersize =
+ node->port->minimum_buffer.size;
+ } else {
+ struct bcm2835_isp_q_data *q_data = &node->q_data;
+
+ f->fmt.pix.width = q_data->width;
+ f->fmt.pix.height = q_data->height;
+ f->fmt.pix.field = V4L2_FIELD_NONE;
+ f->fmt.pix.pixelformat = q_data->fmt->fourcc;
+ f->fmt.pix.bytesperline = q_data->bytesperline;
+ f->fmt.pix.sizeimage = q_data->sizeimage;
+ f->fmt.pix.colorspace = q_data->fmt->colorspace;
+ }
+
+ return 0;
+}
+
+static int bcm2835_isp_node_enum_fmt(struct file *file, void *priv,
+ struct v4l2_fmtdesc *f)
+{
+ struct bcm2835_isp_node *node = video_drvdata(file);
+
+ if (f->type != node->queue.type)
+ return -EINVAL;
+
+ if (f->index < node->num_supported_fmts) {
+ /* Format found */
+ f->pixelformat = node->supported_fmts[f->index]->fourcc;
+ f->flags = 0;
+ return 0;
+ }
+
+ return -EINVAL;
+}
+
+static int bcm2835_isp_enum_framesizes(struct file *file, void *priv,
+ struct v4l2_frmsizeenum *fsize)
+{
+ struct bcm2835_isp_node *node = video_drvdata(file);
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ const struct bcm2835_isp_fmt *fmt;
+
+ if (node_is_stats(node) || fsize->index)
+ return -EINVAL;
+
+ fmt = find_format_by_fourcc(fsize->pixel_format, node);
+ if (!fmt) {
+ v4l2_err(&dev->v4l2_dev, "Invalid pixel code: %x\n",
+ fsize->pixel_format);
+ return -EINVAL;
+ }
+
+ fsize->type = V4L2_FRMSIZE_TYPE_STEPWISE;
+ fsize->stepwise.min_width = MIN_DIM;
+ fsize->stepwise.max_width = MAX_DIM;
+ fsize->stepwise.step_width = fmt->step_size;
+
+ fsize->stepwise.min_height = MIN_DIM;
+ fsize->stepwise.max_height = MAX_DIM;
+ fsize->stepwise.step_height = fmt->step_size;
+
+ return 0;
+}
+
+static int bcm2835_isp_node_try_fmt(struct file *file, void *priv,
+ struct v4l2_format *f)
+{
+ struct bcm2835_isp_node *node = video_drvdata(file);
+ const struct bcm2835_isp_fmt *fmt;
+
+ if (f->type != node->queue.type)
+ return -EINVAL;
+
+ fmt = find_format(f, node);
+ if (!fmt)
+ fmt = get_default_format(node);
+
+ if (!node_is_stats(node)) {
+ f->fmt.pix.width = max(min(f->fmt.pix.width, MAX_DIM),
+ MIN_DIM);
+ f->fmt.pix.height = max(min(f->fmt.pix.height, MAX_DIM),
+ MIN_DIM);
+
+ f->fmt.pix.pixelformat = fmt->fourcc;
+ f->fmt.pix.colorspace = fmt->colorspace;
+ f->fmt.pix.bytesperline = get_bytesperline(f->fmt.pix.width,
+ fmt);
+ f->fmt.pix.field = V4L2_FIELD_NONE;
+ f->fmt.pix.sizeimage =
+ get_sizeimage(f->fmt.pix.bytesperline, f->fmt.pix.width,
+ f->fmt.pix.height, fmt);
+ } else {
+ f->fmt.meta.dataformat = fmt->fourcc;
+ f->fmt.meta.buffersize = node->port->minimum_buffer.size;
+ }
+
+ return 0;
+}
+
+static int bcm2835_isp_node_s_fmt(struct file *file, void *priv,
+ struct v4l2_format *f)
+{
+ struct bcm2835_isp_node *node = video_drvdata(file);
+ int ret;
+
+ if (f->type != node->queue.type)
+ return -EINVAL;
+
+ ret = bcm2835_isp_node_try_fmt(file, priv, f);
+ if (ret)
+ return ret;
+
+ v4l2_dbg(1, debug, &node_get_dev(node)->v4l2_dev,
+ "%s: Set format for node %s[%d]\n",
+ __func__, node->name, node->id);
+
+ return populate_qdata_fmt(f, node);
+}
+
+static int bcm2835_isp_node_s_selection(struct file *file, void *fh,
+ struct v4l2_selection *s)
+{
+ struct mmal_parameter_crop crop;
+ struct bcm2835_isp_node *node = video_drvdata(file);
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+
+ /* This return value is required fro V4L2 compliance. */
+ if (node_is_stats(node))
+ return -ENOTTY;
+
+ if (!s->r.width || !s->r.height)
+ return -EINVAL;
+
+ /* We can only set crop on the input. */
+ switch (s->target) {
+ case V4L2_SEL_TGT_CROP:
+ /*
+ * Adjust the crop window if it goes outside of the frame
+ * dimensions.
+ */
+ s->r.left = min((unsigned int)max(s->r.left, 0),
+ node->q_data.width - MIN_DIM);
+ s->r.top = min((unsigned int)max(s->r.top, 0),
+ node->q_data.height - MIN_DIM);
+ s->r.width = max(min(s->r.width,
+ node->q_data.width - s->r.left), MIN_DIM);
+ s->r.height = max(min(s->r.height,
+ node->q_data.height - s->r.top), MIN_DIM);
+ break;
+ case V4L2_SEL_TGT_CROP_DEFAULT:
+ /* Default (i.e. no) crop window. */
+ s->r.left = 0;
+ s->r.top = 0;
+ s->r.width = node->q_data.width;
+ s->r.height = node->q_data.height;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ crop.rect.x = s->r.left;
+ crop.rect.y = s->r.top;
+ crop.rect.width = s->r.width;
+ crop.rect.height = s->r.height;
+
+ return vchiq_mmal_port_parameter_set(dev->mmal_instance, node->port,
+ MMAL_PARAMETER_CROP,
+ &crop, sizeof(crop));
+}
+
+static int bcm2835_isp_node_g_selection(struct file *file, void *fh,
+ struct v4l2_selection *s)
+{
+ struct mmal_parameter_crop crop;
+ struct bcm2835_isp_node *node = video_drvdata(file);
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ u32 crop_size = sizeof(crop);
+ int ret;
+
+ /* We can only return out an input crop. */
+ switch (s->target) {
+ case V4L2_SEL_TGT_CROP:
+ ret = vchiq_mmal_port_parameter_get(dev->mmal_instance,
+ node->port,
+ MMAL_PARAMETER_CROP,
+ &crop, &crop_size);
+ if (!ret) {
+ s->r.left = crop.rect.x;
+ s->r.top = crop.rect.y;
+ s->r.width = crop.rect.width;
+ s->r.height = crop.rect.height;
+ }
+ break;
+ case V4L2_SEL_TGT_CROP_DEFAULT:
+ case V4L2_SEL_TGT_CROP_BOUNDS:
+ /* Default (i.e. no) crop window. */
+ s->r.left = 0;
+ s->r.top = 0;
+ s->r.width = node->q_data.width;
+ s->r.height = node->q_data.height;
+ ret = 0;
+ break;
+ default:
+ ret = -EINVAL;
+ }
+
+ return ret;
+}
+
+static int bcm3285_isp_subscribe_event(struct v4l2_fh *fh,
+ const struct v4l2_event_subscription *s)
+{
+ switch (s->type) {
+ /* Cannot change source parameters dynamically at runtime. */
+ case V4L2_EVENT_SOURCE_CHANGE:
+ return -EINVAL;
+ case V4L2_EVENT_CTRL:
+ return v4l2_ctrl_subscribe_event(fh, s);
+ default:
+ return v4l2_event_subscribe(fh, s, 4, NULL);
+ }
+}
+
+static const struct v4l2_ioctl_ops bcm2835_isp_node_ioctl_ops = {
+ .vidioc_querycap = bcm2835_isp_node_querycap,
+ .vidioc_g_fmt_vid_cap = bcm2835_isp_node_g_fmt,
+ .vidioc_g_fmt_vid_out = bcm2835_isp_node_g_fmt,
+ .vidioc_g_fmt_meta_cap = bcm2835_isp_node_g_fmt,
+ .vidioc_s_fmt_vid_cap = bcm2835_isp_node_s_fmt,
+ .vidioc_s_fmt_vid_out = bcm2835_isp_node_s_fmt,
+ .vidioc_s_fmt_meta_cap = bcm2835_isp_node_s_fmt,
+ .vidioc_try_fmt_vid_cap = bcm2835_isp_node_try_fmt,
+ .vidioc_try_fmt_vid_out = bcm2835_isp_node_try_fmt,
+ .vidioc_try_fmt_meta_cap = bcm2835_isp_node_try_fmt,
+ .vidioc_s_selection = bcm2835_isp_node_s_selection,
+ .vidioc_g_selection = bcm2835_isp_node_g_selection,
+
+ .vidioc_enum_fmt_vid_cap = bcm2835_isp_node_enum_fmt,
+ .vidioc_enum_fmt_vid_out = bcm2835_isp_node_enum_fmt,
+ .vidioc_enum_fmt_meta_cap = bcm2835_isp_node_enum_fmt,
+ .vidioc_enum_framesizes = bcm2835_isp_enum_framesizes,
+
+ .vidioc_reqbufs = vb2_ioctl_reqbufs,
+ .vidioc_querybuf = vb2_ioctl_querybuf,
+ .vidioc_qbuf = vb2_ioctl_qbuf,
+ .vidioc_dqbuf = vb2_ioctl_dqbuf,
+ .vidioc_expbuf = vb2_ioctl_expbuf,
+ .vidioc_create_bufs = vb2_ioctl_create_bufs,
+ .vidioc_prepare_buf = vb2_ioctl_prepare_buf,
+
+ .vidioc_streamon = vb2_ioctl_streamon,
+ .vidioc_streamoff = vb2_ioctl_streamoff,
+
+ .vidioc_subscribe_event = bcm3285_isp_subscribe_event,
+ .vidioc_unsubscribe_event = v4l2_event_unsubscribe,
+};
+
+/*
+ * Size of the array to provide to the VPU when asking for the list of supported
+ * formats.
+ *
+ * The ISP component currently advertises 62 input formats, so add a small
+ * overhead on that. Should the component advertise more formats then the excess
+ * will be dropped and a warning logged.
+ */
+#define MAX_SUPPORTED_ENCODINGS 70
+
+/* Populate node->supported_fmts with the formats supported by those ports. */
+static int bcm2835_isp_get_supported_fmts(struct bcm2835_isp_node *node)
+{
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ struct bcm2835_isp_fmt const **list;
+ unsigned int i, j, num_encodings;
+ u32 fourccs[MAX_SUPPORTED_ENCODINGS];
+ u32 param_size = sizeof(fourccs);
+ int ret;
+
+ ret = vchiq_mmal_port_parameter_get(dev->mmal_instance, node->port,
+ MMAL_PARAMETER_SUPPORTED_ENCODINGS,
+ &fourccs, &param_size);
+
+ if (ret) {
+ if (ret == MMAL_MSG_STATUS_ENOSPC) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s: port has more encodings than we provided space for. Some are dropped (%zu vs %u).\n",
+ __func__, param_size / sizeof(u32),
+ MAX_SUPPORTED_ENCODINGS);
+ num_encodings = MAX_SUPPORTED_ENCODINGS;
+ } else {
+ v4l2_err(&dev->v4l2_dev, "%s: get_param ret %u.\n",
+ __func__, ret);
+ return -EINVAL;
+ }
+ } else {
+ num_encodings = param_size / sizeof(u32);
+ }
+
+ /*
+ * Assume at this stage that all encodings will be supported in V4L2.
+ * Any that aren't supported will waste a very small amount of memory.
+ */
+ list = devm_kzalloc(dev->dev,
+ sizeof(struct bcm2835_isp_fmt *) * num_encodings,
+ GFP_KERNEL);
+ if (!list)
+ return -ENOMEM;
+ node->supported_fmts = list;
+
+ for (i = 0, j = 0; i < num_encodings; i++) {
+ const struct bcm2835_isp_fmt *fmt = get_fmt(fourccs[i]);
+
+ if (fmt) {
+ list[j] = fmt;
+ j++;
+ }
+ }
+ node->num_supported_fmts = j;
+
+ return 0;
+}
+
+/*
+ * Register a device node /dev/video<N> to go along with one of the ISP's input
+ * or output nodes.
+ */
+static int register_node(struct bcm2835_isp_dev *dev,
+ struct bcm2835_isp_node *node,
+ int index)
+{
+ struct video_device *vfd;
+ struct vb2_queue *queue;
+ int ret;
+
+ mutex_init(&node->lock);
+ mutex_init(&node->queue_lock);
+
+ node->dev = dev;
+ vfd = &node->vfd;
+ queue = &node->queue;
+ queue->type = index_to_queue_type(index);
+ /*
+ * Setup the node type-specific params.
+ *
+ * Only the OUTPUT node can set controls and crop windows. However,
+ * we must allow the s/g_selection ioctl on the stats node as v4l2
+ * compliance expects it to return a -ENOTTY, and the framework
+ * does not handle it if the ioctl is disabled.
+ */
+ switch (queue->type) {
+ case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+ vfd->device_caps = V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_STREAMING;
+ node->id = index;
+ node->vfl_dir = VFL_DIR_TX;
+ node->name = "output";
+ node->port = &dev->component->input[node->id];
+ break;
+ case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+ vfd->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
+ /* First Capture node starts at id 0, etc. */
+ node->id = index - BCM2835_ISP_NUM_OUTPUTS;
+ node->vfl_dir = VFL_DIR_RX;
+ node->name = "capture";
+ node->port = &dev->component->output[node->id];
+ v4l2_disable_ioctl(&node->vfd, VIDIOC_S_CTRL);
+ v4l2_disable_ioctl(&node->vfd, VIDIOC_S_SELECTION);
+ v4l2_disable_ioctl(&node->vfd, VIDIOC_G_SELECTION);
+ break;
+ case V4L2_BUF_TYPE_META_CAPTURE:
+ vfd->device_caps = V4L2_CAP_META_CAPTURE | V4L2_CAP_STREAMING;
+ node->id = index - BCM2835_ISP_NUM_OUTPUTS;
+ node->vfl_dir = VFL_DIR_RX;
+ node->name = "stats";
+ node->port = &dev->component->output[node->id];
+ v4l2_disable_ioctl(&node->vfd, VIDIOC_S_CTRL);
+ v4l2_disable_ioctl(&node->vfd, VIDIOC_S_SELECTION);
+ v4l2_disable_ioctl(&node->vfd, VIDIOC_G_SELECTION);
+ break;
+ }
+
+ /* We use the selection API instead of the old crop API. */
+ v4l2_disable_ioctl(vfd, VIDIOC_CROPCAP);
+ v4l2_disable_ioctl(vfd, VIDIOC_G_CROP);
+ v4l2_disable_ioctl(vfd, VIDIOC_S_CROP);
+
+ ret = bcm2835_isp_get_supported_fmts(node);
+ if (ret)
+ return ret;
+
+ /* Initialise the video node. */
+ vfd->vfl_type = VFL_TYPE_VIDEO;
+ vfd->fops = &bcm2835_isp_fops,
+ vfd->ioctl_ops = &bcm2835_isp_node_ioctl_ops,
+ vfd->minor = -1,
+ vfd->release = video_device_release_empty,
+ vfd->queue = &node->queue;
+ vfd->lock = &node->lock;
+ vfd->v4l2_dev = &dev->v4l2_dev;
+ vfd->vfl_dir = node->vfl_dir;
+
+ node->q_data.fmt = get_default_format(node);
+ node->q_data.width = DEFAULT_DIM;
+ node->q_data.height = DEFAULT_DIM;
+ node->q_data.bytesperline =
+ get_bytesperline(DEFAULT_DIM, node->q_data.fmt);
+ node->q_data.sizeimage = node_is_stats(node) ?
+ node->port->recommended_buffer.size :
+ get_sizeimage(node->q_data.bytesperline,
+ node->q_data.width,
+ node->q_data.height,
+ node->q_data.fmt);
+
+ queue->io_modes = VB2_MMAP | VB2_DMABUF;
+ queue->drv_priv = node;
+ queue->ops = &bcm2835_isp_node_queue_ops;
+ queue->mem_ops = &vb2_dma_contig_memops;
+ queue->buf_struct_size = sizeof(struct bcm2835_isp_buffer);
+ queue->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
+ queue->dev = dev->dev;
+ queue->lock = &node->queue_lock;
+
+ ret = vb2_queue_init(queue);
+ if (ret < 0) {
+ v4l2_info(&dev->v4l2_dev, "vb2_queue_init failed\n");
+ return ret;
+ }
+
+ /* Set some controls and defaults, but only on the VIDEO_OUTPUT node. */
+ if (node_is_output(node)) {
+ unsigned int i;
+
+ /* Use this ctrl template to assign custom ISP ctrls. */
+ struct v4l2_ctrl_config ctrl_template = {
+ .ops = &bcm2835_isp_ctrl_ops,
+ .type = V4L2_CTRL_TYPE_U8,
+ .def = 0,
+ .min = 0x00,
+ .max = 0xff,
+ .step = 1,
+ };
+
+ /* 3 standard controls, and an array of custom controls */
+ ret = v4l2_ctrl_handler_init(&dev->ctrl_handler,
+ 3 + ARRAY_SIZE(custom_ctrls));
+ if (ret) {
+ v4l2_err(&dev->v4l2_dev, "ctrl_handler init failed (%d)\n",
+ ret);
+ goto queue_cleanup;
+ }
+
+ dev->r_gain = 1000;
+ dev->b_gain = 1000;
+
+ v4l2_ctrl_new_std(&dev->ctrl_handler, &bcm2835_isp_ctrl_ops,
+ V4L2_CID_RED_BALANCE, 1, 0xffff, 1,
+ dev->r_gain);
+
+ v4l2_ctrl_new_std(&dev->ctrl_handler, &bcm2835_isp_ctrl_ops,
+ V4L2_CID_BLUE_BALANCE, 1, 0xffff, 1,
+ dev->b_gain);
+
+ v4l2_ctrl_new_std(&dev->ctrl_handler, &bcm2835_isp_ctrl_ops,
+ V4L2_CID_DIGITAL_GAIN, 1, 0xffff, 1, 1000);
+
+ for (i = 0; i < ARRAY_SIZE(custom_ctrls); i++) {
+ ctrl_template.name = custom_ctrls[i].name;
+ ctrl_template.id = custom_ctrls[i].id;
+ ctrl_template.dims[0] = custom_ctrls[i].size;
+ ctrl_template.flags = custom_ctrls[i].flags;
+ v4l2_ctrl_new_custom(&dev->ctrl_handler,
+ &ctrl_template, NULL);
+ }
+
+ node->vfd.ctrl_handler = &dev->ctrl_handler;
+ if (dev->ctrl_handler.error) {
+ ret = dev->ctrl_handler.error;
+ v4l2_err(&dev->v4l2_dev, "controls init failed (%d)\n",
+ ret);
+ v4l2_ctrl_handler_free(&dev->ctrl_handler);
+ goto ctrl_cleanup;
+ }
+ }
+
+ /* Define the device names */
+ snprintf(vfd->name, sizeof(node->vfd.name), "%s-%s%d", BCM2835_ISP_NAME,
+ node->name, node->id);
+
+ ret = video_register_device(vfd, VFL_TYPE_VIDEO, video_nr + index);
+ if (ret) {
+ v4l2_err(&dev->v4l2_dev,
+ "Failed to register video %s[%d] device node\n",
+ node->name, node->id);
+ goto ctrl_cleanup;
+ }
+
+ node->registered = true;
+ video_set_drvdata(vfd, node);
+
+ v4l2_info(&dev->v4l2_dev,
+ "Device node %s[%d] registered as /dev/video%d\n",
+ node->name, node->id, vfd->num);
+
+ return 0;
+
+ctrl_cleanup:
+ if (node_is_output(node))
+ v4l2_ctrl_handler_free(&dev->ctrl_handler);
+queue_cleanup:
+ vb2_queue_release(&node->queue);
+ return ret;
+}
+
+/* Unregister one of the /dev/video<N> nodes associated with the ISP. */
+static void bcm2835_isp_unregister_node(struct bcm2835_isp_node *node)
+{
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+
+ v4l2_info(&dev->v4l2_dev,
+ "Unregistering node %s[%d] device node /dev/video%d\n",
+ node->name, node->id, node->vfd.num);
+
+ if (node->registered) {
+ video_unregister_device(&node->vfd);
+ if (node_is_output(node))
+ v4l2_ctrl_handler_free(&dev->ctrl_handler);
+ vb2_queue_release(&node->queue);
+ }
+
+ /*
+ * node->supported_fmts.list is free'd automatically
+ * as a managed resource.
+ */
+ node->supported_fmts = NULL;
+ node->num_supported_fmts = 0;
+ node->vfd.ctrl_handler = NULL;
+ node->registered = false;
+}
+
+static void media_controller_unregister(struct bcm2835_isp_dev *dev)
+{
+ unsigned int i;
+
+ v4l2_info(&dev->v4l2_dev, "Unregister from media controller\n");
+
+ if (dev->media_device_registered) {
+ media_device_unregister(&dev->mdev);
+ media_device_cleanup(&dev->mdev);
+ dev->media_device_registered = false;
+ }
+
+ kfree(dev->entity.name);
+ dev->entity.name = NULL;
+
+ if (dev->media_entity_registered) {
+ media_device_unregister_entity(&dev->entity);
+ dev->media_entity_registered = false;
+ }
+
+ for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) {
+ struct bcm2835_isp_node *node = &dev->node[i];
+
+ if (node->media_node_registered) {
+ media_remove_intf_links(node->intf_link->intf);
+ media_entity_remove_links(&dev->node[i].vfd.entity);
+ media_devnode_remove(node->intf_devnode);
+ media_device_unregister_entity(&node->vfd.entity);
+ kfree(node->vfd.entity.name);
+ }
+ node->media_node_registered = false;
+ }
+
+ dev->v4l2_dev.mdev = NULL;
+}
+
+static int media_controller_register_node(struct bcm2835_isp_dev *dev, int num)
+{
+ struct bcm2835_isp_node *node = &dev->node[num];
+ struct media_entity *entity = &node->vfd.entity;
+ int output = node_is_output(node);
+ char *name;
+ int ret;
+
+ v4l2_info(&dev->v4l2_dev,
+ "Register %s node %d with media controller\n",
+ output ? "output" : "capture", num);
+ entity->obj_type = MEDIA_ENTITY_TYPE_VIDEO_DEVICE;
+ entity->function = MEDIA_ENT_F_IO_V4L;
+ entity->info.dev.major = VIDEO_MAJOR;
+ entity->info.dev.minor = node->vfd.minor;
+ name = kmalloc(BCM2835_ISP_ENTITY_NAME_LEN, GFP_KERNEL);
+ if (!name) {
+ ret = -ENOMEM;
+ goto error_no_mem;
+ }
+ snprintf(name, BCM2835_ISP_ENTITY_NAME_LEN, "%s0-%s%d",
+ BCM2835_ISP_NAME, output ? "output" : "capture", num);
+ entity->name = name;
+ node->pad.flags = output ? MEDIA_PAD_FL_SOURCE : MEDIA_PAD_FL_SINK;
+ ret = media_entity_pads_init(entity, 1, &node->pad);
+ if (ret)
+ goto error_pads_init;
+ ret = media_device_register_entity(&dev->mdev, entity);
+ if (ret)
+ goto error_register_entity;
+
+ node->intf_devnode = media_devnode_create(&dev->mdev,
+ MEDIA_INTF_T_V4L_VIDEO, 0,
+ VIDEO_MAJOR, node->vfd.minor);
+ if (!node->intf_devnode) {
+ ret = -ENOMEM;
+ goto error_devnode_create;
+ }
+
+ node->intf_link = media_create_intf_link(entity,
+ &node->intf_devnode->intf,
+ MEDIA_LNK_FL_IMMUTABLE |
+ MEDIA_LNK_FL_ENABLED);
+ if (!node->intf_link) {
+ ret = -ENOMEM;
+ goto error_create_intf_link;
+ }
+
+ if (output)
+ ret = media_create_pad_link(entity, 0, &dev->entity, num,
+ MEDIA_LNK_FL_IMMUTABLE |
+ MEDIA_LNK_FL_ENABLED);
+ else
+ ret = media_create_pad_link(&dev->entity, num, entity, 0,
+ MEDIA_LNK_FL_IMMUTABLE |
+ MEDIA_LNK_FL_ENABLED);
+ if (ret)
+ goto error_create_pad_link;
+
+ dev->node[num].media_node_registered = true;
+ return 0;
+
+error_create_pad_link:
+ media_remove_intf_links(&node->intf_devnode->intf);
+error_create_intf_link:
+ media_devnode_remove(node->intf_devnode);
+error_devnode_create:
+ media_device_unregister_entity(&node->vfd.entity);
+error_register_entity:
+error_pads_init:
+ kfree(entity->name);
+ entity->name = NULL;
+error_no_mem:
+ if (ret)
+ v4l2_info(&dev->v4l2_dev, "Error registering node\n");
+
+ return ret;
+}
+
+static int media_controller_register(struct bcm2835_isp_dev *dev)
+{
+ char *name;
+ unsigned int i;
+ int ret;
+
+ v4l2_dbg(2, debug, &dev->v4l2_dev, "Registering with media controller\n");
+ dev->mdev.dev = dev->dev;
+ strscpy(dev->mdev.model, "bcm2835-isp",
+ sizeof(dev->mdev.model));
+ strscpy(dev->mdev.bus_info, "platform:bcm2835-isp",
+ sizeof(dev->mdev.bus_info));
+ media_device_init(&dev->mdev);
+ dev->v4l2_dev.mdev = &dev->mdev;
+
+ v4l2_dbg(2, debug, &dev->v4l2_dev, "Register entity for nodes\n");
+
+ name = kmalloc(BCM2835_ISP_ENTITY_NAME_LEN, GFP_KERNEL);
+ if (!name) {
+ ret = -ENOMEM;
+ goto done;
+ }
+ snprintf(name, BCM2835_ISP_ENTITY_NAME_LEN, "bcm2835_isp0");
+ dev->entity.name = name;
+ dev->entity.obj_type = MEDIA_ENTITY_TYPE_BASE;
+ dev->entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
+
+ for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) {
+ dev->pad[i].flags = node_is_output(&dev->node[i]) ?
+ MEDIA_PAD_FL_SINK : MEDIA_PAD_FL_SOURCE;
+ }
+
+ ret = media_entity_pads_init(&dev->entity, BCM2835_ISP_NUM_NODES,
+ dev->pad);
+ if (ret)
+ goto done;
+
+ ret = media_device_register_entity(&dev->mdev, &dev->entity);
+ if (ret)
+ goto done;
+
+ dev->media_entity_registered = true;
+ for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) {
+ ret = media_controller_register_node(dev, i);
+ if (ret)
+ goto done;
+ }
+
+ ret = media_device_register(&dev->mdev);
+ if (!ret)
+ dev->media_device_registered = true;
+done:
+ return ret;
+}
+
+static int bcm2835_isp_remove(struct platform_device *pdev)
+{
+ struct bcm2835_isp_dev *dev = platform_get_drvdata(pdev);
+ unsigned int i;
+
+ media_controller_unregister(dev);
+
+ for (i = 0; i < BCM2835_ISP_NUM_NODES; i++)
+ bcm2835_isp_unregister_node(&dev->node[i]);
+
+ v4l2_device_unregister(&dev->v4l2_dev);
+
+ if (dev->component)
+ vchiq_mmal_component_finalise(dev->mmal_instance,
+ dev->component);
+
+ vchiq_mmal_finalise(dev->mmal_instance);
+
+ return 0;
+}
+
+static int bcm2835_isp_probe(struct platform_device *pdev)
+{
+ struct bcm2835_isp_dev *dev;
+ unsigned int i;
+ int ret;
+
+ dev = devm_kzalloc(&pdev->dev, sizeof(*dev), GFP_KERNEL);
+ if (!dev)
+ return -ENOMEM;
+
+ dev->dev = &pdev->dev;
+
+ ret = v4l2_device_register(&pdev->dev, &dev->v4l2_dev);
+ if (ret)
+ return ret;
+
+ ret = vchiq_mmal_init(&dev->mmal_instance);
+ if (ret) {
+ v4l2_device_unregister(&dev->v4l2_dev);
+ return ret;
+ }
+
+ ret = vchiq_mmal_component_init(dev->mmal_instance, "ril.isp",
+ &dev->component);
+ if (ret) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s: failed to create ril.isp component\n", __func__);
+ goto error;
+ }
+
+ if (dev->component->inputs < BCM2835_ISP_NUM_OUTPUTS ||
+ dev->component->outputs < BCM2835_ISP_NUM_CAPTURES +
+ BCM2835_ISP_NUM_METADATA) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s: ril.isp returned %d i/p (%d expected), %d o/p (%d expected) ports\n",
+ __func__, dev->component->inputs,
+ BCM2835_ISP_NUM_OUTPUTS,
+ dev->component->outputs,
+ BCM2835_ISP_NUM_CAPTURES + BCM2835_ISP_NUM_METADATA);
+ goto error;
+ }
+
+ atomic_set(&dev->num_streaming, 0);
+
+ for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) {
+ struct bcm2835_isp_node *node = &dev->node[i];
+
+ ret = register_node(dev, node, i);
+ if (ret)
+ goto error;
+ }
+
+ ret = media_controller_register(dev);
+ if (ret)
+ goto error;
+
+ platform_set_drvdata(pdev, dev);
+ v4l2_info(&dev->v4l2_dev, "Loaded V4L2 %s\n", BCM2835_ISP_NAME);
+ return 0;
+
+error:
+ bcm2835_isp_remove(pdev);
+
+ return ret;
+}
+
+static struct platform_driver bcm2835_isp_pdrv = {
+ .probe = bcm2835_isp_probe,
+ .remove = bcm2835_isp_remove,
+ .driver = {
+ .name = BCM2835_ISP_NAME,
+ },
+};
+
+module_platform_driver(bcm2835_isp_pdrv);
+
+MODULE_DESCRIPTION("BCM2835 ISP driver");
+MODULE_AUTHOR("Naushir Patuck <[email protected]>");
+MODULE_LICENSE("GPL");
+MODULE_VERSION("1.0");
+MODULE_ALIAS("platform:bcm2835-isp");
diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h
index 4711877a9711..538c347db433 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h
+++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h
@@ -135,6 +135,10 @@
*/
#define MMAL_ENCODING_EGL_IMAGE MMAL_FOURCC('E', 'G', 'L', 'I')

+/** ISP image statistics format
+ */
+#define MMAL_ENCODING_BRCM_STATS MMAL_FOURCC('S', 'T', 'A', 'T')
+
/* }@ */

/** \name Pre-defined audio encodings */
diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-parameters.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-parameters.h
index a0cdd28101f2..5e6276aca143 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/mmal-parameters.h
+++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-parameters.h
@@ -223,6 +223,63 @@ enum mmal_parameter_camera_type {
MMAL_PARAMETER_SHUTTER_SPEED,
/**< Takes a @ref MMAL_PARAMETER_AWB_GAINS_T */
MMAL_PARAMETER_CUSTOM_AWB_GAINS,
+ /**< Takes a @ref MMAL_PARAMETER_CAMERA_SETTINGS_T */
+ MMAL_PARAMETER_CAMERA_SETTINGS,
+ /**< Takes a @ref MMAL_PARAMETER_PRIVACY_INDICATOR_T */
+ MMAL_PARAMETER_PRIVACY_INDICATOR,
+ /**< Takes a @ref MMAL_PARAMETER_BOOLEAN_T */
+ MMAL_PARAMETER_VIDEO_DENOISE,
+ /**< Takes a @ref MMAL_PARAMETER_BOOLEAN_T */
+ MMAL_PARAMETER_STILLS_DENOISE,
+ /**< Takes a @ref MMAL_PARAMETER_CAMERA_ANNOTATE_T */
+ MMAL_PARAMETER_ANNOTATE,
+ /**< Takes a @ref MMAL_PARAMETER_STEREOSCOPIC_MODE_T */
+ MMAL_PARAMETER_STEREOSCOPIC_MODE,
+ /**< Takes a @ref MMAL_PARAMETER_CAMERA_INTERFACE_T */
+ MMAL_PARAMETER_CAMERA_INTERFACE,
+ /**< Takes a @ref MMAL_PARAMETER_CAMERA_CLOCKING_MODE_T */
+ MMAL_PARAMETER_CAMERA_CLOCKING_MODE,
+ /**< Takes a @ref MMAL_PARAMETER_CAMERA_RX_CONFIG_T */
+ MMAL_PARAMETER_CAMERA_RX_CONFIG,
+ /**< Takes a @ref MMAL_PARAMETER_CAMERA_RX_TIMING_T */
+ MMAL_PARAMETER_CAMERA_RX_TIMING,
+ /**< Takes a @ref MMAL_PARAMETER_UINT32_T */
+ MMAL_PARAMETER_DPF_CONFIG,
+ /* 0x50 */
+ /**< Takes a @ref MMAL_PARAMETER_UINT32_T */
+ MMAL_PARAMETER_JPEG_RESTART_INTERVAL,
+ /**< Takes a @ref MMAL_PARAMETER_UINT32_T */
+ MMAL_PARAMETER_CAMERA_ISP_BLOCK_OVERRIDE,
+ /**< Takes a @ref MMAL_PARAMETER_LENS_SHADING_T */
+ MMAL_PARAMETER_LENS_SHADING_OVERRIDE,
+ /**< Takes a @ref MMAL_PARAMETER_UINT32_T */
+ MMAL_PARAMETER_BLACK_LEVEL,
+ /**< Takes a @ref MMAL_PARAMETER_RESIZE_T */
+ MMAL_PARAMETER_RESIZE_PARAMS,
+ /**< Takes a @ref MMAL_PARAMETER_CROP_T */
+ MMAL_PARAMETER_CROP,
+ /**< Takes a @ref MMAL_PARAMETER_INT32_T */
+ MMAL_PARAMETER_OUTPUT_SHIFT,
+ /**< Takes a @ref MMAL_PARAMETER_INT32_T */
+ MMAL_PARAMETER_CCM_SHIFT,
+ /**< Takes a @ref MMAL_PARAMETER_CUSTOM_CCM_T */
+ MMAL_PARAMETER_CUSTOM_CCM,
+ /**< Takes a @ref MMAL_PARAMETER_RATIONAL_T */
+ MMAL_PARAMETER_ANALOG_GAIN,
+ /**< Takes a @ref MMAL_PARAMETER_RATIONAL_T */
+ MMAL_PARAMETER_DIGITAL_GAIN,
+ /**< Takes a @ref MMAL_PARAMETER_DENOISE_T */
+ MMAL_PARAMETER_DENOISE,
+ /**< Takes a @ref MMAL_PARAMETER_SHARPEN_T */
+ MMAL_PARAMETER_SHARPEN,
+ /**< Takes a @ref MMAL_PARAMETER_GEQ_T */
+ MMAL_PARAMETER_GEQ,
+ /**< Tales a @ref MMAP_PARAMETER_DPC_T */
+ MMAL_PARAMETER_DPC,
+ /**< Tales a @ref MMAP_PARAMETER_GAMMA_T */
+ MMAL_PARAMETER_GAMMA,
+ /**< Takes a @ref MMAL_PARAMETER_CDN_T */
+ MMAL_PARAMETER_CDN,
};

enum mmal_parameter_camera_config_timestamp_mode {
@@ -746,7 +803,113 @@ struct mmal_parameter_camera_info {
struct mmal_parameter_camera_info_camera
cameras[MMAL_PARAMETER_CAMERA_INFO_MAX_CAMERAS];
struct mmal_parameter_camera_info_flash
- flashes[MMAL_PARAMETER_CAMERA_INFO_MAX_FLASHES];
+ flashes[MMAL_PARAMETER_CAMERA_INFO_MAX_FLASHES];
+};
+
+struct mmal_parameter_ccm {
+ struct s32_fract cm[3][3];
+ s32 offsets[3];
+};
+
+struct mmal_parameter_custom_ccm {
+ u32 enabled; /**< Enable the custom CCM. */
+ struct mmal_parameter_ccm ccm; /**< CCM to be used. */
+};
+
+struct mmal_parameter_lens_shading {
+ u32 enabled;
+ u32 grid_cell_size;
+ u32 grid_width;
+ u32 grid_stride;
+ u32 grid_height;
+ u32 mem_handle_table;
+ u32 ref_transform;
+};
+
+enum mmal_parameter_ls_gain_format_type {
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U0P8_1 = 0,
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U1P7_0 = 1,
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U1P7_1 = 2,
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U2P6_0 = 3,
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U2P6_1 = 4,
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U3P5_0 = 5,
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U3P5_1 = 6,
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U4P10 = 7,
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_DUMMY = 0x7FFFFFFF
+};
+
+struct mmal_parameter_lens_shading_v2 {
+ u32 enabled;
+ u32 grid_cell_size;
+ u32 grid_width;
+ u32 grid_stride;
+ u32 grid_height;
+ u32 mem_handle_table;
+ u32 ref_transform;
+ u32 corner_sampled;
+ enum mmal_parameter_ls_gain_format_type gain_format;
+};
+
+struct mmal_parameter_black_level {
+ u32 enabled;
+ u16 black_level_r;
+ u16 black_level_g;
+ u16 black_level_b;
+ u8 pad_[2]; /* Unused */
+};
+
+struct mmal_parameter_geq {
+ u32 enabled;
+ u32 offset;
+ struct s32_fract slope;
+};
+
+#define MMAL_NUM_GAMMA_PTS 33
+struct mmal_parameter_gamma {
+ u32 enabled;
+ u16 x[MMAL_NUM_GAMMA_PTS];
+ u16 y[MMAL_NUM_GAMMA_PTS];
+};
+
+enum mmal_parameter_cdn_mode {
+ MMAL_PARAM_CDN_FAST = 0,
+ MMAL_PARAM_CDN_HIGH_QUALITY = 1,
+ MMAL_PARAM_CDN_DUMMY = 0x7FFFFFFF
+};
+
+struct mmal_parameter_colour_denoise {
+ u32 enabled;
+ enum mmal_parameter_cdn_mode mode;
+};
+
+struct mmal_parameter_denoise {
+ u32 enabled;
+ u32 constant;
+ struct s32_fract slope;
+ struct s32_fract strength;
+};
+
+struct mmal_parameter_sharpen {
+ u32 enabled;
+ struct s32_fract threshold;
+ struct s32_fract strength;
+ struct s32_fract limit;
+};
+
+enum mmal_dpc_mode {
+ MMAL_DPC_MODE_OFF = 0,
+ MMAL_DPC_MODE_NORMAL = 1,
+ MMAL_DPC_MODE_STRONG = 2,
+ MMAL_DPC_MODE_MAX = 0x7FFFFFFF,
+};
+
+struct mmal_parameter_dpc {
+ u32 enabled;
+ u32 strength;
+};
+
+struct mmal_parameter_crop {
+ struct vchiq_mmal_rect rect;
};

#endif
diff --git a/include/uapi/linux/bcm2835-isp.h b/include/uapi/linux/bcm2835-isp.h
index cf8c0437f159..c50e3ca81565 100644
--- a/include/uapi/linux/bcm2835-isp.h
+++ b/include/uapi/linux/bcm2835-isp.h
@@ -31,6 +31,8 @@
(V4L2_CID_USER_BCM2835_ISP_BASE + 0x0007)
#define V4L2_CID_USER_BCM2835_ISP_DPC \
(V4L2_CID_USER_BCM2835_ISP_BASE + 0x0008)
+#define V4L2_CID_USER_BCM2835_ISP_CDN \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0009)

/*
* All structs below are directly mapped onto the equivalent structs in
@@ -175,6 +177,31 @@ struct bcm2835_isp_gamma {
__u16 y[BCM2835_NUM_GAMMA_PTS];
};

+/**
+ * enum bcm2835_isp_cdn_mode - Mode of operation for colour denoise.
+ *
+ * @CDN_MODE_FAST: Fast (but lower quality) colour denoise
+ * algorithm, typically used for video recording.
+ * @CDN_HIGH_QUALITY: High quality (but slower) colour denoise
+ * algorithm, typically used for stills capture.
+ */
+enum bcm2835_isp_cdn_mode {
+ CDN_MODE_FAST = 0,
+ CDN_MODE_HIGH_QUALITY = 1,
+};
+
+/**
+ * struct bcm2835_isp_cdn - Colour denoise parameters set with the
+ * V4L2_CID_USER_BCM2835_ISP_CDN ctrl.
+ *
+ * @enabled: Enable colour denoise.
+ * @mode: Colour denoise operating mode (see enum &bcm2835_isp_cdn_mode)
+ */
+struct bcm2835_isp_cdn {
+ __u32 enabled;
+ __u32 mode;
+};
+
/**
* struct bcm2835_isp_denoise - Denoise parameters set with the
* V4L2_CID_USER_BCM2835_ISP_DENOISE ctrl.
--
2.37.3

2022-11-21 22:31:13

[permalink] [raw]

Subject: [PATCH 11/14] WIP: vc04_services: bcm2835-isp: Permit all sRGB colour spaces on ISP outputs

From: David Plowman <[email protected]>

bcm2835-isp outputs actually support all colour spaces that are
fundamentally sRGB underneath, regardless of whether an RGB or YUV output
format is actually requested.

Signed-off-by: David Plowman <[email protected]>
Signed-off-by: Umang Jain <[email protected]>
---
.../bcm2835-isp/bcm2835-isp-fmts.h | 45 ++++++++++---------
1 file changed, 25 insertions(+), 20 deletions(-)

diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
index a545dbf2b5dd..5ab232ff9bd9 100644
--- a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
+++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
@@ -34,14 +34,19 @@ struct bcm2835_isp_fmt {
#define V4L2_COLORSPACE_MASK_RAW V4L2_COLORSPACE_MASK(V4L2_COLORSPACE_RAW)

/*
- * The colour spaces we support for YUV outputs. SRGB features here because,
- * once you assign the default transfer func and so on, it and JPEG effectively
- * mean the same.
+ * All three colour spaces JPEG, SMPTE170M and REC709 are fundamentally sRGB
+ * underneath (as near as makes no difference to us), just with different YCbCr
+ * encodings. Therefore the ISP can generate sRGB on its main output and any of
+ * the others on its low resolution output. Applications should, when using both
+ * outputs, program the colour spaces on them to be the same, matching whatever
+ * is requested for the low resolution output, even if the main output is
+ * producing an RGB format. In turn this requires us to allow all these colour
+ * spaces for every YUV/RGB output format.
*/
-#define V4L2_COLORSPACE_MASK_YUV (V4L2_COLORSPACE_MASK_JPEG | \
- V4L2_COLORSPACE_MASK_SRGB | \
- V4L2_COLORSPACE_MASK_SMPTE170M | \
- V4L2_COLORSPACE_MASK_REC709)
+#define V4L2_COLORSPACE_MASK_ALL_SRGB (V4L2_COLORSPACE_MASK_JPEG | \
+ V4L2_COLORSPACE_MASK_SRGB | \
+ V4L2_COLORSPACE_MASK_SMPTE170M | \
+ V4L2_COLORSPACE_MASK_REC709)

static const struct bcm2835_isp_fmt supported_formats[] = {
{
@@ -51,7 +56,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_I420,
.size_multiplier_x2 = 3,
- .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_JPEG,
.step_size = 2,
}, {
@@ -60,7 +65,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_YV12,
.size_multiplier_x2 = 3,
- .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
@@ -69,7 +74,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_NV12,
.size_multiplier_x2 = 3,
- .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
@@ -78,7 +83,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_NV21,
.size_multiplier_x2 = 3,
- .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
@@ -87,7 +92,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_YUYV,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
@@ -96,7 +101,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_UYVY,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
@@ -105,7 +110,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_YVYU,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
@@ -114,7 +119,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_VYUY,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
@@ -124,7 +129,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_RGB24,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SRGB,
.step_size = 1,
}, {
@@ -133,7 +138,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_RGB16,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SRGB,
.step_size = 1,
}, {
@@ -142,7 +147,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BGR24,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SRGB,
.step_size = 1,
}, {
@@ -151,7 +156,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_BGRA,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SRGB,
.step_size = 1,
}, {
@@ -160,7 +165,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_RGBA,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SRGB,
.step_size = 1,
}, {
--
2.37.3

2022-11-21 22:55:40

[permalink] [raw]

Subject: [PATCH 12/14] staging: vc04_services: bcm2835_isp: Allow multiple users

From: Naushir Patuck <[email protected]>

Add a second (identical) set of device nodes to allow concurrent use of
the bcm2835-isp hardware by another user. This change effectively
creates a second state structure (struct bcm2835_isp_dev) to maintain
independent state for the second user. Node and media entity names are
appened with the instance index appropriately.

Further users can be added by changing the BCM2835_ISP_NUM_INSTANCES
define.

Signed-off-by: Naushir Patuck <[email protected]>
Signed-off-by: Umang Jain <[email protected]>
---
.../bcm2835-isp/bcm2835-v4l2-isp.c | 77 +++++++++++++++----
1 file changed, 61 insertions(+), 16 deletions(-)

diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
index cb7cdba76682..0dbcb25595e7 100644
--- a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
+++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
@@ -28,13 +28,19 @@

MODULE_IMPORT_NS(DMA_BUF);

+/*
+ * We want to instantiate 2 independent instances allowing 2 simultaneous users
+ * of the ISP hardware.
+ */
+#define BCM2835_ISP_NUM_INSTANCES 2
+
static unsigned int debug;
module_param(debug, uint, 0644);
MODULE_PARM_DESC(debug, "activates debug info");

-static unsigned int video_nr = 13;
-module_param(video_nr, uint, 0644);
-MODULE_PARM_DESC(video_nr, "base video device number");
+static unsigned int video_nr[BCM2835_ISP_NUM_INSTANCES] = { 13, 20 };
+module_param_array(video_nr, uint, NULL, 0644);
+MODULE_PARM_DESC(video_nr, "base video device numbers");

#define BCM2835_ISP_NAME "bcm2835-isp"
#define BCM2835_ISP_ENTITY_NAME_LEN 32
@@ -1286,6 +1292,7 @@ static int bcm2835_isp_get_supported_fmts(struct bcm2835_isp_node *node)
* or output nodes.
*/
static int register_node(struct bcm2835_isp_dev *dev,
+ unsigned int instance,
struct bcm2835_isp_node *node,
int index)
{
@@ -1447,7 +1454,7 @@ static int register_node(struct bcm2835_isp_dev *dev,
snprintf(vfd->name, sizeof(node->vfd.name), "%s-%s%d", BCM2835_ISP_NAME,
node->name, node->id);

- ret = video_register_device(vfd, VFL_TYPE_VIDEO, video_nr + index);
+ ret = video_register_device(vfd, VFL_TYPE_VIDEO, video_nr[instance]);
if (ret) {
v4l2_err(&dev->v4l2_dev,
"Failed to register video %s[%d] device node\n",
@@ -1668,9 +1675,8 @@ static int media_controller_register(struct bcm2835_isp_dev *dev)
return ret;
}

-static int bcm2835_isp_remove(struct platform_device *pdev)
+static void bcm2835_isp_remove_instance(struct bcm2835_isp_dev *dev)
{
- struct bcm2835_isp_dev *dev = platform_get_drvdata(pdev);
unsigned int i;

media_controller_unregister(dev);
@@ -1685,11 +1691,11 @@ static int bcm2835_isp_remove(struct platform_device *pdev)
dev->component);

vchiq_mmal_finalise(dev->mmal_instance);
-
- return 0;
}

-static int bcm2835_isp_probe(struct platform_device *pdev)
+static int bcm2835_isp_probe_instance(struct platform_device *pdev,
+ struct bcm2835_isp_dev **dev_int,
+ unsigned int instance)
{
struct bcm2835_isp_dev *dev;
unsigned int i;
@@ -1699,6 +1705,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
if (!dev)
return -ENOMEM;

+ *dev_int = dev;
dev->dev = &pdev->dev;

ret = v4l2_device_register(&pdev->dev, &dev->v4l2_dev);
@@ -1716,7 +1723,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
if (ret) {
v4l2_err(&dev->v4l2_dev,
"%s: failed to create ril.isp component\n", __func__);
- goto error;
+ return ret;
}

if (dev->component->inputs < BCM2835_ISP_NUM_OUTPUTS ||
@@ -1728,7 +1735,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
BCM2835_ISP_NUM_OUTPUTS,
dev->component->outputs,
BCM2835_ISP_NUM_CAPTURES + BCM2835_ISP_NUM_METADATA);
- goto error;
+ return -EINVAL;
}

atomic_set(&dev->num_streaming, 0);
@@ -1736,17 +1743,55 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) {
struct bcm2835_isp_node *node = &dev->node[i];

- ret = register_node(dev, node, i);
+ ret = register_node(dev, instance, node, i);
if (ret)
- goto error;
+ return ret;
}

ret = media_controller_register(dev);
if (ret)
- goto error;
+ return ret;
+
+ return 0;
+}
+
+static int bcm2835_isp_remove(struct platform_device *pdev)
+{
+ struct bcm2835_isp_dev **bcm2835_isp_instances;
+ unsigned int i;
+
+ bcm2835_isp_instances = platform_get_drvdata(pdev);
+ for (i = 0; i < BCM2835_ISP_NUM_INSTANCES; i++) {
+ if (bcm2835_isp_instances[i])
+ bcm2835_isp_remove_instance(bcm2835_isp_instances[i]);
+ }
+
+ return 0;
+}
+
+static int bcm2835_isp_probe(struct platform_device *pdev)
+{
+ struct bcm2835_isp_dev **bcm2835_isp_instances;
+ unsigned int i;
+ int ret;
+
+ bcm2835_isp_instances = devm_kzalloc(&pdev->dev,
+ sizeof(bcm2835_isp_instances) *
+ BCM2835_ISP_NUM_INSTANCES,
+ GFP_KERNEL);
+ if (!bcm2835_isp_instances)
+ return -ENOMEM;
+
+ platform_set_drvdata(pdev, bcm2835_isp_instances);
+
+ for (i = 0; i < BCM2835_ISP_NUM_INSTANCES; i++) {
+ ret = bcm2835_isp_probe_instance(pdev,
+ &bcm2835_isp_instances[i], i);
+ if (ret)
+ goto error;
+ }

- platform_set_drvdata(pdev, dev);
- v4l2_info(&dev->v4l2_dev, "Loaded V4L2 %s\n", BCM2835_ISP_NAME);
+ dev_info(&pdev->dev, "Loaded V4L2 %s\n", BCM2835_ISP_NAME);
return 0;

error:
--
2.37.3

2022-11-21 22:58:48

[permalink] [raw]

Subject: Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support

Hi Umang,

Nice to see this series on the list !

On Tue, Nov 22, 2022 at 03:17:08AM +0530, Umang Jain wrote:
> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> SoCs (namely BCM2711). Unicam is still under active development to work
> with multistream support to get into mainline. Hence only the ISP driver
> will remain the primary area of this series.
>
> Patch (01-02)/14 adds a new driver named vc-sm-cma to handle memory sharing
> with the VC4 VPU.
>
> Patch 03/14 adds a small extension to videobuf2 to allow exporting as a
> dma_buf instead of a file-descriptor.
>
> Patch (04-05)/14 adds a couple of improvements/support for
> bcm2835-isp(event callback and zero-copy) to vchiq-mmal.
>
> Patch (06-09)/14 adds the core bcm2835-isp driver along with headers
> and format defintions.
>
> Patch (10-11)/14 deals with the colorspace support.
> Note: This is still WIP since the implementation of colorspace is still
> getting ironed out (especially around setting of colorspace flags handling).
>
> Patch 12/14 allows multiple instances of the ISP.
>
> Patch 13/14 adds a admin-guide document on bcm2835-isp.
>
> Patch 14/14 wires all this up with the vchiq-mmal driver.
>
> Testing:
> Tested with RPi Model 4B running linux mainline v6.1-rc6. To test
> end-to-end, I choose to cherry-pick the Unicam patches and OV5647 DT
> bindings from [1]). Once done, functional testing was conducted with
> libcamera[3] and its utility tools.
>
> Also note: Reviews given on [2] for the relevant ISP driver patches have
> been incorporated in this version.
>
> Known issues:
> - Colorspace handling

This will require further discussions, I'll try to comment on this topic
in the review of the ISP driver patch.

> - vc-sm-cma spamming kernel log with
> - pr_err("%s: Expecting an uncached alias for dma_addr %pad\n"

Do you have any plan to address this ? Is the root cause known ?

> [1]: https://github.com/raspberrypi/linux
> [2]: https://lore.kernel.org/linux-media/[email protected]/
> [3]: https://libcamera.org/getting-started.html
>
> Dave Stevenson (7):
> staging: vc04_services: Add new vc-sm-cma driver
> staging: vchiq_arm: Register vcsm-cma as a platform driver
> media: videobuf2: Allow exporting of a struct dmabuf
> staging: mmal-vchiq: Add support for event callbacks
> staging: mmal-vchiq: Use vc-sm-cma to support zero copy
> staging: mmal_vchiq: Add image formats to be used by bcm2835-isp
> uapi: bcm2835-isp: Add bcm2835-isp uapi header file
>
> David Plowman (2):
> vc04_services: bcm2835-isp: Allow formats with different colour spaces
> vc04_services: bcm2835-isp: Permit all sRGB colour spaces on ISP
> outputs
>
> Naushir Patuck (5):
> media: uapi: v4l2-core: Add ISP statistics output V4L2 fourcc type
> staging: vc04_services: bcm2835-isp: Add a more complex ISP processing
> component
> staging: vc04_services: bcm2835_isp: Allow multiple users
> docs: admin-guide: media: bcm2835-isp: Add documentation for
> bcm2835-isp
> staging: vc04_services: vchiq: Load bcm2835_isp driver from vchiq
>
> .../admin-guide/media/bcm2835-isp.rst | 127 ++
> .../userspace-api/media/drivers/index.rst | 1 +
> .../userspace-api/media/v4l/meta-formats.rst | 1 +
> .../v4l/pixfmt-meta-bcm2835-isp-stats.rst | 41 +
> MAINTAINERS | 9 +
> .../media/common/videobuf2/videobuf2-core.c | 36 +-
> drivers/media/v4l2-core/v4l2-ioctl.c | 1 +
> drivers/staging/vc04_services/Kconfig | 4 +
> drivers/staging/vc04_services/Makefile | 2 +
> .../staging/vc04_services/bcm2835-isp/Kconfig | 14 +
> .../vc04_services/bcm2835-isp/Makefile | 8 +
> .../bcm2835-isp/bcm2835-isp-ctrls.h | 72 +
> .../bcm2835-isp/bcm2835-isp-fmts.h | 558 +++++
> .../bcm2835-isp/bcm2835-v4l2-isp.c | 1817 +++++++++++++++++
> .../interface/vchiq_arm/vchiq_arm.c | 6 +
> .../staging/vc04_services/vc-sm-cma/Kconfig | 10 +
> .../staging/vc04_services/vc-sm-cma/Makefile | 12 +
> .../staging/vc04_services/vc-sm-cma/vc_sm.c | 801 ++++++++
> .../staging/vc04_services/vc-sm-cma/vc_sm.h | 54 +
> .../vc04_services/vc-sm-cma/vc_sm_cma_vchi.c | 507 +++++
> .../vc04_services/vc-sm-cma/vc_sm_cma_vchi.h | 63 +
> .../vc04_services/vc-sm-cma/vc_sm_defs.h | 187 ++
> .../vc04_services/vc-sm-cma/vc_sm_knl.h | 28 +
> .../staging/vc04_services/vchiq-mmal/Kconfig | 1 +
> .../vc04_services/vchiq-mmal/mmal-common.h | 5 +
> .../vc04_services/vchiq-mmal/mmal-encodings.h | 66 +
> .../vc04_services/vchiq-mmal/mmal-msg.h | 35 +
> .../vchiq-mmal/mmal-parameters.h | 165 +-
> .../vc04_services/vchiq-mmal/mmal-vchiq.c | 253 ++-
> .../vc04_services/vchiq-mmal/mmal-vchiq.h | 5 +
> include/media/videobuf2-core.h | 15 +
> include/uapi/linux/bcm2835-isp.h | 347 ++++
> include/uapi/linux/v4l2-controls.h | 6 +
> include/uapi/linux/videodev2.h | 1 +
> 34 files changed, 5225 insertions(+), 33 deletions(-)
> create mode 100644 Documentation/admin-guide/media/bcm2835-isp.rst
> create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst
> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Kconfig
> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Makefile
> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h
> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Kconfig
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Makefile
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.c
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.h
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h
> create mode 100644 include/uapi/linux/bcm2835-isp.h

--
Regards,

Laurent Pinchart

2022-11-21 22:59:22

[permalink] [raw]

Subject: [PATCH 05/14] staging: mmal-vchiq: Use vc-sm-cma to support zero copy

From: Dave Stevenson <[email protected]>

With the vc-sm-cma driver we can support zero copy of buffers between
the kernel and VPU. Add this support to mmal-vchiq.

Signed-off-by: Dave Stevenson <[email protected]>
Signed-off-by: Umang Jain <[email protected]>
---
.../staging/vc04_services/vchiq-mmal/Kconfig | 1 +
.../vc04_services/vchiq-mmal/mmal-common.h | 4 +
.../vc04_services/vchiq-mmal/mmal-vchiq.c | 83 +++++++++++++++++--
.../vc04_services/vchiq-mmal/mmal-vchiq.h | 1 +
4 files changed, 82 insertions(+), 7 deletions(-)

diff --git a/drivers/staging/vc04_services/vchiq-mmal/Kconfig b/drivers/staging/vc04_services/vchiq-mmal/Kconfig
index c99525a0bb45..a7c1a7bf516e 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/Kconfig
+++ b/drivers/staging/vc04_services/vchiq-mmal/Kconfig
@@ -1,6 +1,7 @@
config BCM2835_VCHIQ_MMAL
tristate "BCM2835 MMAL VCHIQ service"
depends on BCM2835_VCHIQ
+ select BCM_VC_SM_CMA
help
Enables the MMAL API over VCHIQ interface as used for the
majority of the multimedia services on VideoCore.
diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h
index 0443be8198ea..a643cad54b12 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h
+++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h
@@ -50,6 +50,10 @@ struct mmal_buffer {

struct mmal_msg_context *msg_context;

+ struct dma_buf *dma_buf;/* Exported dmabuf fd from videobuf2 */
+ void *vcsm_handle; /* VCSM handle having imported the dmabuf */
+ u32 vc_handle; /* VC handle to that dmabuf */
+
u32 cmd; /* MMAL command. 0=data. */
unsigned long length;
u32 mmal_flags;
diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c
index 1c0047eead7e..f26fb64a2785 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c
+++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c
@@ -27,9 +27,11 @@
#include <media/videobuf2-vmalloc.h>

#include "mmal-common.h"
+#include "mmal-parameters.h"
#include "mmal-vchiq.h"
#include "mmal-msg.h"

+#include "vc-sm-cma/vc_sm_knl.h"
/*
* maximum number of components supported.
* This matches the maximum permitted by default on the VPU
@@ -416,14 +418,27 @@ buffer_from_host(struct vchiq_mmal_instance *instance,

/* buffer header */
m.u.buffer_from_host.buffer_header.cmd = 0;
- m.u.buffer_from_host.buffer_header.data =
- (u32)(unsigned long)buf->buffer;
+ if (port->zero_copy) {
+ m.u.buffer_from_host.buffer_header.data = buf->vc_handle;
+ } else {
+ m.u.buffer_from_host.buffer_header.data =
+ (u32)(unsigned long)buf->buffer;
+ }
+
m.u.buffer_from_host.buffer_header.alloc_size = buf->buffer_size;
- m.u.buffer_from_host.buffer_header.length = 0; /* nothing used yet */
- m.u.buffer_from_host.buffer_header.offset = 0; /* no offset */
- m.u.buffer_from_host.buffer_header.flags = 0; /* no flags */
- m.u.buffer_from_host.buffer_header.pts = MMAL_TIME_UNKNOWN;
- m.u.buffer_from_host.buffer_header.dts = MMAL_TIME_UNKNOWN;
+ if (port->type == MMAL_PORT_TYPE_OUTPUT) {
+ m.u.buffer_from_host.buffer_header.length = 0;
+ m.u.buffer_from_host.buffer_header.offset = 0;
+ m.u.buffer_from_host.buffer_header.flags = 0;
+ m.u.buffer_from_host.buffer_header.pts = MMAL_TIME_UNKNOWN;
+ m.u.buffer_from_host.buffer_header.dts = MMAL_TIME_UNKNOWN;
+ } else {
+ m.u.buffer_from_host.buffer_header.length = buf->length;
+ m.u.buffer_from_host.buffer_header.offset = 0;
+ m.u.buffer_from_host.buffer_header.flags = buf->mmal_flags;
+ m.u.buffer_from_host.buffer_header.pts = buf->pts;
+ m.u.buffer_from_host.buffer_header.dts = buf->dts;
+ }

/* clear buffer type specific data */
memset(&m.u.buffer_from_host.buffer_header_type_specific, 0,
@@ -575,6 +590,22 @@ static void buffer_to_host_cb(struct vchiq_mmal_instance *instance,

msg_context->u.bulk.status = msg->h.status;

+ } else if (msg->u.buffer_from_host.is_zero_copy) {
+ /*
+ * Zero copy buffer, so nothing to do.
+ * Copy buffer info and make callback.
+ */
+ msg_context->u.bulk.buffer_used =
+ msg->u.buffer_from_host.buffer_header.length;
+ msg_context->u.bulk.mmal_flags =
+ msg->u.buffer_from_host.buffer_header.flags;
+ msg_context->u.bulk.dts =
+ msg->u.buffer_from_host.buffer_header.dts;
+ msg_context->u.bulk.pts =
+ msg->u.buffer_from_host.buffer_header.pts;
+ msg_context->u.bulk.cmd =
+ msg->u.buffer_from_host.buffer_header.cmd;
+
} else if (msg->u.buffer_from_host.buffer_header.length == 0) {
/* empty buffer */
if (msg->u.buffer_from_host.buffer_header.flags &
@@ -1520,6 +1551,9 @@ int vchiq_mmal_port_parameter_set(struct vchiq_mmal_instance *instance,

mutex_unlock(&instance->vchiq_mutex);

+ if (parameter == MMAL_PARAMETER_ZERO_COPY && !ret)
+ port->zero_copy = !!(*(bool *)value);
+
return ret;
}
EXPORT_SYMBOL_GPL(vchiq_mmal_port_parameter_set);
@@ -1688,6 +1722,31 @@ int vchiq_mmal_submit_buffer(struct vchiq_mmal_instance *instance,
unsigned long flags = 0;
int ret;

+ /*
+ * We really want to do this in mmal_vchi_buffer_init but can't as
+ * videobuf2 won't let us have the dmabuf there.
+ */
+ if (port->zero_copy && buffer->dma_buf && !buffer->vcsm_handle) {
+ pr_debug("%s: import dmabuf %p\n", __func__, buffer->dma_buf);
+ ret = vc_sm_cma_import_dmabuf(buffer->dma_buf,
+ &buffer->vcsm_handle);
+ if (ret) {
+ pr_err("%s: vc_sm_import_dmabuf_fd failed, ret %d\n",
+ __func__, ret);
+ return ret;
+ }
+
+ buffer->vc_handle = vc_sm_cma_int_handle(buffer->vcsm_handle);
+ if (!buffer->vc_handle) {
+ pr_err("%s: vc_sm_int_handle failed %d\n",
+ __func__, ret);
+ vc_sm_cma_free(buffer->vcsm_handle);
+ return ret;
+ }
+ pr_debug("%s: import dmabuf %p - got vc handle %08X\n",
+ __func__, buffer->dma_buf, buffer->vc_handle);
+ }
+
ret = buffer_from_host(instance, port, buffer);
if (ret == -EINVAL) {
/* Port is disabled. Queue for when it is enabled. */
@@ -1721,6 +1780,16 @@ int mmal_vchi_buffer_cleanup(struct mmal_buffer *buf)
release_msg_context(msg_context);
buf->msg_context = NULL;

+ if (buf->vcsm_handle) {
+ int ret;
+
+ pr_debug("%s: vc_sm_cma_free on handle %p\n", __func__,
+ buf->vcsm_handle);
+ ret = vc_sm_cma_free(buf->vcsm_handle);
+ if (ret)
+ pr_err("%s: vcsm_free failed, ret %d\n", __func__, ret);
+ buf->vcsm_handle = 0;
+ }
return 0;
}
EXPORT_SYMBOL_GPL(mmal_vchi_buffer_cleanup);
diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h
index e3ecd70b9951..9addd627ff5b 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h
+++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h
@@ -49,6 +49,7 @@ typedef void (*vchiq_mmal_buffer_cb)(

struct vchiq_mmal_port {
u32 enabled:1;
+ u32 zero_copy:1;
u32 handle;
u32 type; /* port type, cached to use on port info set */
u32 index; /* port index, cached to use on port info set */
--
2.37.3

2022-11-22 00:01:57

[permalink] [raw]

Subject: Re: [PATCH 08/14] uapi: bcm2835-isp: Add bcm2835-isp uapi header file

Hi Umang and Dave,

Thank you for the patch.

On Tue, Nov 22, 2022 at 03:17:16AM +0530, Umang Jain wrote:
> From: Dave Stevenson <[email protected]>
>
> This file defines the userland interface to the bcm2835-isp driver
> that will follow in a separate commit.
>
> Signed-off-by: Naushir Patuck <[email protected]>
> Signed-off-by: Dave Stevenson <[email protected]>
> Signed-off-by: Umang Jain <[email protected]>
> ---
> include/uapi/linux/bcm2835-isp.h | 320 +++++++++++++++++++++++++++++
> include/uapi/linux/v4l2-controls.h | 6 +
> 2 files changed, 326 insertions(+)
> create mode 100644 include/uapi/linux/bcm2835-isp.h
>
> diff --git a/include/uapi/linux/bcm2835-isp.h b/include/uapi/linux/bcm2835-isp.h
> new file mode 100644
> index 000000000000..cf8c0437f159
> --- /dev/null
> +++ b/include/uapi/linux/bcm2835-isp.h
> @@ -0,0 +1,320 @@
> +/* SPDX-License-Identifier: ((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) */
> +/*
> + * bcm2835-isp.h
> + *
> + * BCM2835 ISP driver - user space header file.
> + *
> + * Copyright © 2019-2020 Raspberry Pi (Trading) Ltd.
> + *
> + * Author: Naushir Patuck ([email protected])
> + *
> + */
> +
> +#ifndef __BCM2835_ISP_H_
> +#define __BCM2835_ISP_H_
> +
> +#include <linux/v4l2-controls.h>
> +
> +#define V4L2_CID_USER_BCM2835_ISP_CC_MATRIX \
> + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0001)
> +#define V4L2_CID_USER_BCM2835_ISP_LENS_SHADING \
> + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0002)
> +#define V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL \
> + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0003)
> +#define V4L2_CID_USER_BCM2835_ISP_GEQ \
> + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0004)
> +#define V4L2_CID_USER_BCM2835_ISP_GAMMA \
> + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0005)
> +#define V4L2_CID_USER_BCM2835_ISP_DENOISE \
> + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0006)
> +#define V4L2_CID_USER_BCM2835_ISP_SHARPEN \
> + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0007)
> +#define V4L2_CID_USER_BCM2835_ISP_DPC \
> + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0008)
> +
> +/*
> + * All structs below are directly mapped onto the equivalent structs in
> + * drivers/staging/vc04_services/vchiq-mmal/mmal-parameters.h
> + * for convenience.
> + */
> +
> +/**
> + * struct bcm2835_isp_rational - Rational value type.
> + *
> + * @num: Numerator.
> + * @den: Denominator.
> + */
> +struct bcm2835_isp_rational {
> + __s32 num;
> + __u32 den;
> +};
> +
> +/**
> + * struct bcm2835_isp_ccm - Colour correction matrix.
> + *
> + * @ccm: 3x3 correction matrix coefficients.
> + * @offsets: 1x3 correction offsets.
> + */
> +struct bcm2835_isp_ccm {
> + struct bcm2835_isp_rational ccm[3][3];
> + __s32 offsets[3];
> +};
> +
> +/**
> + * struct bcm2835_isp_custom_ccm - Custom CCM applied with the
> + * V4L2_CID_USER_BCM2835_ISP_CC_MATRIX ctrl.
> + *
> + * @enabled: Enable custom CCM.
> + * @ccm: Custom CCM coefficients and offsets.
> + */
> +struct bcm2835_isp_custom_ccm {
> + __u32 enabled;
> + struct bcm2835_isp_ccm ccm;
> +};
> +
> +/**
> + * enum bcm2835_isp_gain_format - format of the gains in the lens shading
> + * tables used with the
> + * V4L2_CID_USER_BCM2835_ISP_LENS_SHADING ctrl.
> + *
> + * @GAIN_FORMAT_U0P8_1: Gains are u0.8 format, starting at 1.0
> + * @GAIN_FORMAT_U1P7_0: Gains are u1.7 format, starting at 0.0
> + * @GAIN_FORMAT_U1P7_1: Gains are u1.7 format, starting at 1.0
> + * @GAIN_FORMAT_U2P6_0: Gains are u2.6 format, starting at 0.0
> + * @GAIN_FORMAT_U2P6_1: Gains are u2.6 format, starting at 1.0
> + * @GAIN_FORMAT_U3P5_0: Gains are u3.5 format, starting at 0.0
> + * @GAIN_FORMAT_U3P5_1: Gains are u3.5 format, starting at 1.0
> + * @GAIN_FORMAT_U4P10: Gains are u4.10 format, starting at 0.0
> + */
> +enum bcm2835_isp_gain_format {
> + GAIN_FORMAT_U0P8_1 = 0,
> + GAIN_FORMAT_U1P7_0 = 1,
> + GAIN_FORMAT_U1P7_1 = 2,
> + GAIN_FORMAT_U2P6_0 = 3,
> + GAIN_FORMAT_U2P6_1 = 4,
> + GAIN_FORMAT_U3P5_0 = 5,
> + GAIN_FORMAT_U3P5_1 = 6,
> + GAIN_FORMAT_U4P10 = 7,
> +};
> +
> +/**
> + * struct bcm2835_isp_lens_shading - Lens shading tables supplied with the
> + * V4L2_CID_USER_BCM2835_ISP_LENS_SHADING
> + * ctrl.
> + *
> + * @enabled: Enable lens shading.
> + * @grid_cell_size: Size of grid cells in samples (16, 32, 64, 128 or 256).
> + * @grid_width: Width of lens shading tables in grid cells.
> + * @grid_stride: Row to row distance (in grid cells) between grid cells
> + * in the same horizontal location.
> + * @grid_height: Height of lens shading tables in grid cells.
> + * @dmabuf: dmabuf file handle containing the table.
> + * @ref_transform: Reference transform - unsupported, please pass zero.
> + * @corner_sampled: Whether the gains are sampled at the corner points
> + * of the grid cells or in the cell centres.
> + * @gain_format: Format of the gains (see enum &bcm2835_isp_gain_format).
> + */
> +struct bcm2835_isp_lens_shading {
> + __u32 enabled;
> + __u32 grid_cell_size;
> + __u32 grid_width;
> + __u32 grid_stride;
> + __u32 grid_height;
> + __s32 dmabuf;
> + __u32 ref_transform;
> + __u32 corner_sampled;
> + __u32 gain_format;
> +};
> +
> +/**
> + * struct bcm2835_isp_black_level - Sensor black level set with the
> + * V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL ctrl.
> + *
> + * @enabled: Enable black level.
> + * @black_level_r: Black level for red channel.
> + * @black_level_g: Black level for green channels.
> + * @black_level_b: Black level for blue channel.
> + */
> +struct bcm2835_isp_black_level {
> + __u32 enabled;
> + __u16 black_level_r;
> + __u16 black_level_g;
> + __u16 black_level_b;
> + __u8 padding[2]; /* Unused */
> +};
> +
> +/**
> + * struct bcm2835_isp_geq - Green equalisation parameters set with the
> + * V4L2_CID_USER_BCM2835_ISP_GEQ ctrl.
> + *
> + * @enabled: Enable green equalisation.
> + * @offset: Fixed offset of the green equalisation threshold.
> + * @slope: Slope of the green equalisation threshold.
> + */
> +struct bcm2835_isp_geq {
> + __u32 enabled;
> + __u32 offset;
> + struct bcm2835_isp_rational slope;
> +};
> +
> +#define BCM2835_NUM_GAMMA_PTS 33
> +
> +/**
> + * struct bcm2835_isp_gamma - Gamma parameters set with the
> + * V4L2_CID_USER_BCM2835_ISP_GAMMA ctrl.
> + *
> + * @enabled: Enable gamma adjustment.
> + * @X: X values of the points defining the gamma curve.
> + * Values should be scaled to 16 bits.
> + * @Y: Y values of the points defining the gamma curve.
> + * Values should be scaled to 16 bits.
> + */
> +struct bcm2835_isp_gamma {
> + __u32 enabled;
> + __u16 x[BCM2835_NUM_GAMMA_PTS];
> + __u16 y[BCM2835_NUM_GAMMA_PTS];
> +};
> +
> +/**
> + * struct bcm2835_isp_denoise - Denoise parameters set with the
> + * V4L2_CID_USER_BCM2835_ISP_DENOISE ctrl.
> + *
> + * @enabled: Enable denoise.
> + * @constant: Fixed offset of the noise threshold.
> + * @slope: Slope of the noise threshold.
> + * @strength: Denoise strength between 0.0 (off) and 1.0 (maximum).
> + */
> +struct bcm2835_isp_denoise {
> + __u32 enabled;
> + __u32 constant;
> + struct bcm2835_isp_rational slope;
> + struct bcm2835_isp_rational strength;
> +};
> +
> +/**
> + * struct bcm2835_isp_sharpen - Sharpen parameters set with the
> + * V4L2_CID_USER_BCM2835_ISP_SHARPEN ctrl.
> + *
> + * @enabled: Enable sharpening.
> + * @threshold: Threshold at which to start sharpening pixels.
> + * @strength: Strength with which pixel sharpening increases.
> + * @limit: Limit to the amount of sharpening applied.
> + */
> +struct bcm2835_isp_sharpen {
> + __u32 enabled;
> + struct bcm2835_isp_rational threshold;
> + struct bcm2835_isp_rational strength;
> + struct bcm2835_isp_rational limit;
> +};
> +
> +/**
> + * enum bcm2835_isp_dpc_mode - defective pixel correction (DPC) strength.
> + *
> + * @DPC_MODE_OFF: No DPC.
> + * @DPC_MODE_NORMAL: Normal DPC.
> + * @DPC_MODE_STRONG: Strong DPC.
> + */
> +enum bcm2835_isp_dpc_mode {
> + DPC_MODE_OFF = 0,
> + DPC_MODE_NORMAL = 1,
> + DPC_MODE_STRONG = 2,
> +};
> +
> +/**
> + * struct bcm2835_isp_dpc - Defective pixel correction (DPC) parameters set
> + * with the V4L2_CID_USER_BCM2835_ISP_DPC ctrl.
> + *
> + * @enabled: Enable DPC.
> + * @strength: DPC strength (see enum &bcm2835_isp_dpc_mode).
> + */
> +struct bcm2835_isp_dpc {
> + __u32 enabled;
> + __u32 strength;
> +};
> +
> +/*
> + * ISP statistics structures.
> + *
> + * The bcm2835_isp_stats structure is generated at the output of the
> + * statistics node. Note that this does not directly map onto the statistics
> + * output of the ISP HW. Instead, the MMAL firmware code maps the HW statistics
> + * to the bcm2835_isp_stats structure.
> + */
> +#define DEFAULT_AWB_REGIONS_X 16
> +#define DEFAULT_AWB_REGIONS_Y 12
> +
> +#define NUM_HISTOGRAMS 2
> +#define NUM_HISTOGRAM_BINS 128
> +#define AWB_REGIONS (DEFAULT_AWB_REGIONS_X * DEFAULT_AWB_REGIONS_Y)
> +#define FLOATING_REGIONS 16
> +#define AGC_REGIONS 16
> +#define FOCUS_REGIONS 12
> +
> +/**
> + * struct bcm2835_isp_stats_hist - Histogram statistics
> + *
> + * @r_hist: Red channel histogram.
> + * @g_hist: Combined green channel histogram.
> + * @b_hist: Blue channel histogram.
> + */
> +struct bcm2835_isp_stats_hist {
> + __u32 r_hist[NUM_HISTOGRAM_BINS];
> + __u32 g_hist[NUM_HISTOGRAM_BINS];
> + __u32 b_hist[NUM_HISTOGRAM_BINS];
> +};
> +
> +/**
> + * struct bcm2835_isp_stats_region - Region sums.
> + *
> + * @counted: The number of 2x2 bayer tiles accumulated.
> + * @notcounted: The number of 2x2 bayer tiles not accumulated.
> + * @r_sum: Total sum of counted pixels in the red channel for a region.
> + * @g_sum: Total sum of counted pixels in the green channel for a region.
> + * @b_sum: Total sum of counted pixels in the blue channel for a region.
> + */
> +struct bcm2835_isp_stats_region {
> + __u32 counted;
> + __u32 notcounted;
> + __u64 r_sum;
> + __u64 g_sum;
> + __u64 b_sum;
> +};
> +
> +/**
> + * struct bcm2835_isp_stats_focus - Focus statistics.
> + *
> + * @contrast_val: Focus measure - accumulated output of the focus filter.
> + * In the first dimension, index [0] counts pixels below a
> + * preset threshold, and index [1] counts pixels above the
> + * threshold. In the second dimension, index [0] uses the
> + * first predefined filter, and index [1] uses the second
> + * predefined filter.
> + * @contrast_val_num: The number of counted pixels in the above accumulation.
> + */
> +struct bcm2835_isp_stats_focus {
> + __u64 contrast_val[2][2];
> + __u32 contrast_val_num[2][2];
> +};
> +
> +/**
> + * struct bcm2835_isp_stats - ISP statistics.
> + *
> + * @version: Version of the bcm2835_isp_stats structure.
> + * @size: Size of the bcm2835_isp_stats structure.
> + * @hist: Histogram statistics for the entire image.
> + * @awb_stats: Statistics for the regions defined for AWB calculations.
> + * @floating_stats: Statistics for arbitrarily placed (floating) regions.
> + * @agc_stats: Statistics for the regions defined for AGC calculations.
> + * @focus_stats: Focus filter statistics for the focus regions.
> + */
> +struct bcm2835_isp_stats {
> + __u32 version;
> + __u32 size;
> + struct bcm2835_isp_stats_hist hist[NUM_HISTOGRAMS];
> + struct bcm2835_isp_stats_region awb_stats[AWB_REGIONS];
> + struct bcm2835_isp_stats_region floating_stats[FLOATING_REGIONS];
> + struct bcm2835_isp_stats_region agc_stats[AGC_REGIONS];
> + struct bcm2835_isp_stats_focus focus_stats[FOCUS_REGIONS];
> +};
> +
> +#endif /* __BCM2835_ISP_H_ */
> diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h
> index b5e7d082b8ad..31606d42ec58 100644
> --- a/include/uapi/linux/v4l2-controls.h
> +++ b/include/uapi/linux/v4l2-controls.h
> @@ -217,6 +217,7 @@ enum v4l2_colorfx {
> * The base for Allegro driver controls.
> * We reserve 16 controls for this driver.
> */
> +

Unrelated change.

> #define V4L2_CID_USER_ALLEGRO_BASE (V4L2_CID_USER_BASE + 0x1170)
>
> /*
> @@ -231,6 +232,11 @@ enum v4l2_colorfx {
> */
> #define V4L2_CID_USER_DW100_BASE (V4L2_CID_USER_BASE + 0x1190)
>
> +/* The base for the bcm2835-isp driver controls.

/*
* The base for the bcm2835-isp driver controls.

> + * We reserve 16 controls for this driver.
> + */
> +#define V4L2_CID_USER_BCM2835_ISP_BASE (V4L2_CID_USER_BASE + 0x11a0)

This will be interesting to handle in libcamera, as it breaks the ABI
with the downstream driver.

> +
> /* MPEG-class control IDs */
> /* The MPEG controls are applicable to all codec controls
> * and the 'MPEG' part of the define is historical */

--
Regards,

Laurent Pinchart

2022-11-22 00:29:31

[permalink] [raw]

Subject: Re: [PATCH 12/14] staging: vc04_services: bcm2835_isp: Allow multiple users

On Tue, Nov 22, 2022 at 01:29:48AM +0200, Laurent Pinchart wrote:
> Hi Umang and Naush,
>
> Thank you for the patch.
>
> On Tue, Nov 22, 2022 at 03:17:20AM +0530, Umang Jain wrote:
> > From: Naushir Patuck <[email protected]>
> >
> > Add a second (identical) set of device nodes to allow concurrent use of
> > the bcm2835-isp hardware by another user. This change effectively
> > creates a second state structure (struct bcm2835_isp_dev) to maintain
> > independent state for the second user. Node and media entity names are
> > appened with the instance index appropriately.
> >
> > Further users can be added by changing the BCM2835_ISP_NUM_INSTANCES
> > define.
> >
> > Signed-off-by: Naushir Patuck <[email protected]>
> > Signed-off-by: Umang Jain <[email protected]>
>
> Let's squash this with 09/14.

Also, the documentation in 13/14 should be updated.

> > ---
> > .../bcm2835-isp/bcm2835-v4l2-isp.c | 77 +++++++++++++++----
> > 1 file changed, 61 insertions(+), 16 deletions(-)
> >
> > diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
> > index cb7cdba76682..0dbcb25595e7 100644
> > --- a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
> > +++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
> > @@ -28,13 +28,19 @@
> >
> > MODULE_IMPORT_NS(DMA_BUF);
> >
> > +/*
> > + * We want to instantiate 2 independent instances allowing 2 simultaneous users
> > + * of the ISP hardware.
> > + */
> > +#define BCM2835_ISP_NUM_INSTANCES 2
> > +
> > static unsigned int debug;
> > module_param(debug, uint, 0644);
> > MODULE_PARM_DESC(debug, "activates debug info");
> >
> > -static unsigned int video_nr = 13;
> > -module_param(video_nr, uint, 0644);
> > -MODULE_PARM_DESC(video_nr, "base video device number");
> > +static unsigned int video_nr[BCM2835_ISP_NUM_INSTANCES] = { 13, 20 };
> > +module_param_array(video_nr, uint, NULL, 0644);
> > +MODULE_PARM_DESC(video_nr, "base video device numbers");
> >
> > #define BCM2835_ISP_NAME "bcm2835-isp"
> > #define BCM2835_ISP_ENTITY_NAME_LEN 32
> > @@ -1286,6 +1292,7 @@ static int bcm2835_isp_get_supported_fmts(struct bcm2835_isp_node *node)
> > * or output nodes.
> > */
> > static int register_node(struct bcm2835_isp_dev *dev,
> > + unsigned int instance,
> > struct bcm2835_isp_node *node,
> > int index)
> > {
> > @@ -1447,7 +1454,7 @@ static int register_node(struct bcm2835_isp_dev *dev,
> > snprintf(vfd->name, sizeof(node->vfd.name), "%s-%s%d", BCM2835_ISP_NAME,
> > node->name, node->id);
> >
> > - ret = video_register_device(vfd, VFL_TYPE_VIDEO, video_nr + index);
> > + ret = video_register_device(vfd, VFL_TYPE_VIDEO, video_nr[instance]);
> > if (ret) {
> > v4l2_err(&dev->v4l2_dev,
> > "Failed to register video %s[%d] device node\n",
> > @@ -1668,9 +1675,8 @@ static int media_controller_register(struct bcm2835_isp_dev *dev)
> > return ret;
> > }
> >
> > -static int bcm2835_isp_remove(struct platform_device *pdev)
> > +static void bcm2835_isp_remove_instance(struct bcm2835_isp_dev *dev)
> > {
> > - struct bcm2835_isp_dev *dev = platform_get_drvdata(pdev);
> > unsigned int i;
> >
> > media_controller_unregister(dev);
> > @@ -1685,11 +1691,11 @@ static int bcm2835_isp_remove(struct platform_device *pdev)
> > dev->component);
> >
> > vchiq_mmal_finalise(dev->mmal_instance);
> > -
> > - return 0;
> > }
> >
> > -static int bcm2835_isp_probe(struct platform_device *pdev)
> > +static int bcm2835_isp_probe_instance(struct platform_device *pdev,
> > + struct bcm2835_isp_dev **dev_int,
> > + unsigned int instance)
> > {
> > struct bcm2835_isp_dev *dev;
> > unsigned int i;
> > @@ -1699,6 +1705,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
> > if (!dev)
> > return -ENOMEM;
> >
> > + *dev_int = dev;
> > dev->dev = &pdev->dev;
> >
> > ret = v4l2_device_register(&pdev->dev, &dev->v4l2_dev);
> > @@ -1716,7 +1723,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
> > if (ret) {
> > v4l2_err(&dev->v4l2_dev,
> > "%s: failed to create ril.isp component\n", __func__);
> > - goto error;
> > + return ret;
> > }
> >
> > if (dev->component->inputs < BCM2835_ISP_NUM_OUTPUTS ||
> > @@ -1728,7 +1735,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
> > BCM2835_ISP_NUM_OUTPUTS,
> > dev->component->outputs,
> > BCM2835_ISP_NUM_CAPTURES + BCM2835_ISP_NUM_METADATA);
> > - goto error;
> > + return -EINVAL;
> > }
> >
> > atomic_set(&dev->num_streaming, 0);
> > @@ -1736,17 +1743,55 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
> > for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) {
> > struct bcm2835_isp_node *node = &dev->node[i];
> >
> > - ret = register_node(dev, node, i);
> > + ret = register_node(dev, instance, node, i);
> > if (ret)
> > - goto error;
> > + return ret;
> > }
> >
> > ret = media_controller_register(dev);
> > if (ret)
> > - goto error;
> > + return ret;
> > +
> > + return 0;
> > +}
> > +
> > +static int bcm2835_isp_remove(struct platform_device *pdev)
> > +{
> > + struct bcm2835_isp_dev **bcm2835_isp_instances;
> > + unsigned int i;
> > +
> > + bcm2835_isp_instances = platform_get_drvdata(pdev);
> > + for (i = 0; i < BCM2835_ISP_NUM_INSTANCES; i++) {
> > + if (bcm2835_isp_instances[i])
> > + bcm2835_isp_remove_instance(bcm2835_isp_instances[i]);
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static int bcm2835_isp_probe(struct platform_device *pdev)
> > +{
> > + struct bcm2835_isp_dev **bcm2835_isp_instances;
> > + unsigned int i;
> > + int ret;
> > +
> > + bcm2835_isp_instances = devm_kzalloc(&pdev->dev,
> > + sizeof(bcm2835_isp_instances) *
> > + BCM2835_ISP_NUM_INSTANCES,
> > + GFP_KERNEL);
> > + if (!bcm2835_isp_instances)
> > + return -ENOMEM;
> > +
> > + platform_set_drvdata(pdev, bcm2835_isp_instances);
> > +
> > + for (i = 0; i < BCM2835_ISP_NUM_INSTANCES; i++) {
> > + ret = bcm2835_isp_probe_instance(pdev,
> > + &bcm2835_isp_instances[i], i);
> > + if (ret)
> > + goto error;
> > + }
> >
> > - platform_set_drvdata(pdev, dev);
> > - v4l2_info(&dev->v4l2_dev, "Loaded V4L2 %s\n", BCM2835_ISP_NAME);
> > + dev_info(&pdev->dev, "Loaded V4L2 %s\n", BCM2835_ISP_NAME);
> > return 0;
> >
> > error:

--
Regards,

Laurent Pinchart

2022-11-22 00:30:57

[permalink] [raw]

Subject: Re: [PATCH 02/14] staging: vchiq_arm: Register vcsm-cma as a platform driver

Hi Umang and Dave,

Thank you for the patch.

On Tue, Nov 22, 2022 at 03:17:10AM +0530, Umang Jain wrote:
> From: Dave Stevenson <[email protected]>
>
> Following the same pattern as bcm2835-camera and bcm2835-audio,
> register the vcsm-cma driver as a platform driver.
>
> Signed-off-by: Dave Stevenson <[email protected]>
> Signed-off-by: Umang Jain <[email protected]>

Reviewed-by: Laurent Pinchart <[email protected]>

> ---
> drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c b/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
> index dc33490ba7fb..642fdbc0d654 100644
> --- a/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
> +++ b/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
> @@ -67,6 +67,7 @@ struct vchiq_state g_state;
>
> static struct platform_device *bcm2835_camera;
> static struct platform_device *bcm2835_audio;
> +static struct platform_device *vcsm_cma;
>
> struct vchiq_drvdata {
> const unsigned int cache_line_size;
> @@ -1832,6 +1833,7 @@ static int vchiq_probe(struct platform_device *pdev)
> goto error_exit;
> }
>
> + vcsm_cma = vchiq_register_child(pdev, "vcsm-cma");
> bcm2835_camera = vchiq_register_child(pdev, "bcm2835-camera");
> bcm2835_audio = vchiq_register_child(pdev, "bcm2835_audio");
>
> @@ -1847,6 +1849,7 @@ static int vchiq_remove(struct platform_device *pdev)
> {
> platform_device_unregister(bcm2835_audio);
> platform_device_unregister(bcm2835_camera);
> + platform_device_unregister(vcsm_cma);
> vchiq_debugfs_deinit();
> vchiq_deregister_chrdev();
>

--
Regards,

Laurent Pinchart

2022-11-22 00:38:20

[permalink] [raw]

Subject: Re: [PATCH 12/14] staging: vc04_services: bcm2835_isp: Allow multiple users

Hi Umang and Naush,

Thank you for the patch.

On Tue, Nov 22, 2022 at 03:17:20AM +0530, Umang Jain wrote:
> From: Naushir Patuck <[email protected]>
>
> Add a second (identical) set of device nodes to allow concurrent use of
> the bcm2835-isp hardware by another user. This change effectively
> creates a second state structure (struct bcm2835_isp_dev) to maintain
> independent state for the second user. Node and media entity names are
> appened with the instance index appropriately.
>
> Further users can be added by changing the BCM2835_ISP_NUM_INSTANCES
> define.
>
> Signed-off-by: Naushir Patuck <[email protected]>
> Signed-off-by: Umang Jain <[email protected]>

Let's squash this with 09/14.

> ---
> .../bcm2835-isp/bcm2835-v4l2-isp.c | 77 +++++++++++++++----
> 1 file changed, 61 insertions(+), 16 deletions(-)
>
> diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
> index cb7cdba76682..0dbcb25595e7 100644
> --- a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
> +++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
> @@ -28,13 +28,19 @@
>
> MODULE_IMPORT_NS(DMA_BUF);
>
> +/*
> + * We want to instantiate 2 independent instances allowing 2 simultaneous users
> + * of the ISP hardware.
> + */
> +#define BCM2835_ISP_NUM_INSTANCES 2
> +
> static unsigned int debug;
> module_param(debug, uint, 0644);
> MODULE_PARM_DESC(debug, "activates debug info");
>
> -static unsigned int video_nr = 13;
> -module_param(video_nr, uint, 0644);
> -MODULE_PARM_DESC(video_nr, "base video device number");
> +static unsigned int video_nr[BCM2835_ISP_NUM_INSTANCES] = { 13, 20 };
> +module_param_array(video_nr, uint, NULL, 0644);
> +MODULE_PARM_DESC(video_nr, "base video device numbers");
>
> #define BCM2835_ISP_NAME "bcm2835-isp"
> #define BCM2835_ISP_ENTITY_NAME_LEN 32
> @@ -1286,6 +1292,7 @@ static int bcm2835_isp_get_supported_fmts(struct bcm2835_isp_node *node)
> * or output nodes.
> */
> static int register_node(struct bcm2835_isp_dev *dev,
> + unsigned int instance,
> struct bcm2835_isp_node *node,
> int index)
> {
> @@ -1447,7 +1454,7 @@ static int register_node(struct bcm2835_isp_dev *dev,
> snprintf(vfd->name, sizeof(node->vfd.name), "%s-%s%d", BCM2835_ISP_NAME,
> node->name, node->id);
>
> - ret = video_register_device(vfd, VFL_TYPE_VIDEO, video_nr + index);
> + ret = video_register_device(vfd, VFL_TYPE_VIDEO, video_nr[instance]);
> if (ret) {
> v4l2_err(&dev->v4l2_dev,
> "Failed to register video %s[%d] device node\n",
> @@ -1668,9 +1675,8 @@ static int media_controller_register(struct bcm2835_isp_dev *dev)
> return ret;
> }
>
> -static int bcm2835_isp_remove(struct platform_device *pdev)
> +static void bcm2835_isp_remove_instance(struct bcm2835_isp_dev *dev)
> {
> - struct bcm2835_isp_dev *dev = platform_get_drvdata(pdev);
> unsigned int i;
>
> media_controller_unregister(dev);
> @@ -1685,11 +1691,11 @@ static int bcm2835_isp_remove(struct platform_device *pdev)
> dev->component);
>
> vchiq_mmal_finalise(dev->mmal_instance);
> -
> - return 0;
> }
>
> -static int bcm2835_isp_probe(struct platform_device *pdev)
> +static int bcm2835_isp_probe_instance(struct platform_device *pdev,
> + struct bcm2835_isp_dev **dev_int,
> + unsigned int instance)
> {
> struct bcm2835_isp_dev *dev;
> unsigned int i;
> @@ -1699,6 +1705,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
> if (!dev)
> return -ENOMEM;
>
> + *dev_int = dev;
> dev->dev = &pdev->dev;
>
> ret = v4l2_device_register(&pdev->dev, &dev->v4l2_dev);
> @@ -1716,7 +1723,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
> if (ret) {
> v4l2_err(&dev->v4l2_dev,
> "%s: failed to create ril.isp component\n", __func__);
> - goto error;
> + return ret;
> }
>
> if (dev->component->inputs < BCM2835_ISP_NUM_OUTPUTS ||
> @@ -1728,7 +1735,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
> BCM2835_ISP_NUM_OUTPUTS,
> dev->component->outputs,
> BCM2835_ISP_NUM_CAPTURES + BCM2835_ISP_NUM_METADATA);
> - goto error;
> + return -EINVAL;
> }
>
> atomic_set(&dev->num_streaming, 0);
> @@ -1736,17 +1743,55 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
> for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) {
> struct bcm2835_isp_node *node = &dev->node[i];
>
> - ret = register_node(dev, node, i);
> + ret = register_node(dev, instance, node, i);
> if (ret)
> - goto error;
> + return ret;
> }
>
> ret = media_controller_register(dev);
> if (ret)
> - goto error;
> + return ret;
> +
> + return 0;
> +}
> +
> +static int bcm2835_isp_remove(struct platform_device *pdev)
> +{
> + struct bcm2835_isp_dev **bcm2835_isp_instances;
> + unsigned int i;
> +
> + bcm2835_isp_instances = platform_get_drvdata(pdev);
> + for (i = 0; i < BCM2835_ISP_NUM_INSTANCES; i++) {
> + if (bcm2835_isp_instances[i])
> + bcm2835_isp_remove_instance(bcm2835_isp_instances[i]);
> + }
> +
> + return 0;
> +}
> +
> +static int bcm2835_isp_probe(struct platform_device *pdev)
> +{
> + struct bcm2835_isp_dev **bcm2835_isp_instances;
> + unsigned int i;
> + int ret;
> +
> + bcm2835_isp_instances = devm_kzalloc(&pdev->dev,
> + sizeof(bcm2835_isp_instances) *
> + BCM2835_ISP_NUM_INSTANCES,
> + GFP_KERNEL);
> + if (!bcm2835_isp_instances)
> + return -ENOMEM;
> +
> + platform_set_drvdata(pdev, bcm2835_isp_instances);
> +
> + for (i = 0; i < BCM2835_ISP_NUM_INSTANCES; i++) {
> + ret = bcm2835_isp_probe_instance(pdev,
> + &bcm2835_isp_instances[i], i);
> + if (ret)
> + goto error;
> + }
>
> - platform_set_drvdata(pdev, dev);
> - v4l2_info(&dev->v4l2_dev, "Loaded V4L2 %s\n", BCM2835_ISP_NAME);
> + dev_info(&pdev->dev, "Loaded V4L2 %s\n", BCM2835_ISP_NAME);
> return 0;
>
> error:

--
Regards,

Laurent Pinchart

2022-11-22 12:42:06

[permalink] [raw]

Subject: Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support

Hi Umang and Laurent

On Mon, 21 Nov 2022 at 22:16, Laurent Pinchart
<[email protected]> wrote:
>
> Hi Umang,
>
> Nice to see this series on the list !
>
> On Tue, Nov 22, 2022 at 03:17:08AM +0530, Umang Jain wrote:
> > This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> > independent subset of earlier series [2] posted to upport CSI-2/CCP2
> > receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> > SoCs (namely BCM2711). Unicam is still under active development to work
> > with multistream support to get into mainline. Hence only the ISP driver
> > will remain the primary area of this series.
> >
> > Patch (01-02)/14 adds a new driver named vc-sm-cma to handle memory sharing
> > with the VC4 VPU.
> >
> > Patch 03/14 adds a small extension to videobuf2 to allow exporting as a
> > dma_buf instead of a file-descriptor.
> >
> > Patch (04-05)/14 adds a couple of improvements/support for
> > bcm2835-isp(event callback and zero-copy) to vchiq-mmal.
> >
> > Patch (06-09)/14 adds the core bcm2835-isp driver along with headers
> > and format defintions.
> >
> > Patch (10-11)/14 deals with the colorspace support.
> > Note: This is still WIP since the implementation of colorspace is still
> > getting ironed out (especially around setting of colorspace flags handling).
> >
> > Patch 12/14 allows multiple instances of the ISP.
> >
> > Patch 13/14 adds a admin-guide document on bcm2835-isp.
> >
> > Patch 14/14 wires all this up with the vchiq-mmal driver.
> >
> > Testing:
> > Tested with RPi Model 4B running linux mainline v6.1-rc6. To test
> > end-to-end, I choose to cherry-pick the Unicam patches and OV5647 DT
> > bindings from [1]). Once done, functional testing was conducted with
> > libcamera[3] and its utility tools.
> >
> > Also note: Reviews given on [2] for the relevant ISP driver patches have
> > been incorporated in this version.
> >
> > Known issues:
> > - Colorspace handling
>
> This will require further discussions, I'll try to comment on this topic
> in the review of the ISP driver patch.
>
> > - vc-sm-cma spamming kernel log with
> > - pr_err("%s: Expecting an uncached alias for dma_addr %pad\n"
>
> Do you have any plan to address this ? Is the root cause known ?

You've picked up an old version of the downstream driver.
Pi0&1 share the VPU L2 cache with the ARM in the architecture, so they
use the 0x8 cache alias.
See https://github.com/raspberrypi/linux/commit/e22927f8ec9dc87772ac61d6aba00dc8046b4f49

Dave

> > [1]: https://github.com/raspberrypi/linux
> > [2]: https://lore.kernel.org/linux-media/[email protected]/
> > [3]: https://libcamera.org/getting-started.html
> >
> > Dave Stevenson (7):
> > staging: vc04_services: Add new vc-sm-cma driver
> > staging: vchiq_arm: Register vcsm-cma as a platform driver
> > media: videobuf2: Allow exporting of a struct dmabuf
> > staging: mmal-vchiq: Add support for event callbacks
> > staging: mmal-vchiq: Use vc-sm-cma to support zero copy
> > staging: mmal_vchiq: Add image formats to be used by bcm2835-isp
> > uapi: bcm2835-isp: Add bcm2835-isp uapi header file
> >
> > David Plowman (2):
> > vc04_services: bcm2835-isp: Allow formats with different colour spaces
> > vc04_services: bcm2835-isp: Permit all sRGB colour spaces on ISP
> > outputs
> >
> > Naushir Patuck (5):
> > media: uapi: v4l2-core: Add ISP statistics output V4L2 fourcc type
> > staging: vc04_services: bcm2835-isp: Add a more complex ISP processing
> > component
> > staging: vc04_services: bcm2835_isp: Allow multiple users
> > docs: admin-guide: media: bcm2835-isp: Add documentation for
> > bcm2835-isp
> > staging: vc04_services: vchiq: Load bcm2835_isp driver from vchiq
> >
> > .../admin-guide/media/bcm2835-isp.rst | 127 ++
> > .../userspace-api/media/drivers/index.rst | 1 +
> > .../userspace-api/media/v4l/meta-formats.rst | 1 +
> > .../v4l/pixfmt-meta-bcm2835-isp-stats.rst | 41 +
> > MAINTAINERS | 9 +
> > .../media/common/videobuf2/videobuf2-core.c | 36 +-
> > drivers/media/v4l2-core/v4l2-ioctl.c | 1 +
> > drivers/staging/vc04_services/Kconfig | 4 +
> > drivers/staging/vc04_services/Makefile | 2 +
> > .../staging/vc04_services/bcm2835-isp/Kconfig | 14 +
> > .../vc04_services/bcm2835-isp/Makefile | 8 +
> > .../bcm2835-isp/bcm2835-isp-ctrls.h | 72 +
> > .../bcm2835-isp/bcm2835-isp-fmts.h | 558 +++++
> > .../bcm2835-isp/bcm2835-v4l2-isp.c | 1817 +++++++++++++++++
> > .../interface/vchiq_arm/vchiq_arm.c | 6 +
> > .../staging/vc04_services/vc-sm-cma/Kconfig | 10 +
> > .../staging/vc04_services/vc-sm-cma/Makefile | 12 +
> > .../staging/vc04_services/vc-sm-cma/vc_sm.c | 801 ++++++++
> > .../staging/vc04_services/vc-sm-cma/vc_sm.h | 54 +
> > .../vc04_services/vc-sm-cma/vc_sm_cma_vchi.c | 507 +++++
> > .../vc04_services/vc-sm-cma/vc_sm_cma_vchi.h | 63 +
> > .../vc04_services/vc-sm-cma/vc_sm_defs.h | 187 ++
> > .../vc04_services/vc-sm-cma/vc_sm_knl.h | 28 +
> > .../staging/vc04_services/vchiq-mmal/Kconfig | 1 +
> > .../vc04_services/vchiq-mmal/mmal-common.h | 5 +
> > .../vc04_services/vchiq-mmal/mmal-encodings.h | 66 +
> > .../vc04_services/vchiq-mmal/mmal-msg.h | 35 +
> > .../vchiq-mmal/mmal-parameters.h | 165 +-
> > .../vc04_services/vchiq-mmal/mmal-vchiq.c | 253 ++-
> > .../vc04_services/vchiq-mmal/mmal-vchiq.h | 5 +
> > include/media/videobuf2-core.h | 15 +
> > include/uapi/linux/bcm2835-isp.h | 347 ++++
> > include/uapi/linux/v4l2-controls.h | 6 +
> > include/uapi/linux/videodev2.h | 1 +
> > 34 files changed, 5225 insertions(+), 33 deletions(-)
> > create mode 100644 Documentation/admin-guide/media/bcm2835-isp.rst
> > create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst
> > create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Kconfig
> > create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Makefile
> > create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h
> > create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
> > create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
> > create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Kconfig
> > create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Makefile
> > create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.c
> > create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.h
> > create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c
> > create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h
> > create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h
> > create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h
> > create mode 100644 include/uapi/linux/bcm2835-isp.h
>
> --
> Regards,
>
> Laurent Pinchart

2022-11-22 13:22:34

[permalink] [raw]

Subject: Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support

Hi Dave,

On 11/22/22 5:12 PM, Dave Stevenson wrote:
> Hi Umang and Laurent
>
> On Mon, 21 Nov 2022 at 22:16, Laurent Pinchart
> <[email protected]> wrote:
>> Hi Umang,
>>
>> Nice to see this series on the list !
>>
>> On Tue, Nov 22, 2022 at 03:17:08AM +0530, Umang Jain wrote:
>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
>>> SoCs (namely BCM2711). Unicam is still under active development to work
>>> with multistream support to get into mainline. Hence only the ISP driver
>>> will remain the primary area of this series.
>>>
>>> Patch (01-02)/14 adds a new driver named vc-sm-cma to handle memory sharing
>>> with the VC4 VPU.
>>>
>>> Patch 03/14 adds a small extension to videobuf2 to allow exporting as a
>>> dma_buf instead of a file-descriptor.
>>>
>>> Patch (04-05)/14 adds a couple of improvements/support for
>>> bcm2835-isp(event callback and zero-copy) to vchiq-mmal.
>>>
>>> Patch (06-09)/14 adds the core bcm2835-isp driver along with headers
>>> and format defintions.
>>>
>>> Patch (10-11)/14 deals with the colorspace support.
>>> Note: This is still WIP since the implementation of colorspace is still
>>> getting ironed out (especially around setting of colorspace flags handling).
>>>
>>> Patch 12/14 allows multiple instances of the ISP.
>>>
>>> Patch 13/14 adds a admin-guide document on bcm2835-isp.
>>>
>>> Patch 14/14 wires all this up with the vchiq-mmal driver.
>>>
>>> Testing:
>>> Tested with RPi Model 4B running linux mainline v6.1-rc6. To test
>>> end-to-end, I choose to cherry-pick the Unicam patches and OV5647 DT
>>> bindings from [1]). Once done, functional testing was conducted with
>>> libcamera[3] and its utility tools.
>>>
>>> Also note: Reviews given on [2] for the relevant ISP driver patches have
>>> been incorporated in this version.
>>>
>>> Known issues:
>>> - Colorspace handling
>> This will require further discussions, I'll try to comment on this topic
>> in the review of the ISP driver patch.
>>
>>> - vc-sm-cma spamming kernel log with
>>> - pr_err("%s: Expecting an uncached alias for dma_addr %pad\n"
>> Do you have any plan to address this ? Is the root cause known ?
> You've picked up an old version of the downstream driver.

I didn't pick version of driver rather, I picked commits that were
leading to the rpi-5.15.y branch of the downstream kernel

> Pi0&1 share the VPU L2 cache with the ARM in the architecture, so they
> use the 0x8 cache alias.
> See https://github.com/raspberrypi/linux/commit/e22927f8ec9dc87772ac61d6aba00dc8046b4f49

And certainly I missed to notice / pick this commit out on the vc-sm-cma
driver.
Thanks for providing the link! I'll squash it for v2.
>
> Dave
>
>>> [1]: https://github.com/raspberrypi/linux
>>> [2]: https://lore.kernel.org/linux-media/[email protected]/
>>> [3]: https://libcamera.org/getting-started.html
>>>
>>> Dave Stevenson (7):
>>> staging: vc04_services: Add new vc-sm-cma driver
>>> staging: vchiq_arm: Register vcsm-cma as a platform driver
>>> media: videobuf2: Allow exporting of a struct dmabuf
>>> staging: mmal-vchiq: Add support for event callbacks
>>> staging: mmal-vchiq: Use vc-sm-cma to support zero copy
>>> staging: mmal_vchiq: Add image formats to be used by bcm2835-isp
>>> uapi: bcm2835-isp: Add bcm2835-isp uapi header file
>>>
>>> David Plowman (2):
>>> vc04_services: bcm2835-isp: Allow formats with different colour spaces
>>> vc04_services: bcm2835-isp: Permit all sRGB colour spaces on ISP
>>> outputs
>>>
>>> Naushir Patuck (5):
>>> media: uapi: v4l2-core: Add ISP statistics output V4L2 fourcc type
>>> staging: vc04_services: bcm2835-isp: Add a more complex ISP processing
>>> component
>>> staging: vc04_services: bcm2835_isp: Allow multiple users
>>> docs: admin-guide: media: bcm2835-isp: Add documentation for
>>> bcm2835-isp
>>> staging: vc04_services: vchiq: Load bcm2835_isp driver from vchiq
>>>
>>> .../admin-guide/media/bcm2835-isp.rst | 127 ++
>>> .../userspace-api/media/drivers/index.rst | 1 +
>>> .../userspace-api/media/v4l/meta-formats.rst | 1 +
>>> .../v4l/pixfmt-meta-bcm2835-isp-stats.rst | 41 +
>>> MAINTAINERS | 9 +
>>> .../media/common/videobuf2/videobuf2-core.c | 36 +-
>>> drivers/media/v4l2-core/v4l2-ioctl.c | 1 +
>>> drivers/staging/vc04_services/Kconfig | 4 +
>>> drivers/staging/vc04_services/Makefile | 2 +
>>> .../staging/vc04_services/bcm2835-isp/Kconfig | 14 +
>>> .../vc04_services/bcm2835-isp/Makefile | 8 +
>>> .../bcm2835-isp/bcm2835-isp-ctrls.h | 72 +
>>> .../bcm2835-isp/bcm2835-isp-fmts.h | 558 +++++
>>> .../bcm2835-isp/bcm2835-v4l2-isp.c | 1817 +++++++++++++++++
>>> .../interface/vchiq_arm/vchiq_arm.c | 6 +
>>> .../staging/vc04_services/vc-sm-cma/Kconfig | 10 +
>>> .../staging/vc04_services/vc-sm-cma/Makefile | 12 +
>>> .../staging/vc04_services/vc-sm-cma/vc_sm.c | 801 ++++++++
>>> .../staging/vc04_services/vc-sm-cma/vc_sm.h | 54 +
>>> .../vc04_services/vc-sm-cma/vc_sm_cma_vchi.c | 507 +++++
>>> .../vc04_services/vc-sm-cma/vc_sm_cma_vchi.h | 63 +
>>> .../vc04_services/vc-sm-cma/vc_sm_defs.h | 187 ++
>>> .../vc04_services/vc-sm-cma/vc_sm_knl.h | 28 +
>>> .../staging/vc04_services/vchiq-mmal/Kconfig | 1 +
>>> .../vc04_services/vchiq-mmal/mmal-common.h | 5 +
>>> .../vc04_services/vchiq-mmal/mmal-encodings.h | 66 +
>>> .../vc04_services/vchiq-mmal/mmal-msg.h | 35 +
>>> .../vchiq-mmal/mmal-parameters.h | 165 +-
>>> .../vc04_services/vchiq-mmal/mmal-vchiq.c | 253 ++-
>>> .../vc04_services/vchiq-mmal/mmal-vchiq.h | 5 +
>>> include/media/videobuf2-core.h | 15 +
>>> include/uapi/linux/bcm2835-isp.h | 347 ++++
>>> include/uapi/linux/v4l2-controls.h | 6 +
>>> include/uapi/linux/videodev2.h | 1 +
>>> 34 files changed, 5225 insertions(+), 33 deletions(-)
>>> create mode 100644 Documentation/admin-guide/media/bcm2835-isp.rst
>>> create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst
>>> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Kconfig
>>> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Makefile
>>> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h
>>> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
>>> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
>>> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Kconfig
>>> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Makefile
>>> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.c
>>> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.h
>>> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c
>>> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h
>>> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h
>>> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h
>>> create mode 100644 include/uapi/linux/bcm2835-isp.h
>> --
>> Regards,
>>
>> Laurent Pinchart

2022-11-26 15:45:47

[permalink] [raw]

Subject: Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support

Hi Umang,

Am 21.11.22 um 22:47 schrieb Umang Jain:
> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> SoCs (namely BCM2711). Unicam is still under active development to work
> with multistream support to get into mainline. Hence only the ISP driver
> will remain the primary area of this series.

thanks for working on this. But honestly i would prefer that vchiq comes
out of staging before adding more features. As Greg said some time ago
staging is not a place to "dump code and run away". These new files are
in the same bad shape as the rest of vc04 before the clean-up here in
staging started.

I agree that VCSM is on the TODO list for vchiq, but this driver is not
necessary for making bcm2835-audio & bcm2835-camera leave staging. It
just binds more resources on a new feature.

Unfortuntately i hadn't much time to work on vchiq by myself.

Just my two cents
Stefan

2022-11-26 16:31:12

[permalink] [raw]

Subject: Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support

Hi Stefan

On 11/26/22 8:12 PM, Stefan Wahren wrote:
> Hi Umang,
>
> Am 21.11.22 um 22:47 schrieb Umang Jain:
>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
>> receiver IP core("Unicam) + the ISP driver found in BCM283x and
>> compatible
>> SoCs (namely BCM2711). Unicam is still under active development to work
>> with multistream support to get into mainline. Hence only the ISP driver
>> will remain the primary area of this series.
>
> thanks for working on this. But honestly i would prefer that vchiq
> comes out of staging before adding more features. As Greg said some
> time ago staging is not a place to "dump code and run away". These new
> files are in the same bad shape as the rest of vc04 before the
> clean-up here in staging started.

Certainly, I am not here to do that - but I am still learning the ropes.

If the staging issue is becoming a blocker for bcm2835-isp going
upstream, I would be happy to help here! Though I must mention that I
still have limited visibility so my aim would be to chart out a plan of
things needed to be done to get vc04_services out of staging!

>
> I agree that VCSM is on the TODO list for vchiq, but this driver is
> not necessary for making bcm2835-audio & bcm2835-camera leave staging.
> It just binds more resources on a new feature.

I see two TODO files in vc04_services:
./bcm2835-camera/TODO
./interface/TODO

One of the bcm2835-camera TODO points to the vc-sm-cma driver itself. So
that's address in the series. The other remaining one - I will need to
take a deeper look before commenting on it.

The main chunk of TODO are in vc04_services/interfaces/TODO. Doing a
cursory reading of them suggests that these apply to *all* vc04_services
components? Am I right?

Are these are the specific bits of cleanup you are referring to in your
comment?

>
> Unfortuntately i hadn't much time to work on vchiq by myself.
>
> Just my two cents
> Stefan
>

2022-11-26 23:16:20

[permalink] [raw]

Subject: Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support

Hi Umang,

Am 26.11.22 um 17:26 schrieb Umang Jain:
> Hi Stefan
>
> On 11/26/22 8:12 PM, Stefan Wahren wrote:
>> Hi Umang,
>>
>> Am 21.11.22 um 22:47 schrieb Umang Jain:
>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and
>>> compatible
>>> SoCs (namely BCM2711). Unicam is still under active development to work
>>> with multistream support to get into mainline. Hence only the ISP
>>> driver
>>> will remain the primary area of this series.
>>
>> thanks for working on this. But honestly i would prefer that vchiq
>> comes out of staging before adding more features. As Greg said some
>> time ago staging is not a place to "dump code and run away". These
>> new files are in the same bad shape as the rest of vc04 before the
>> clean-up here in staging started.
>
> Certainly, I am not here to do that - but I am still learning the ropes.
no problem.
>
> If the staging issue is becoming a blocker for bcm2835-isp going
> upstream, I would be happy to help here! Though I must mention that I
> still have limited visibility so my aim would be to chart out a plan
> of things needed to be done to get vc04_services out of staging!

The vchiq driver is in staging since 2016, so every step forwards is
good. Unfortunately all of the low hanging fruits has been gathered.

For me the most important, but not to tricky steps to get vchiq out of
staging would be:

* Cleanup logging mechanism

* Get rid of custom function return values

There was already an attempt for this [1]

* Get rid of all non essential global structures and create a proper per
device structure

>
>>
>> I agree that VCSM is on the TODO list for vchiq, but this driver is
>> not necessary for making bcm2835-audio & bcm2835-camera leave
>> staging. It just binds more resources on a new feature.
>
> I see two TODO files in vc04_services:
> ./bcm2835-camera/TODO
> ./interface/TODO
>
> One of the bcm2835-camera TODO points to the vc-sm-cma driver itself.
> So that's address in the series. The other remaining one - I will need
> to take a deeper look before commenting on it.
>
> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing a
> cursory reading of them suggests that these apply to *all*
> vc04_services components? Am I right?
Actually these applies just for the interfaces directory. Some of them
could apply to the services, but this is no priority.
>
> Are these are the specific bits of cleanup you are referring to in
> your comment?

You mean about bcm2835-isp? There were too many changes to vchiq that i
don't remember them all. The first that come to my mind was those fancy
comment sections which is not kernel coding style. It has been removed.

[1] -
https://lore.kernel.org/linux-staging/[email protected]/

>
>
>>
>> Unfortuntately i hadn't much time to work on vchiq by myself.
>>
>> Just my two cents
>> Stefan
>>
>
>
> _______________________________________________
> linux-arm-kernel mailing list
> [email protected]
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

2022-11-30 11:32:38

[permalink] [raw]

Subject: Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support

Hi Stefan,

On 11/27/22 6:56 AM, Stefan Wahren wrote:
> Hi Umang,
>
> Am 26.11.22 um 17:26 schrieb Umang Jain:
>> Hi Stefan
>>
>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
>>> Hi Umang,
>>>
>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and
>>>> is a
>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and
>>>> compatible
>>>> SoCs (namely BCM2711). Unicam is still under active development to
>>>> work
>>>> with multistream support to get into mainline. Hence only the ISP
>>>> driver
>>>> will remain the primary area of this series.
>>>
>>> thanks for working on this. But honestly i would prefer that vchiq
>>> comes out of staging before adding more features. As Greg said some
>>> time ago staging is not a place to "dump code and run away". These
>>> new files are in the same bad shape as the rest of vc04 before the
>>> clean-up here in staging started.
>>
>> Certainly, I am not here to do that - but I am still learning the ropes.
> no problem.
>>
>> If the staging issue is becoming a blocker for bcm2835-isp going
>> upstream, I would be happy to help here! Though I must mention that I
>> still have limited visibility so my aim would be to chart out a plan
>> of things needed to be done to get vc04_services out of staging!
>
> The vchiq driver is in staging since 2016, so every step forwards is
> good. Unfortunately all of the low hanging fruits has been gathered.
>
> For me the most important, but not to tricky steps to get vchiq out of
> staging would be:
>
> * Cleanup logging mechanism
>
> * Get rid of custom function return values
>
> There was already an attempt for this [1]
>
> * Get rid of all non essential global structures and create a proper per
> device structure
>
>>
>>>
>>> I agree that VCSM is on the TODO list for vchiq, but this driver is
>>> not necessary for making bcm2835-audio & bcm2835-camera leave
>>> staging. It just binds more resources on a new feature.

bcm2835-camera is the legacy camera stack which probably need to be
dropped from hereon...
>>
>> I see two TODO files in vc04_services:
>> ./bcm2835-camera/TODO
>> ./interface/TODO
>>
>> One of the bcm2835-camera TODO points to the vc-sm-cma driver itself.
>> So that's address in the series. The other remaining one - I will
>> need to take a deeper look before commenting on it.
>>
>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing a
>> cursory reading of them suggests that these apply to *all*
>> vc04_services components? Am I right?
> Actually these applies just for the interfaces directory. Some of them
> could apply to the services, but this is no priority.

By no priority, you mean this doesn't affect the criteria required to
ful-fill to get these out of staging?
>>
>> Are these are the specific bits of cleanup you are referring to in
>> your comment?
>
> You mean about bcm2835-isp? There were too many changes to vchiq that
> i don't remember them all. The first that come to my mind was those
> fancy comment sections which is not kernel coding style. It has been
> removed.

No, I don't mean the bcm2835-isp changes (those are upcoming /
out-of-tree still so...). I mean what are the specific bits / points
that needs to be addressed to get vc04_services out of the staging.

You have mentioned it above now, so I'll follow up on those. The many
vchiq changes you referred to above comment (that you don't remember)
are from [1] as well or some other series ?

>
> [1] -
> https://lore.kernel.org/linux-staging/[email protected]/
>
>>
>>
>>>
>>> Unfortuntately i hadn't much time to work on vchiq by myself.
>>>
>>> Just my two cents
>>> Stefan
>>>
>>
>>
>> _______________________________________________
>> linux-arm-kernel mailing list
>> [email protected]
>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

2022-12-01 23:13:28

[permalink] [raw]

Subject: Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support

Hi Umang,

Am 30.11.22 um 11:58 schrieb Umang Jain:
> Hi Stefan,
>
> On 11/27/22 6:56 AM, Stefan Wahren wrote:
>> Hi Umang,
>>
>> Am 26.11.22 um 17:26 schrieb Umang Jain:
>>> Hi Stefan
>>>
>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
>>>> Hi Umang,
>>>>
>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and
>>>>> is a
>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and
>>>>> compatible
>>>>> SoCs (namely BCM2711). Unicam is still under active development to
>>>>> work
>>>>> with multistream support to get into mainline. Hence only the ISP
>>>>> driver
>>>>> will remain the primary area of this series.
>>>>
>>>> thanks for working on this. But honestly i would prefer that vchiq
>>>> comes out of staging before adding more features. As Greg said some
>>>> time ago staging is not a place to "dump code and run away". These
>>>> new files are in the same bad shape as the rest of vc04 before the
>>>> clean-up here in staging started.
>>>
>>> Certainly, I am not here to do that - but I am still learning the
>>> ropes.
>> no problem.
>>>
>>> If the staging issue is becoming a blocker for bcm2835-isp going
>>> upstream, I would be happy to help here! Though I must mention that
>>> I still have limited visibility so my aim would be to chart out a
>>> plan of things needed to be done to get vc04_services out of staging!
>>
>> The vchiq driver is in staging since 2016, so every step forwards is
>> good. Unfortunately all of the low hanging fruits has been gathered.
>>
>> For me the most important, but not to tricky steps to get vchiq out
>> of staging would be:
>>
>> * Cleanup logging mechanism
>>
>> * Get rid of custom function return values
>>
>> There was already an attempt for this [1]
>>
>> * Get rid of all non essential global structures and create a proper per
>> device structure
>>
>>>
>>>>
>>>> I agree that VCSM is on the TODO list for vchiq, but this driver is
>>>> not necessary for making bcm2835-audio & bcm2835-camera leave
>>>> staging. It just binds more resources on a new feature.
>
> bcm2835-camera is the legacy camera stack which probably need to be
> dropped from hereon...
I don't not know if there any users left, so i would be careful here.
Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this dumb
question but i'm not expert here.
>>>
>>> I see two TODO files in vc04_services:
>>> ./bcm2835-camera/TODO
>>> ./interface/TODO
>>>
>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
>>> itself. So that's address in the series. The other remaining one - I
>>> will need to take a deeper look before commenting on it.
>>>
>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing a
>>> cursory reading of them suggests that these apply to *all*
>>> vc04_services components? Am I right?
>> Actually these applies just for the interfaces directory. Some of
>> them could apply to the services, but this is no priority.
>
> By no priority, you mean this doesn't affect the criteria required to
> ful-fill to get these out of staging?
Correct
>>>
>>> Are these are the specific bits of cleanup you are referring to in
>>> your comment?
>>
>> You mean about bcm2835-isp? There were too many changes to vchiq that
>> i don't remember them all. The first that come to my mind was those
>> fancy comment sections which is not kernel coding style. It has been
>> removed.
>
> No, I don't mean the bcm2835-isp changes (those are upcoming /
> out-of-tree still so...). I mean what are the specific bits / points
> that needs to be addressed to get vc04_services out of the staging.
These were the points which i mentioned in my last email. They came from
interface/TODO.
>
> You have mentioned it above now, so I'll follow up on those.
That would be great :)
> The many vchiq changes you referred to above comment (that you don't
> remember) are from [1] as well or some other series ?
Sorry, for the confusing. The many changes i refer were the dozens of
clean up patches for vc04_interfaces in mainline staging since the last
years. [1] was just a single patch which has been accepted yet.
>
>>
>> [1] -
>> https://lore.kernel.org/linux-staging/[email protected]/
>>
>>>
>>>
>>>>
>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
>>>>
>>>> Just my two cents
>>>> Stefan
>>>>
>>>
>>>
>>> _______________________________________________
>>> linux-arm-kernel mailing list
>>> [email protected]
>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>

2022-12-02 04:13:58

[permalink] [raw]

Subject: Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support

Hi Stefan,

On 12/2/22 6:45 AM, Stefan Wahren wrote:
> Hi Umang,
>
> Am 30.11.22 um 11:58 schrieb Umang Jain:
>> Hi Stefan,
>>
>> On 11/27/22 6:56 AM, Stefan Wahren wrote:
>>> Hi Umang,
>>>
>>> Am 26.11.22 um 17:26 schrieb Umang Jain:
>>>> Hi Stefan
>>>>
>>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
>>>>> Hi Umang,
>>>>>
>>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
>>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1]
>>>>>> and is a
>>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
>>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and
>>>>>> compatible
>>>>>> SoCs (namely BCM2711). Unicam is still under active development
>>>>>> to work
>>>>>> with multistream support to get into mainline. Hence only the ISP
>>>>>> driver
>>>>>> will remain the primary area of this series.
>>>>>
>>>>> thanks for working on this. But honestly i would prefer that vchiq
>>>>> comes out of staging before adding more features. As Greg said
>>>>> some time ago staging is not a place to "dump code and run away".
>>>>> These new files are in the same bad shape as the rest of vc04
>>>>> before the clean-up here in staging started.
>>>>
>>>> Certainly, I am not here to do that - but I am still learning the
>>>> ropes.
>>> no problem.
>>>>
>>>> If the staging issue is becoming a blocker for bcm2835-isp going
>>>> upstream, I would be happy to help here! Though I must mention that
>>>> I still have limited visibility so my aim would be to chart out a
>>>> plan of things needed to be done to get vc04_services out of staging!
>>>
>>> The vchiq driver is in staging since 2016, so every step forwards is
>>> good. Unfortunately all of the low hanging fruits has been gathered.
>>>
>>> For me the most important, but not to tricky steps to get vchiq out
>>> of staging would be:
>>>
>>> * Cleanup logging mechanism
>>>
>>> * Get rid of custom function return values
>>>
>>> There was already an attempt for this [1]
>>>
>>> * Get rid of all non essential global structures and create a proper
>>> per
>>> device structure
>>>
>>>>
>>>>>
>>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
>>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
>>>>> staging. It just binds more resources on a new feature.
>>
>> bcm2835-camera is the legacy camera stack which probably need to be
>> dropped from hereon...
> I don't not know if there any users left, so i would be careful here.
> Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
> dumb question but i'm not expert here.

I am careful too here and probably need Input from RaspberryPi in order
to proceed to drop it. But from my perspective - bcm2835-camera is _not_
going out of staging - it'll either sit here (or probably dropped) as
statied from [1]

```
+ * There are two camera drivers in the kernel for BCM283x - this one
+ * and bcm2835-camera (currently in staging).
```

The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
(bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
mentioned in my cover the testing of bcm2835-isp happened on top of
unicam patches.

[1]:
https://lore.kernel.org/linux-media/[email protected]/
>>>>
>>>> I see two TODO files in vc04_services:
>>>> ./bcm2835-camera/TODO
>>>> ./interface/TODO
>>>>
>>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
>>>> itself. So that's address in the series. The other remaining one -
>>>> I will need to take a deeper look before commenting on it.
>>>>
>>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
>>>> a cursory reading of them suggests that these apply to *all*
>>>> vc04_services components? Am I right?
>>> Actually these applies just for the interfaces directory. Some of
>>> them could apply to the services, but this is no priority.
>>
>> By no priority, you mean this doesn't affect the criteria required to
>> ful-fill to get these out of staging?
> Correct
>>>>
>>>> Are these are the specific bits of cleanup you are referring to in
>>>> your comment?
>>>
>>> You mean about bcm2835-isp? There were too many changes to vchiq
>>> that i don't remember them all. The first that come to my mind was
>>> those fancy comment sections which is not kernel coding style. It
>>> has been removed.
>>
>> No, I don't mean the bcm2835-isp changes (those are upcoming /
>> out-of-tree still so...). I mean what are the specific bits / points
>> that needs to be addressed to get vc04_services out of the staging.
> These were the points which i mentioned in my last email. They came
> from interface/TODO.
>>
>> You have mentioned it above now, so I'll follow up on those.
> That would be great :)
>> The many vchiq changes you referred to above comment (that you don't
>> remember) are from [1] as well or some other series ?
> Sorry, for the confusing. The many changes i refer were the dozens of
> clean up patches for vc04_interfaces in mainline staging since the
> last years. [1] was just a single patch which has been accepted yet.

Ah I see. There are many others that I've to dig out then. Thanks for
clarifying!
>>
>>>
>>> [1] -
>>> https://lore.kernel.org/linux-staging/[email protected]/
>>>
>>>>
>>>>
>>>>>
>>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
>>>>>
>>>>> Just my two cents
>>>>> Stefan
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> linux-arm-kernel mailing list
>>>> [email protected]
>>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>

2022-12-02 10:05:46

[permalink] [raw]

Subject: Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support

Hi Umang,

On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
> On 12/2/22 6:45 AM, Stefan Wahren wrote:
> > Am 30.11.22 um 11:58 schrieb Umang Jain:
> >> On 11/27/22 6:56 AM, Stefan Wahren wrote:
> >>> Am 26.11.22 um 17:26 schrieb Umang Jain:
> >>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
> >>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
> >>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> >>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> >>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> >>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
> >>>>>> with multistream support to get into mainline. Hence only the ISP driver
> >>>>>> will remain the primary area of this series.
> >>>>>
> >>>>> thanks for working on this. But honestly i would prefer that vchiq
> >>>>> comes out of staging before adding more features. As Greg said
> >>>>> some time ago staging is not a place to "dump code and run away".
> >>>>> These new files are in the same bad shape as the rest of vc04
> >>>>> before the clean-up here in staging started.
> >>>>
> >>>> Certainly, I am not here to do that - but I am still learning the ropes.
> >>>
> >>> no problem.
> >>>
> >>>> If the staging issue is becoming a blocker for bcm2835-isp going
> >>>> upstream, I would be happy to help here! Though I must mention that
> >>>> I still have limited visibility so my aim would be to chart out a
> >>>> plan of things needed to be done to get vc04_services out of staging!
> >>>
> >>> The vchiq driver is in staging since 2016, so every step forwards is
> >>> good. Unfortunately all of the low hanging fruits has been gathered.
> >>>
> >>> For me the most important, but not to tricky steps to get vchiq out
> >>> of staging would be:
> >>>
> >>> * Cleanup logging mechanism
> >>>
> >>> * Get rid of custom function return values
> >>>
> >>> There was already an attempt for this [1]
> >>>
> >>> * Get rid of all non essential global structures and create a proper per
> >>> device structure
> >>>
> >>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
> >>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
> >>>>> staging. It just binds more resources on a new feature.
> >>
> >> bcm2835-camera is the legacy camera stack which probably need to be
> >> dropped from hereon...
> >
> > I don't not know if there any users left, so i would be careful here.
> > Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
> > dumb question but i'm not expert here.
>
> I am careful too here and probably need Input from RaspberryPi in order
> to proceed to drop it. But from my perspective - bcm2835-camera is _not_
> going out of staging - it'll either sit here (or probably dropped) as
> statied from [1]
>
> ```
> + * There are two camera drivers in the kernel for BCM283x - this one
> + * and bcm2835-camera (currently in staging).
> ```
>
> The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
> (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
> mentioned in my cover the testing of bcm2835-isp happened on top of
> unicam patches.

To be accurate, the bcm2835-camera driver supports the VC4
firmware-based camera stack. In that setup, the camera sensors (OV5647
or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
firmware, which provides a high-level interface towards the kernel. This
architecture has been replaced by Linux-side control of the camera
sensors (through existing drivers in drivers/media/i2c/), Unicam
(through the driver from [1]) and ISP (through this driver). Moving
control to the Linux side requires complex processing in userspace,
handled by libcamera.

bcm2835-camera is thus replaced by multiple drivers combined with
libcamera, and that is the camera stack that is shipped by Raspberry Pi
these days. While this may affect some userspace use cases), we will not
work on destaging bcm2835-camera, and as far as I'm aware, nobody else
is planning to do so either. I don't mind much if the driver stays in
staging for some more time, but I'd rather drop it if possible.

> [1]: https://lore.kernel.org/linux-media/[email protected]/
>
> >>>>
> >>>> I see two TODO files in vc04_services:
> >>>> ./bcm2835-camera/TODO
> >>>> ./interface/TODO
> >>>>
> >>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
> >>>> itself. So that's address in the series. The other remaining one -
> >>>> I will need to take a deeper look before commenting on it.
> >>>>
> >>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
> >>>> a cursory reading of them suggests that these apply to *all*
> >>>> vc04_services components? Am I right?
> >>>
> >>> Actually these applies just for the interfaces directory. Some of
> >>> them could apply to the services, but this is no priority.
> >>
> >> By no priority, you mean this doesn't affect the criteria required to
> >> ful-fill to get these out of staging?
> >
> > Correct
> >
> >>>> Are these are the specific bits of cleanup you are referring to in
> >>>> your comment?
> >>>
> >>> You mean about bcm2835-isp? There were too many changes to vchiq
> >>> that i don't remember them all. The first that come to my mind was
> >>> those fancy comment sections which is not kernel coding style. It
> >>> has been removed.
> >>
> >> No, I don't mean the bcm2835-isp changes (those are upcoming /
> >> out-of-tree still so...). I mean what are the specific bits / points
> >> that needs to be addressed to get vc04_services out of the staging.
> >
> > These were the points which i mentioned in my last email. They came
> > from interface/TODO.
> >
> >> You have mentioned it above now, so I'll follow up on those.
> >
> > That would be great :)
> >
> >> The many vchiq changes you referred to above comment (that you don't
> >> remember) are from [1] as well or some other series ?
> >
> > Sorry, for the confusing. The many changes i refer were the dozens of
> > clean up patches for vc04_interfaces in mainline staging since the
> > last years. [1] was just a single patch which has been accepted yet.
>
> Ah I see. There are many others that I've to dig out then. Thanks for
> clarifying!
>
> >>> [1] -
> >>> https://lore.kernel.org/linux-staging/[email protected]/
> >>>
> >>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
> >>>>>
> >>>>> Just my two cents
> >>>>> Stefan

--
Regards,

Laurent Pinchart

2022-12-02 11:35:22

[permalink] [raw]

Subject: Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support

Hi Laurent, Umang, and Stefan.

On Fri, 2 Dec 2022 at 09:17, Laurent Pinchart
<[email protected]> wrote:
>
> Hi Umang,
>
> On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
> > On 12/2/22 6:45 AM, Stefan Wahren wrote:
> > > Am 30.11.22 um 11:58 schrieb Umang Jain:
> > >> On 11/27/22 6:56 AM, Stefan Wahren wrote:
> > >>> Am 26.11.22 um 17:26 schrieb Umang Jain:
> > >>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
> > >>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
> > >>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> > >>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> > >>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> > >>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
> > >>>>>> with multistream support to get into mainline. Hence only the ISP driver
> > >>>>>> will remain the primary area of this series.
> > >>>>>
> > >>>>> thanks for working on this. But honestly i would prefer that vchiq
> > >>>>> comes out of staging before adding more features. As Greg said
> > >>>>> some time ago staging is not a place to "dump code and run away".
> > >>>>> These new files are in the same bad shape as the rest of vc04
> > >>>>> before the clean-up here in staging started.
> > >>>>
> > >>>> Certainly, I am not here to do that - but I am still learning the ropes.
> > >>>
> > >>> no problem.
> > >>>
> > >>>> If the staging issue is becoming a blocker for bcm2835-isp going
> > >>>> upstream, I would be happy to help here! Though I must mention that
> > >>>> I still have limited visibility so my aim would be to chart out a
> > >>>> plan of things needed to be done to get vc04_services out of staging!
> > >>>
> > >>> The vchiq driver is in staging since 2016, so every step forwards is
> > >>> good. Unfortunately all of the low hanging fruits has been gathered.
> > >>>
> > >>> For me the most important, but not to tricky steps to get vchiq out
> > >>> of staging would be:
> > >>>
> > >>> * Cleanup logging mechanism
> > >>>
> > >>> * Get rid of custom function return values
> > >>>
> > >>> There was already an attempt for this [1]
> > >>>
> > >>> * Get rid of all non essential global structures and create a proper per
> > >>> device structure
> > >>>
> > >>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
> > >>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
> > >>>>> staging. It just binds more resources on a new feature.
> > >>
> > >> bcm2835-camera is the legacy camera stack which probably need to be
> > >> dropped from hereon...
> > >
> > > I don't not know if there any users left, so i would be careful here.
> > > Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
> > > dumb question but i'm not expert here.
> >
> > I am careful too here and probably need Input from RaspberryPi in order
> > to proceed to drop it. But from my perspective - bcm2835-camera is _not_
> > going out of staging - it'll either sit here (or probably dropped) as
> > statied from [1]
> >
> > ```
> > + * There are two camera drivers in the kernel for BCM283x - this one
> > + * and bcm2835-camera (currently in staging).
> > ```
> >
> > The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
> > (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
> > mentioned in my cover the testing of bcm2835-isp happened on top of
> > unicam patches.
>
> To be accurate, the bcm2835-camera driver supports the VC4
> firmware-based camera stack. In that setup, the camera sensors (OV5647
> or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
> firmware, which provides a high-level interface towards the kernel. This
> architecture has been replaced by Linux-side control of the camera
> sensors (through existing drivers in drivers/media/i2c/), Unicam
> (through the driver from [1]) and ISP (through this driver). Moving
> control to the Linux side requires complex processing in userspace,
> handled by libcamera.
>
> bcm2835-camera is thus replaced by multiple drivers combined with
> libcamera, and that is the camera stack that is shipped by Raspberry Pi
> these days. While this may affect some userspace use cases), we will not
> work on destaging bcm2835-camera, and as far as I'm aware, nobody else
> is planning to do so either. I don't mind much if the driver stays in
> staging for some more time, but I'd rather drop it if possible.

It would be reasonable to drop it at the point that Libcamera can work
to a similar level with at least the following list of applications:
- FFmpeg
- Gstreamer
- Chromium
- Firefox
- Motion
And that still leaves a huge number of existing V4L2 apps out in the cold.

Do you wish to make any predictions as to when that would be
achievable? Or even when a v1.0 release of libcamera is going to
happen?
Dropping anything prior to those points would be rather premature in my book.

The TODOs on bcm2835-camera are:
1) Zero copy. That comes almost for free as bcm2835-isp already does
this, but it does rely on vcsm-cma.
The main reason I haven't pushed it is that it then requires
reasonable amounts of CMA heap for all the buffers, which until
recently haven't been present in the default configurations. With the
vc4 DRM driver now being default (at least for the vendor kernel) and
also requiring CMA, making the change makes more sense.
AFAIK there is no easy way to have one driver choosing between using
vb2_vmalloc_memops and vb2_dma_contig_memops at runtime, but I may be
wrong.
Actually bcm2835_defconfig appears to only allocate a 32MB CMA heap,
so perhaps we don't get very far.

2) This isn't workable within the current V4L2 frameworks. The
multi-planar V4L2 pixel formats are currently allocated as independent
buffers for each plane, whereas the firmware needs a single buffer
with (currently) specific offsets for the chroma planes. The
V4L2/videobuf2 core changes required to implement that are going to be
significant, and have minimal gain.
The specific stride handling is already dealt with (set bytesperline
appropriately), it's the padding of the height to a multiple of 16
before the chroma planes on YUV420 and NV12 formats that require the
firmware to do a small amount of repacking. The performance hit is
actually minimal anyway.

If bcm2835-camera is the only thing holding back vc04_services, then I
can have a look at it.

Dave

> > [1]: https://lore.kernel.org/linux-media/[email protected]/
> >
> > >>>>
> > >>>> I see two TODO files in vc04_services:
> > >>>> ./bcm2835-camera/TODO
> > >>>> ./interface/TODO
> > >>>>
> > >>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
> > >>>> itself. So that's address in the series. The other remaining one -
> > >>>> I will need to take a deeper look before commenting on it.
> > >>>>
> > >>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
> > >>>> a cursory reading of them suggests that these apply to *all*
> > >>>> vc04_services components? Am I right?
> > >>>
> > >>> Actually these applies just for the interfaces directory. Some of
> > >>> them could apply to the services, but this is no priority.
> > >>
> > >> By no priority, you mean this doesn't affect the criteria required to
> > >> ful-fill to get these out of staging?
> > >
> > > Correct
> > >
> > >>>> Are these are the specific bits of cleanup you are referring to in
> > >>>> your comment?
> > >>>
> > >>> You mean about bcm2835-isp? There were too many changes to vchiq
> > >>> that i don't remember them all. The first that come to my mind was
> > >>> those fancy comment sections which is not kernel coding style. It
> > >>> has been removed.
> > >>
> > >> No, I don't mean the bcm2835-isp changes (those are upcoming /
> > >> out-of-tree still so...). I mean what are the specific bits / points
> > >> that needs to be addressed to get vc04_services out of the staging.
> > >
> > > These were the points which i mentioned in my last email. They came
> > > from interface/TODO.
> > >
> > >> You have mentioned it above now, so I'll follow up on those.
> > >
> > > That would be great :)
> > >
> > >> The many vchiq changes you referred to above comment (that you don't
> > >> remember) are from [1] as well or some other series ?
> > >
> > > Sorry, for the confusing. The many changes i refer were the dozens of
> > > clean up patches for vc04_interfaces in mainline staging since the
> > > last years. [1] was just a single patch which has been accepted yet.
> >
> > Ah I see. There are many others that I've to dig out then. Thanks for
> > clarifying!
> >
> > >>> [1] -
> > >>> https://lore.kernel.org/linux-staging/[email protected]/
> > >>>
> > >>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
> > >>>>>
> > >>>>> Just my two cents
> > >>>>> Stefan
>
> --
> Regards,
>
> Laurent Pinchart

2022-12-02 12:24:57

[permalink] [raw]

Subject: Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support

Hi Dave,

On Fri, Dec 02, 2022 at 11:23:29AM +0000, Dave Stevenson wrote:
> On Fri, 2 Dec 2022 at 09:17, Laurent Pinchart wrote:
> > On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
> > > On 12/2/22 6:45 AM, Stefan Wahren wrote:
> > > > Am 30.11.22 um 11:58 schrieb Umang Jain:
> > > >> On 11/27/22 6:56 AM, Stefan Wahren wrote:
> > > >>> Am 26.11.22 um 17:26 schrieb Umang Jain:
> > > >>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
> > > >>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
> > > >>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> > > >>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> > > >>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> > > >>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
> > > >>>>>> with multistream support to get into mainline. Hence only the ISP driver
> > > >>>>>> will remain the primary area of this series.
> > > >>>>>
> > > >>>>> thanks for working on this. But honestly i would prefer that vchiq
> > > >>>>> comes out of staging before adding more features. As Greg said
> > > >>>>> some time ago staging is not a place to "dump code and run away".
> > > >>>>> These new files are in the same bad shape as the rest of vc04
> > > >>>>> before the clean-up here in staging started.
> > > >>>>
> > > >>>> Certainly, I am not here to do that - but I am still learning the ropes.
> > > >>>
> > > >>> no problem.
> > > >>>
> > > >>>> If the staging issue is becoming a blocker for bcm2835-isp going
> > > >>>> upstream, I would be happy to help here! Though I must mention that
> > > >>>> I still have limited visibility so my aim would be to chart out a
> > > >>>> plan of things needed to be done to get vc04_services out of staging!
> > > >>>
> > > >>> The vchiq driver is in staging since 2016, so every step forwards is
> > > >>> good. Unfortunately all of the low hanging fruits has been gathered.
> > > >>>
> > > >>> For me the most important, but not to tricky steps to get vchiq out
> > > >>> of staging would be:
> > > >>>
> > > >>> * Cleanup logging mechanism
> > > >>>
> > > >>> * Get rid of custom function return values
> > > >>>
> > > >>> There was already an attempt for this [1]
> > > >>>
> > > >>> * Get rid of all non essential global structures and create a proper per
> > > >>> device structure
> > > >>>
> > > >>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
> > > >>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
> > > >>>>> staging. It just binds more resources on a new feature.
> > > >>
> > > >> bcm2835-camera is the legacy camera stack which probably need to be
> > > >> dropped from hereon...
> > > >
> > > > I don't not know if there any users left, so i would be careful here.
> > > > Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
> > > > dumb question but i'm not expert here.
> > >
> > > I am careful too here and probably need Input from RaspberryPi in order
> > > to proceed to drop it. But from my perspective - bcm2835-camera is _not_
> > > going out of staging - it'll either sit here (or probably dropped) as
> > > statied from [1]
> > >
> > > ```
> > > + * There are two camera drivers in the kernel for BCM283x - this one
> > > + * and bcm2835-camera (currently in staging).
> > > ```
> > >
> > > The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
> > > (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
> > > mentioned in my cover the testing of bcm2835-isp happened on top of
> > > unicam patches.
> >
> > To be accurate, the bcm2835-camera driver supports the VC4
> > firmware-based camera stack. In that setup, the camera sensors (OV5647
> > or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
> > firmware, which provides a high-level interface towards the kernel. This
> > architecture has been replaced by Linux-side control of the camera
> > sensors (through existing drivers in drivers/media/i2c/), Unicam
> > (through the driver from [1]) and ISP (through this driver). Moving
> > control to the Linux side requires complex processing in userspace,
> > handled by libcamera.
> >
> > bcm2835-camera is thus replaced by multiple drivers combined with
> > libcamera, and that is the camera stack that is shipped by Raspberry Pi
> > these days. While this may affect some userspace use cases), we will not
> > work on destaging bcm2835-camera, and as far as I'm aware, nobody else
> > is planning to do so either. I don't mind much if the driver stays in
> > staging for some more time, but I'd rather drop it if possible.
>
> It would be reasonable to drop it at the point that Libcamera can work
> to a similar level with at least the following list of applications:
> - FFmpeg
> - Gstreamer
> - Chromium
> - Firefox
> - Motion
> And that still leaves a huge number of existing V4L2 apps out in the cold.

That's exactly the kind of input we were looking for, thanks.

GStreamer is already addressed.

Chromium and Firefox will go through PipeWire. There is a working
implementation in libwebrtc, Kieran may be able to comment on the
upstreaming state. It will take some time for distributions to switch,
and I can't predict the time line, but that seems to clearly be the
direction the Linux desktop is taking.

I haven't looked at FFmpeg yet (maybe someone has ?). It probably makes
sense to add native libcamera support to FFmpeg, even if PipeWire
support could also make sense. It could also make sense to expose in the
libcamera V4L2 compat layer the libv4l2 API functions, as that would
allow linking FFmpeg (when compiled with CONFIG_LIBV4L2) to libcamera
instead of libv4l2, but Debian doesn't set CONFIG_LIBV4L2, so this isn't
an immediate solution to the problem.

Same thing for motion, except it has no libv4l2 support. The V4L2 compat
layer could still be used with LD_PRELOAD, but that's not a great
solution. Native libcamera support would make more sense (or possibly
even GStreamer support, I don't know if upstream would accept that).

In a side note, how do the above applications work today on Raspberry Pi
platforms that use a sensor not supported by the legacy camera stack ?

> Do you wish to make any predictions as to when that would be
> achievable? Or even when a v1.0 release of libcamera is going to
> happen?

Now that we have started tagging releases, we've also decided to publish
a roadmap with the development still needed to stabilize the API. We'll
likely start working on it this month.

> Dropping anything prior to those points would be rather premature in my book.

Something I forgot to mention is that there should be no issue at all
keeping bcm2835-camera fully supported in the Raspberry Pi downstream
kernel for a longer period of time. It's in upstream that I don't think
it should be destaged, as it's already considered legacy and should be
phased out. Do you know if there are users of that driver with a
mainline kernel ?

> The TODOs on bcm2835-camera are:
> 1) Zero copy. That comes almost for free as bcm2835-isp already does
> this, but it does rely on vcsm-cma.
> The main reason I haven't pushed it is that it then requires
> reasonable amounts of CMA heap for all the buffers, which until
> recently haven't been present in the default configurations. With the
> vc4 DRM driver now being default (at least for the vendor kernel) and
> also requiring CMA, making the change makes more sense.
> AFAIK there is no easy way to have one driver choosing between using
> vb2_vmalloc_memops and vb2_dma_contig_memops at runtime, but I may be
> wrong.

Some drivers use a module parameter for that, but that's not great.

> Actually bcm2835_defconfig appears to only allocate a 32MB CMA heap,
> so perhaps we don't get very far.
>
> 2) This isn't workable within the current V4L2 frameworks. The
> multi-planar V4L2 pixel formats are currently allocated as independent
> buffers for each plane, whereas the firmware needs a single buffer
> with (currently) specific offsets for the chroma planes. The
> V4L2/videobuf2 core changes required to implement that are going to be
> significant, and have minimal gain.
> The specific stride handling is already dealt with (set bytesperline
> appropriately), it's the padding of the height to a multiple of 16
> before the chroma planes on YUV420 and NV12 formats that require the
> firmware to do a small amount of repacking. The performance hit is
> actually minimal anyway.
>
> If bcm2835-camera is the only thing holding back vc04_services, then I
> can have a look at it.

I'll let Umang comment on whether it's holding vc04_services back, but
my understanding it that we could destage vc04_services while keeping
bcm2835-camera in staging for the time being. If anyone disagrees with
that, please let me know.

> > > [1]: https://lore.kernel.org/linux-media/[email protected]/
> > >
> > > >>>>
> > > >>>> I see two TODO files in vc04_services:
> > > >>>> ./bcm2835-camera/TODO
> > > >>>> ./interface/TODO
> > > >>>>
> > > >>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
> > > >>>> itself. So that's address in the series. The other remaining one -
> > > >>>> I will need to take a deeper look before commenting on it.
> > > >>>>
> > > >>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
> > > >>>> a cursory reading of them suggests that these apply to *all*
> > > >>>> vc04_services components? Am I right?
> > > >>>
> > > >>> Actually these applies just for the interfaces directory. Some of
> > > >>> them could apply to the services, but this is no priority.
> > > >>
> > > >> By no priority, you mean this doesn't affect the criteria required to
> > > >> ful-fill to get these out of staging?
> > > >
> > > > Correct
> > > >
> > > >>>> Are these are the specific bits of cleanup you are referring to in
> > > >>>> your comment?
> > > >>>
> > > >>> You mean about bcm2835-isp? There were too many changes to vchiq
> > > >>> that i don't remember them all. The first that come to my mind was
> > > >>> those fancy comment sections which is not kernel coding style. It
> > > >>> has been removed.
> > > >>
> > > >> No, I don't mean the bcm2835-isp changes (those are upcoming /
> > > >> out-of-tree still so...). I mean what are the specific bits / points
> > > >> that needs to be addressed to get vc04_services out of the staging.
> > > >
> > > > These were the points which i mentioned in my last email. They came
> > > > from interface/TODO.
> > > >
> > > >> You have mentioned it above now, so I'll follow up on those.
> > > >
> > > > That would be great :)
> > > >
> > > >> The many vchiq changes you referred to above comment (that you don't
> > > >> remember) are from [1] as well or some other series ?
> > > >
> > > > Sorry, for the confusing. The many changes i refer were the dozens of
> > > > clean up patches for vc04_interfaces in mainline staging since the
> > > > last years. [1] was just a single patch which has been accepted yet.
> > >
> > > Ah I see. There are many others that I've to dig out then. Thanks for
> > > clarifying!
> > >
> > > >>> [1] -
> > > >>> https://lore.kernel.org/linux-staging/[email protected]/
> > > >>>
> > > >>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
> > > >>>>>
> > > >>>>> Just my two cents
> > > >>>>> Stefan

--
Regards,

Laurent Pinchart

2022-12-02 12:51:20

[permalink] [raw]

Subject: Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support

Hi Laurent

On Fri, 2 Dec 2022 at 12:10, Laurent Pinchart
<[email protected]> wrote:
>
> Hi Dave,
>
> On Fri, Dec 02, 2022 at 11:23:29AM +0000, Dave Stevenson wrote:
> > On Fri, 2 Dec 2022 at 09:17, Laurent Pinchart wrote:
> > > On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
> > > > On 12/2/22 6:45 AM, Stefan Wahren wrote:
> > > > > Am 30.11.22 um 11:58 schrieb Umang Jain:
> > > > >> On 11/27/22 6:56 AM, Stefan Wahren wrote:
> > > > >>> Am 26.11.22 um 17:26 schrieb Umang Jain:
> > > > >>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
> > > > >>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
> > > > >>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> > > > >>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> > > > >>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> > > > >>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
> > > > >>>>>> with multistream support to get into mainline. Hence only the ISP driver
> > > > >>>>>> will remain the primary area of this series.
> > > > >>>>>
> > > > >>>>> thanks for working on this. But honestly i would prefer that vchiq
> > > > >>>>> comes out of staging before adding more features. As Greg said
> > > > >>>>> some time ago staging is not a place to "dump code and run away".
> > > > >>>>> These new files are in the same bad shape as the rest of vc04
> > > > >>>>> before the clean-up here in staging started.
> > > > >>>>
> > > > >>>> Certainly, I am not here to do that - but I am still learning the ropes.
> > > > >>>
> > > > >>> no problem.
> > > > >>>
> > > > >>>> If the staging issue is becoming a blocker for bcm2835-isp going
> > > > >>>> upstream, I would be happy to help here! Though I must mention that
> > > > >>>> I still have limited visibility so my aim would be to chart out a
> > > > >>>> plan of things needed to be done to get vc04_services out of staging!
> > > > >>>
> > > > >>> The vchiq driver is in staging since 2016, so every step forwards is
> > > > >>> good. Unfortunately all of the low hanging fruits has been gathered.
> > > > >>>
> > > > >>> For me the most important, but not to tricky steps to get vchiq out
> > > > >>> of staging would be:
> > > > >>>
> > > > >>> * Cleanup logging mechanism
> > > > >>>
> > > > >>> * Get rid of custom function return values
> > > > >>>
> > > > >>> There was already an attempt for this [1]
> > > > >>>
> > > > >>> * Get rid of all non essential global structures and create a proper per
> > > > >>> device structure
> > > > >>>
> > > > >>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
> > > > >>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
> > > > >>>>> staging. It just binds more resources on a new feature.
> > > > >>
> > > > >> bcm2835-camera is the legacy camera stack which probably need to be
> > > > >> dropped from hereon...
> > > > >
> > > > > I don't not know if there any users left, so i would be careful here.
> > > > > Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
> > > > > dumb question but i'm not expert here.
> > > >
> > > > I am careful too here and probably need Input from RaspberryPi in order
> > > > to proceed to drop it. But from my perspective - bcm2835-camera is _not_
> > > > going out of staging - it'll either sit here (or probably dropped) as
> > > > statied from [1]
> > > >
> > > > ```
> > > > + * There are two camera drivers in the kernel for BCM283x - this one
> > > > + * and bcm2835-camera (currently in staging).
> > > > ```
> > > >
> > > > The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
> > > > (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
> > > > mentioned in my cover the testing of bcm2835-isp happened on top of
> > > > unicam patches.
> > >
> > > To be accurate, the bcm2835-camera driver supports the VC4
> > > firmware-based camera stack. In that setup, the camera sensors (OV5647
> > > or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
> > > firmware, which provides a high-level interface towards the kernel. This
> > > architecture has been replaced by Linux-side control of the camera
> > > sensors (through existing drivers in drivers/media/i2c/), Unicam
> > > (through the driver from [1]) and ISP (through this driver). Moving
> > > control to the Linux side requires complex processing in userspace,
> > > handled by libcamera.
> > >
> > > bcm2835-camera is thus replaced by multiple drivers combined with
> > > libcamera, and that is the camera stack that is shipped by Raspberry Pi
> > > these days. While this may affect some userspace use cases), we will not
> > > work on destaging bcm2835-camera, and as far as I'm aware, nobody else
> > > is planning to do so either. I don't mind much if the driver stays in
> > > staging for some more time, but I'd rather drop it if possible.
> >
> > It would be reasonable to drop it at the point that Libcamera can work
> > to a similar level with at least the following list of applications:
> > - FFmpeg
> > - Gstreamer
> > - Chromium
> > - Firefox
> > - Motion
> > And that still leaves a huge number of existing V4L2 apps out in the cold.
>
> That's exactly the kind of input we were looking for, thanks.

Please note that those are only the 5 major ones that came immediately
to mind as having significant numbers of users. There are obviously
many more V4L2 apps around which currently "just work" if they use the
API correctly.

> GStreamer is already addressed.
>
> Chromium and Firefox will go through PipeWire. There is a working
> implementation in libwebrtc, Kieran may be able to comment on the
> upstreaming state. It will take some time for distributions to switch,
> and I can't predict the time line, but that seems to clearly be the
> direction the Linux desktop is taking.
>
> I haven't looked at FFmpeg yet (maybe someone has ?). It probably makes
> sense to add native libcamera support to FFmpeg, even if PipeWire
> support could also make sense. It could also make sense to expose in the
> libcamera V4L2 compat layer the libv4l2 API functions, as that would
> allow linking FFmpeg (when compiled with CONFIG_LIBV4L2) to libcamera
> instead of libv4l2, but Debian doesn't set CONFIG_LIBV4L2, so this isn't
> an immediate solution to the problem.

Native would be nice in my book, but isn't always as straightforward.

> Same thing for motion, except it has no libv4l2 support. The V4L2 compat
> layer could still be used with LD_PRELOAD, but that's not a great
> solution. Native libcamera support would make more sense (or possibly
> even GStreamer support, I don't know if upstream would accept that).

The main bit missing with the compat layer is controls. AIUI fixing
that isn't a priority.
Things like Motion being used for security purposes need to be able to
set properties like exposure modes in order to achieve usable images
under changing lighting conditions.

> In a side note, how do the above applications work today on Raspberry Pi
> platforms that use a sensor not supported by the legacy camera stack ?

They don't for raw sensors, but do work with USB webcams.
Raspberry Pi have sold camera modules since 2013 (OV5647), with v2
(IMX219) coming along in 2016, and HQ (IMX477) in 2020. Linux has only
had frameworks to sensibly support raw image sensors since libcamera
came on the scene around 2019.

> > Do you wish to make any predictions as to when that would be
> > achievable? Or even when a v1.0 release of libcamera is going to
> > happen?
>
> Now that we have started tagging releases, we've also decided to publish
> a roadmap with the development still needed to stabilize the API. We'll
> likely start working on it this month.
>
> > Dropping anything prior to those points would be rather premature in my book.
>
> Something I forgot to mention is that there should be no issue at all
> keeping bcm2835-camera fully supported in the Raspberry Pi downstream
> kernel for a longer period of time. It's in upstream that I don't think
> it should be destaged, as it's already considered legacy and should be
> phased out. Do you know if there are users of that driver with a
> mainline kernel ?

There are a number of distros that use mainline kernels rather than
our vendor kernel. I don't follow distros in detail, but I believe
both Gentoo and Opensuse fall into that category.
Any users of the camera module under those will be using the mainline
bcm2835-camera module. I have no stats for who uses those on a Pi.

> > The TODOs on bcm2835-camera are:
> > 1) Zero copy. That comes almost for free as bcm2835-isp already does
> > this, but it does rely on vcsm-cma.
> > The main reason I haven't pushed it is that it then requires
> > reasonable amounts of CMA heap for all the buffers, which until
> > recently haven't been present in the default configurations. With the
> > vc4 DRM driver now being default (at least for the vendor kernel) and
> > also requiring CMA, making the change makes more sense.
> > AFAIK there is no easy way to have one driver choosing between using
> > vb2_vmalloc_memops and vb2_dma_contig_memops at runtime, but I may be
> > wrong.
>
> Some drivers use a module parameter for that, but that's not great.

I had thought of a module parameter, but it becomes yet another
configuration thing to get incorrectly set.
Perhaps Kconfig based on the setting for CMA_SIZE_MBYTES, but that can
be overridden.

Dave

> > Actually bcm2835_defconfig appears to only allocate a 32MB CMA heap,
> > so perhaps we don't get very far.
> >
> > 2) This isn't workable within the current V4L2 frameworks. The
> > multi-planar V4L2 pixel formats are currently allocated as independent
> > buffers for each plane, whereas the firmware needs a single buffer
> > with (currently) specific offsets for the chroma planes. The
> > V4L2/videobuf2 core changes required to implement that are going to be
> > significant, and have minimal gain.
> > The specific stride handling is already dealt with (set bytesperline
> > appropriately), it's the padding of the height to a multiple of 16
> > before the chroma planes on YUV420 and NV12 formats that require the
> > firmware to do a small amount of repacking. The performance hit is
> > actually minimal anyway.
> >
> > If bcm2835-camera is the only thing holding back vc04_services, then I
> > can have a look at it.
>
> I'll let Umang comment on whether it's holding vc04_services back, but
> my understanding it that we could destage vc04_services while keeping
> bcm2835-camera in staging for the time being. If anyone disagrees with
> that, please let me know.
>
> > > > [1]: https://lore.kernel.org/linux-media/[email protected]/
> > > >
> > > > >>>>
> > > > >>>> I see two TODO files in vc04_services:
> > > > >>>> ./bcm2835-camera/TODO
> > > > >>>> ./interface/TODO
> > > > >>>>
> > > > >>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
> > > > >>>> itself. So that's address in the series. The other remaining one -
> > > > >>>> I will need to take a deeper look before commenting on it.
> > > > >>>>
> > > > >>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
> > > > >>>> a cursory reading of them suggests that these apply to *all*
> > > > >>>> vc04_services components? Am I right?
> > > > >>>
> > > > >>> Actually these applies just for the interfaces directory. Some of
> > > > >>> them could apply to the services, but this is no priority.
> > > > >>
> > > > >> By no priority, you mean this doesn't affect the criteria required to
> > > > >> ful-fill to get these out of staging?
> > > > >
> > > > > Correct
> > > > >
> > > > >>>> Are these are the specific bits of cleanup you are referring to in
> > > > >>>> your comment?
> > > > >>>
> > > > >>> You mean about bcm2835-isp? There were too many changes to vchiq
> > > > >>> that i don't remember them all. The first that come to my mind was
> > > > >>> those fancy comment sections which is not kernel coding style. It
> > > > >>> has been removed.
> > > > >>
> > > > >> No, I don't mean the bcm2835-isp changes (those are upcoming /
> > > > >> out-of-tree still so...). I mean what are the specific bits / points
> > > > >> that needs to be addressed to get vc04_services out of the staging.
> > > > >
> > > > > These were the points which i mentioned in my last email. They came
> > > > > from interface/TODO.
> > > > >
> > > > >> You have mentioned it above now, so I'll follow up on those.
> > > > >
> > > > > That would be great :)
> > > > >
> > > > >> The many vchiq changes you referred to above comment (that you don't
> > > > >> remember) are from [1] as well or some other series ?
> > > > >
> > > > > Sorry, for the confusing. The many changes i refer were the dozens of
> > > > > clean up patches for vc04_interfaces in mainline staging since the
> > > > > last years. [1] was just a single patch which has been accepted yet.
> > > >
> > > > Ah I see. There are many others that I've to dig out then. Thanks for
> > > > clarifying!
> > > >
> > > > >>> [1] -
> > > > >>> https://lore.kernel.org/linux-staging/[email protected]/
> > > > >>>
> > > > >>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
> > > > >>>>>
> > > > >>>>> Just my two cents
> > > > >>>>> Stefan
>
> --
> Regards,
>
> Laurent Pinchart

2022-12-02 12:53:14

[permalink] [raw]

Subject: Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support

Hi Laurent,

Am 02.12.22 um 13:10 schrieb Laurent Pinchart:
> Hi Dave,
>
> On Fri, Dec 02, 2022 at 11:23:29AM +0000, Dave Stevenson wrote:
>> Dropping anything prior to those points would be rather premature in my book.
> Something I forgot to mention is that there should be no issue at all
> keeping bcm2835-camera fully supported in the Raspberry Pi downstream
> kernel for a longer period of time. It's in upstream that I don't think
> it should be destaged, as it's already considered legacy and should be
> phased out. Do you know if there are users of that driver with a
> mainline kernel ?
>
In Fedora there seems to be no official support [1], but openSuSE seems
to mention it [2].

[1] -
https://docs.fedoraproject.org/en-US/quick-docs/raspberry-pi/#_does_the_add_on_camera_work

[2] - https://en.opensuse.org/HCL:Raspberry_Pi_Camera_Modules

2022-12-02 12:53:42

[permalink] [raw]

Subject: Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support

Hi Dave,

Am 02.12.22 um 12:23 schrieb Dave Stevenson:
> Hi Laurent, Umang, and Stefan.
>
> On Fri, 2 Dec 2022 at 09:17, Laurent Pinchart
> <[email protected]> wrote:
>> Hi Umang,
>>
>> On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
>>> On 12/2/22 6:45 AM, Stefan Wahren wrote:
>>>> Am 30.11.22 um 11:58 schrieb Umang Jain:
>>>>> On 11/27/22 6:56 AM, Stefan Wahren wrote:
>>>>>> Am 26.11.22 um 17:26 schrieb Umang Jain:
>>>>>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
>>>>>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
>>>>>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
>>>>>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
>>>>>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
>>>>>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
>>>>>>>>> with multistream support to get into mainline. Hence only the ISP driver
>>>>>>>>> will remain the primary area of this series.
>>>>>>>> thanks for working on this. But honestly i would prefer that vchiq
>>>>>>>> comes out of staging before adding more features. As Greg said
>>>>>>>> some time ago staging is not a place to "dump code and run away".
>>>>>>>> These new files are in the same bad shape as the rest of vc04
>>>>>>>> before the clean-up here in staging started.
>>>>>>> Certainly, I am not here to do that - but I am still learning the ropes.
>>>>>> no problem.
>>>>>>
>>>>>>> If the staging issue is becoming a blocker for bcm2835-isp going
>>>>>>> upstream, I would be happy to help here! Though I must mention that
>>>>>>> I still have limited visibility so my aim would be to chart out a
>>>>>>> plan of things needed to be done to get vc04_services out of staging!
>>>>>> The vchiq driver is in staging since 2016, so every step forwards is
>>>>>> good. Unfortunately all of the low hanging fruits has been gathered.
>>>>>>
>>>>>> For me the most important, but not to tricky steps to get vchiq out
>>>>>> of staging would be:
>>>>>>
>>>>>> * Cleanup logging mechanism
>>>>>>
>>>>>> * Get rid of custom function return values
>>>>>>
>>>>>> There was already an attempt for this [1]
>>>>>>
>>>>>> * Get rid of all non essential global structures and create a proper per
>>>>>> device structure
>>>>>>
>>>>>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
>>>>>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
>>>>>>>> staging. It just binds more resources on a new feature.
>>>>> bcm2835-camera is the legacy camera stack which probably need to be
>>>>> dropped from hereon...
>>>> I don't not know if there any users left, so i would be careful here.
>>>> Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
>>>> dumb question but i'm not expert here.
>>> I am careful too here and probably need Input from RaspberryPi in order
>>> to proceed to drop it. But from my perspective - bcm2835-camera is _not_
>>> going out of staging - it'll either sit here (or probably dropped) as
>>> statied from [1]
>>>
>>> ```
>>> + * There are two camera drivers in the kernel for BCM283x - this one
>>> + * and bcm2835-camera (currently in staging).
>>> ```
>>>
>>> The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
>>> (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
>>> mentioned in my cover the testing of bcm2835-isp happened on top of
>>> unicam patches.
>> To be accurate, the bcm2835-camera driver supports the VC4
>> firmware-based camera stack. In that setup, the camera sensors (OV5647
>> or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
>> firmware, which provides a high-level interface towards the kernel. This
>> architecture has been replaced by Linux-side control of the camera
>> sensors (through existing drivers in drivers/media/i2c/), Unicam
>> (through the driver from [1]) and ISP (through this driver). Moving
>> control to the Linux side requires complex processing in userspace,
>> handled by libcamera.
>>
>> bcm2835-camera is thus replaced by multiple drivers combined with
>> libcamera, and that is the camera stack that is shipped by Raspberry Pi
>> these days. While this may affect some userspace use cases), we will not
>> work on destaging bcm2835-camera, and as far as I'm aware, nobody else
>> is planning to do so either. I don't mind much if the driver stays in
>> staging for some more time, but I'd rather drop it if possible.
Thanks for clarification. Okay, so Unicam + bcm2835-isp are able to
handle the old camera (OV5647)?
> It would be reasonable to drop it at the point that Libcamera can work
> to a similar level with at least the following list of applications:
> - FFmpeg
> - Gstreamer
> - Chromium
> - Firefox
> - Motion
> And that still leaves a huge number of existing V4L2 apps out in the cold.
>
> Do you wish to make any predictions as to when that would be
> achievable? Or even when a v1.0 release of libcamera is going to
> happen?
> Dropping anything prior to those points would be rather premature in my book.
>
>
> The TODOs on bcm2835-camera are:
> 1) Zero copy. That comes almost for free as bcm2835-isp already does
> this, but it does rely on vcsm-cma.
> The main reason I haven't pushed it is that it then requires
> reasonable amounts of CMA heap for all the buffers, which until
> recently haven't been present in the default configurations. With the
> vc4 DRM driver now being default (at least for the vendor kernel) and
> also requiring CMA, making the change makes more sense.
> AFAIK there is no easy way to have one driver choosing between using
> vb2_vmalloc_memops and vb2_dma_contig_memops at runtime, but I may be
> wrong.
> Actually bcm2835_defconfig appears to only allocate a 32MB CMA heap,
> so perhaps we don't get very far.
CMA configuration should actually happen in device tree or kernel
cmdline. The bcm2835_defconfig is limited to make it work even with the
original Pi.
>
> 2) This isn't workable within the current V4L2 frameworks. The
> multi-planar V4L2 pixel formats are currently allocated as independent
> buffers for each plane, whereas the firmware needs a single buffer
> with (currently) specific offsets for the chroma planes. The
> V4L2/videobuf2 core changes required to implement that are going to be
> significant, and have minimal gain.
> The specific stride handling is already dealt with (set bytesperline
> appropriately), it's the padding of the height to a multiple of 16
> before the chroma planes on YUV420 and NV12 formats that require the
> firmware to do a small amount of repacking. The performance hit is
> actually minimal anyway.
>
> If bcm2835-camera is the only thing holding back vc04_services, then I
> can have a look at it.

No, it's the vchiq interface which needs the work.

Thanks Stefan

>
> Dave
>
>>> [1]: https://lore.kernel.org/linux-media/[email protected]/
>>>
>>>>>>> I see two TODO files in vc04_services:
>>>>>>> ./bcm2835-camera/TODO
>>>>>>> ./interface/TODO
>>>>>>>
>>>>>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
>>>>>>> itself. So that's address in the series. The other remaining one -
>>>>>>> I will need to take a deeper look before commenting on it.
>>>>>>>
>>>>>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
>>>>>>> a cursory reading of them suggests that these apply to *all*
>>>>>>> vc04_services components? Am I right?
>>>>>> Actually these applies just for the interfaces directory. Some of
>>>>>> them could apply to the services, but this is no priority.
>>>>> By no priority, you mean this doesn't affect the criteria required to
>>>>> ful-fill to get these out of staging?
>>>> Correct
>>>>
>>>>>>> Are these are the specific bits of cleanup you are referring to in
>>>>>>> your comment?
>>>>>> You mean about bcm2835-isp? There were too many changes to vchiq
>>>>>> that i don't remember them all. The first that come to my mind was
>>>>>> those fancy comment sections which is not kernel coding style. It
>>>>>> has been removed.
>>>>> No, I don't mean the bcm2835-isp changes (those are upcoming /
>>>>> out-of-tree still so...). I mean what are the specific bits / points
>>>>> that needs to be addressed to get vc04_services out of the staging.
>>>> These were the points which i mentioned in my last email. They came
>>>> from interface/TODO.
>>>>
>>>>> You have mentioned it above now, so I'll follow up on those.
>>>> That would be great :)
>>>>
>>>>> The many vchiq changes you referred to above comment (that you don't
>>>>> remember) are from [1] as well or some other series ?
>>>> Sorry, for the confusing. The many changes i refer were the dozens of
>>>> clean up patches for vc04_interfaces in mainline staging since the
>>>> last years. [1] was just a single patch which has been accepted yet.
>>> Ah I see. There are many others that I've to dig out then. Thanks for
>>> clarifying!
>>>
>>>>>> [1] -
>>>>>> https://lore.kernel.org/linux-staging/[email protected]/
>>>>>>
>>>>>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
>>>>>>>>
>>>>>>>> Just my two cents
>>>>>>>> Stefan
>> --
>> Regards,
>>
>> Laurent Pinchart
> _______________________________________________
> linux-arm-kernel mailing list
> [email protected]
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

2022-12-02 13:34:26

by Peter Robinson

[permalink] [raw]

Subject: Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support

Hi Folks,

> > On Fri, Dec 02, 2022 at 11:23:29AM +0000, Dave Stevenson wrote:
> >> Dropping anything prior to those points would be rather premature in my book.
> > Something I forgot to mention is that there should be no issue at all
> > keeping bcm2835-camera fully supported in the Raspberry Pi downstream
> > kernel for a longer period of time. It's in upstream that I don't think
> > it should be destaged, as it's already considered legacy and should be
> > phased out. Do you know if there are users of that driver with a
> > mainline kernel ?
> >
> In Fedora there seems to be no official support [1], but openSuSE seems
> to mention it [2].

It's enabled in Fedora but I've never had a huge amount of luck with
it, I would be OK with the firmware driver to disappear when ever.

From our point of view I'd sooner we can just move to the unicam in
kernel one sooner rather than later even swapping one for the other in
staging to get the libcamera one in and improve it in kernel via the
staging process, I feel that would be better use of the very few
cycles I have for RPi bits.

Peter

> [1] -
> https://docs.fedoraproject.org/en-US/quick-docs/raspberry-pi/#_does_the_add_on_camera_work
>
> [2] - https://en.opensuse.org/HCL:Raspberry_Pi_Camera_Modules
>

2022-12-02 13:38:25

[permalink] [raw]

Subject: Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support

Hi Dave,

On Fri, Dec 02, 2022 at 12:38:09PM +0000, Dave Stevenson wrote:
> On Fri, 2 Dec 2022 at 12:10, Laurent Pinchart wrote:
> > On Fri, Dec 02, 2022 at 11:23:29AM +0000, Dave Stevenson wrote:
> > > On Fri, 2 Dec 2022 at 09:17, Laurent Pinchart wrote:
> > > > On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
> > > > > On 12/2/22 6:45 AM, Stefan Wahren wrote:
> > > > > > Am 30.11.22 um 11:58 schrieb Umang Jain:
> > > > > >> On 11/27/22 6:56 AM, Stefan Wahren wrote:
> > > > > >>> Am 26.11.22 um 17:26 schrieb Umang Jain:
> > > > > >>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
> > > > > >>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
> > > > > >>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> > > > > >>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> > > > > >>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> > > > > >>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
> > > > > >>>>>> with multistream support to get into mainline. Hence only the ISP driver
> > > > > >>>>>> will remain the primary area of this series.
> > > > > >>>>>
> > > > > >>>>> thanks for working on this. But honestly i would prefer that vchiq
> > > > > >>>>> comes out of staging before adding more features. As Greg said
> > > > > >>>>> some time ago staging is not a place to "dump code and run away".
> > > > > >>>>> These new files are in the same bad shape as the rest of vc04
> > > > > >>>>> before the clean-up here in staging started.
> > > > > >>>>
> > > > > >>>> Certainly, I am not here to do that - but I am still learning the ropes.
> > > > > >>>
> > > > > >>> no problem.
> > > > > >>>
> > > > > >>>> If the staging issue is becoming a blocker for bcm2835-isp going
> > > > > >>>> upstream, I would be happy to help here! Though I must mention that
> > > > > >>>> I still have limited visibility so my aim would be to chart out a
> > > > > >>>> plan of things needed to be done to get vc04_services out of staging!
> > > > > >>>
> > > > > >>> The vchiq driver is in staging since 2016, so every step forwards is
> > > > > >>> good. Unfortunately all of the low hanging fruits has been gathered.
> > > > > >>>
> > > > > >>> For me the most important, but not to tricky steps to get vchiq out
> > > > > >>> of staging would be:
> > > > > >>>
> > > > > >>> * Cleanup logging mechanism
> > > > > >>>
> > > > > >>> * Get rid of custom function return values
> > > > > >>>
> > > > > >>> There was already an attempt for this [1]
> > > > > >>>
> > > > > >>> * Get rid of all non essential global structures and create a proper per
> > > > > >>> device structure
> > > > > >>>
> > > > > >>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
> > > > > >>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
> > > > > >>>>> staging. It just binds more resources on a new feature.
> > > > > >>
> > > > > >> bcm2835-camera is the legacy camera stack which probably need to be
> > > > > >> dropped from hereon...
> > > > > >
> > > > > > I don't not know if there any users left, so i would be careful here.
> > > > > > Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
> > > > > > dumb question but i'm not expert here.
> > > > >
> > > > > I am careful too here and probably need Input from RaspberryPi in order
> > > > > to proceed to drop it. But from my perspective - bcm2835-camera is _not_
> > > > > going out of staging - it'll either sit here (or probably dropped) as
> > > > > statied from [1]
> > > > >
> > > > > ```
> > > > > + * There are two camera drivers in the kernel for BCM283x - this one
> > > > > + * and bcm2835-camera (currently in staging).
> > > > > ```
> > > > >
> > > > > The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
> > > > > (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
> > > > > mentioned in my cover the testing of bcm2835-isp happened on top of
> > > > > unicam patches.
> > > >
> > > > To be accurate, the bcm2835-camera driver supports the VC4
> > > > firmware-based camera stack. In that setup, the camera sensors (OV5647
> > > > or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
> > > > firmware, which provides a high-level interface towards the kernel. This
> > > > architecture has been replaced by Linux-side control of the camera
> > > > sensors (through existing drivers in drivers/media/i2c/), Unicam
> > > > (through the driver from [1]) and ISP (through this driver). Moving
> > > > control to the Linux side requires complex processing in userspace,
> > > > handled by libcamera.
> > > >
> > > > bcm2835-camera is thus replaced by multiple drivers combined with
> > > > libcamera, and that is the camera stack that is shipped by Raspberry Pi
> > > > these days. While this may affect some userspace use cases), we will not
> > > > work on destaging bcm2835-camera, and as far as I'm aware, nobody else
> > > > is planning to do so either. I don't mind much if the driver stays in
> > > > staging for some more time, but I'd rather drop it if possible.
> > >
> > > It would be reasonable to drop it at the point that Libcamera can work
> > > to a similar level with at least the following list of applications:
> > > - FFmpeg
> > > - Gstreamer
> > > - Chromium
> > > - Firefox
> > > - Motion
> > > And that still leaves a huge number of existing V4L2 apps out in the cold.
> >
> > That's exactly the kind of input we were looking for, thanks.
>
> Please note that those are only the 5 major ones that came immediately
> to mind as having significant numbers of users. There are obviously
> many more V4L2 apps around which currently "just work" if they use the
> API correctly.

Sure, I'll keep that in mind.

> > GStreamer is already addressed.
> >
> > Chromium and Firefox will go through PipeWire. There is a working
> > implementation in libwebrtc, Kieran may be able to comment on the
> > upstreaming state. It will take some time for distributions to switch,
> > and I can't predict the time line, but that seems to clearly be the
> > direction the Linux desktop is taking.
> >
> > I haven't looked at FFmpeg yet (maybe someone has ?). It probably makes
> > sense to add native libcamera support to FFmpeg, even if PipeWire
> > support could also make sense. It could also make sense to expose in the
> > libcamera V4L2 compat layer the libv4l2 API functions, as that would
> > allow linking FFmpeg (when compiled with CONFIG_LIBV4L2) to libcamera
> > instead of libv4l2, but Debian doesn't set CONFIG_LIBV4L2, so this isn't
> > an immediate solution to the problem.
>
> Native would be nice in my book, but isn't always as straightforward.

I agree, native would be good.

> > Same thing for motion, except it has no libv4l2 support. The V4L2 compat
> > layer could still be used with LD_PRELOAD, but that's not a great
> > solution. Native libcamera support would make more sense (or possibly
> > even GStreamer support, I don't know if upstream would accept that).
>
> The main bit missing with the compat layer is controls. AIUI fixing
> that isn't a priority.

Good point. That's something we should fix. Not a high priority indeed,
but still good to have (and it shouldn't take too long).

> Things like Motion being used for security purposes need to be able to
> set properties like exposure modes in order to achieve usable images
> under changing lighting conditions.
>
> > In a side note, how do the above applications work today on Raspberry Pi
> > platforms that use a sensor not supported by the legacy camera stack ?
>
> They don't for raw sensors, but do work with USB webcams.
> Raspberry Pi have sold camera modules since 2013 (OV5647), with v2
> (IMX219) coming along in 2016, and HQ (IMX477) in 2020. Linux has only
> had frameworks to sensibly support raw image sensors since libcamera
> came on the scene around 2019.

Do I understand correctly that the need to keep bcm2835-camera working
is for support of the v1 and v2 camera modules only ? Or are there other
use cases ?

> > > Do you wish to make any predictions as to when that would be
> > > achievable? Or even when a v1.0 release of libcamera is going to
> > > happen?
> >
> > Now that we have started tagging releases, we've also decided to publish
> > a roadmap with the development still needed to stabilize the API. We'll
> > likely start working on it this month.
> >
> > > Dropping anything prior to those points would be rather premature in my book.
> >
> > Something I forgot to mention is that there should be no issue at all
> > keeping bcm2835-camera fully supported in the Raspberry Pi downstream
> > kernel for a longer period of time. It's in upstream that I don't think
> > it should be destaged, as it's already considered legacy and should be
> > phased out. Do you know if there are users of that driver with a
> > mainline kernel ?
>
> There are a number of distros that use mainline kernels rather than
> our vendor kernel. I don't follow distros in detail, but I believe
> both Gentoo and Opensuse fall into that category.
> Any users of the camera module under those will be using the mainline
> bcm2835-camera module. I have no stats for who uses those on a Pi.

OK.

> > > The TODOs on bcm2835-camera are:
> > > 1) Zero copy. That comes almost for free as bcm2835-isp already does
> > > this, but it does rely on vcsm-cma.
> > > The main reason I haven't pushed it is that it then requires
> > > reasonable amounts of CMA heap for all the buffers, which until
> > > recently haven't been present in the default configurations. With the
> > > vc4 DRM driver now being default (at least for the vendor kernel) and
> > > also requiring CMA, making the change makes more sense.
> > > AFAIK there is no easy way to have one driver choosing between using
> > > vb2_vmalloc_memops and vb2_dma_contig_memops at runtime, but I may be
> > > wrong.
> >
> > Some drivers use a module parameter for that, but that's not great.
>
> I had thought of a module parameter, but it becomes yet another
> configuration thing to get incorrectly set.
> Perhaps Kconfig based on the setting for CMA_SIZE_MBYTES, but that can
> be overridden.

The trouble with vb2-vmalloc is that you'll end up copying data, which I
expect to cause various performance issues that may generate bug reports
difficult to address. Increasing the CMA size would be better.

> > > Actually bcm2835_defconfig appears to only allocate a 32MB CMA heap,
> > > so perhaps we don't get very far.
> > >
> > > 2) This isn't workable within the current V4L2 frameworks. The
> > > multi-planar V4L2 pixel formats are currently allocated as independent
> > > buffers for each plane, whereas the firmware needs a single buffer
> > > with (currently) specific offsets for the chroma planes. The
> > > V4L2/videobuf2 core changes required to implement that are going to be
> > > significant, and have minimal gain.
> > > The specific stride handling is already dealt with (set bytesperline
> > > appropriately), it's the padding of the height to a multiple of 16
> > > before the chroma planes on YUV420 and NV12 formats that require the
> > > firmware to do a small amount of repacking. The performance hit is
> > > actually minimal anyway.
> > >
> > > If bcm2835-camera is the only thing holding back vc04_services, then I
> > > can have a look at it.
> >
> > I'll let Umang comment on whether it's holding vc04_services back, but
> > my understanding it that we could destage vc04_services while keeping
> > bcm2835-camera in staging for the time being. If anyone disagrees with
> > that, please let me know.
> >
> > > > > [1]: https://lore.kernel.org/linux-media/[email protected]/
> > > > >
> > > > > >>>>
> > > > > >>>> I see two TODO files in vc04_services:
> > > > > >>>> ./bcm2835-camera/TODO
> > > > > >>>> ./interface/TODO
> > > > > >>>>
> > > > > >>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
> > > > > >>>> itself. So that's address in the series. The other remaining one -
> > > > > >>>> I will need to take a deeper look before commenting on it.
> > > > > >>>>
> > > > > >>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
> > > > > >>>> a cursory reading of them suggests that these apply to *all*
> > > > > >>>> vc04_services components? Am I right?
> > > > > >>>
> > > > > >>> Actually these applies just for the interfaces directory. Some of
> > > > > >>> them could apply to the services, but this is no priority.
> > > > > >>
> > > > > >> By no priority, you mean this doesn't affect the criteria required to
> > > > > >> ful-fill to get these out of staging?
> > > > > >
> > > > > > Correct
> > > > > >
> > > > > >>>> Are these are the specific bits of cleanup you are referring to in
> > > > > >>>> your comment?
> > > > > >>>
> > > > > >>> You mean about bcm2835-isp? There were too many changes to vchiq
> > > > > >>> that i don't remember them all. The first that come to my mind was
> > > > > >>> those fancy comment sections which is not kernel coding style. It
> > > > > >>> has been removed.
> > > > > >>
> > > > > >> No, I don't mean the bcm2835-isp changes (those are upcoming /
> > > > > >> out-of-tree still so...). I mean what are the specific bits / points
> > > > > >> that needs to be addressed to get vc04_services out of the staging.
> > > > > >
> > > > > > These were the points which i mentioned in my last email. They came
> > > > > > from interface/TODO.
> > > > > >
> > > > > >> You have mentioned it above now, so I'll follow up on those.
> > > > > >
> > > > > > That would be great :)
> > > > > >
> > > > > >> The many vchiq changes you referred to above comment (that you don't
> > > > > >> remember) are from [1] as well or some other series ?
> > > > > >
> > > > > > Sorry, for the confusing. The many changes i refer were the dozens of
> > > > > > clean up patches for vc04_interfaces in mainline staging since the
> > > > > > last years. [1] was just a single patch which has been accepted yet.
> > > > >
> > > > > Ah I see. There are many others that I've to dig out then. Thanks for
> > > > > clarifying!
> > > > >
> > > > > >>> [1] -
> > > > > >>> https://lore.kernel.org/linux-staging/[email protected]/
> > > > > >>>
> > > > > >>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
> > > > > >>>>>
> > > > > >>>>> Just my two cents
> > > > > >>>>> Stefan

--
Regards,

Laurent Pinchart

2022-12-02 13:38:37

[permalink] [raw]

Subject: Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support

Hi Stefan,

On Fri, Dec 02, 2022 at 01:41:53PM +0100, Stefan Wahren wrote:
> Am 02.12.22 um 12:23 schrieb Dave Stevenson:
> > On Fri, 2 Dec 2022 at 09:17, Laurent Pinchart wrote:
> >> On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
> >>> On 12/2/22 6:45 AM, Stefan Wahren wrote:
> >>>> Am 30.11.22 um 11:58 schrieb Umang Jain:
> >>>>> On 11/27/22 6:56 AM, Stefan Wahren wrote:
> >>>>>> Am 26.11.22 um 17:26 schrieb Umang Jain:
> >>>>>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
> >>>>>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
> >>>>>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> >>>>>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> >>>>>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> >>>>>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
> >>>>>>>>> with multistream support to get into mainline. Hence only the ISP driver
> >>>>>>>>> will remain the primary area of this series.
> >>>>>>>> thanks for working on this. But honestly i would prefer that vchiq
> >>>>>>>> comes out of staging before adding more features. As Greg said
> >>>>>>>> some time ago staging is not a place to "dump code and run away".
> >>>>>>>> These new files are in the same bad shape as the rest of vc04
> >>>>>>>> before the clean-up here in staging started.
> >>>>>>>
> >>>>>>> Certainly, I am not here to do that - but I am still learning the ropes.
> >>>>>>
> >>>>>> no problem.
> >>>>>>
> >>>>>>> If the staging issue is becoming a blocker for bcm2835-isp going
> >>>>>>> upstream, I would be happy to help here! Though I must mention that
> >>>>>>> I still have limited visibility so my aim would be to chart out a
> >>>>>>> plan of things needed to be done to get vc04_services out of staging!
> >>>>>>
> >>>>>> The vchiq driver is in staging since 2016, so every step forwards is
> >>>>>> good. Unfortunately all of the low hanging fruits has been gathered.
> >>>>>>
> >>>>>> For me the most important, but not to tricky steps to get vchiq out
> >>>>>> of staging would be:
> >>>>>>
> >>>>>> * Cleanup logging mechanism
> >>>>>>
> >>>>>> * Get rid of custom function return values
> >>>>>>
> >>>>>> There was already an attempt for this [1]
> >>>>>>
> >>>>>> * Get rid of all non essential global structures and create a proper per
> >>>>>> device structure
> >>>>>>
> >>>>>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
> >>>>>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
> >>>>>>>> staging. It just binds more resources on a new feature.
> >>>>>
> >>>>> bcm2835-camera is the legacy camera stack which probably need to be
> >>>>> dropped from hereon...
> >>>>
> >>>> I don't not know if there any users left, so i would be careful here.
> >>>> Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
> >>>> dumb question but i'm not expert here.
> >>>
> >>> I am careful too here and probably need Input from RaspberryPi in order
> >>> to proceed to drop it. But from my perspective - bcm2835-camera is _not_
> >>> going out of staging - it'll either sit here (or probably dropped) as
> >>> statied from [1]
> >>>
> >>> ```
> >>> + * There are two camera drivers in the kernel for BCM283x - this one
> >>> + * and bcm2835-camera (currently in staging).
> >>> ```
> >>>
> >>> The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
> >>> (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
> >>> mentioned in my cover the testing of bcm2835-isp happened on top of
> >>> unicam patches.
> >>
> >> To be accurate, the bcm2835-camera driver supports the VC4
> >> firmware-based camera stack. In that setup, the camera sensors (OV5647
> >> or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
> >> firmware, which provides a high-level interface towards the kernel. This
> >> architecture has been replaced by Linux-side control of the camera
> >> sensors (through existing drivers in drivers/media/i2c/), Unicam
> >> (through the driver from [1]) and ISP (through this driver). Moving
> >> control to the Linux side requires complex processing in userspace,
> >> handled by libcamera.
> >>
> >> bcm2835-camera is thus replaced by multiple drivers combined with
> >> libcamera, and that is the camera stack that is shipped by Raspberry Pi
> >> these days. While this may affect some userspace use cases), we will not
> >> work on destaging bcm2835-camera, and as far as I'm aware, nobody else
> >> is planning to do so either. I don't mind much if the driver stays in
> >> staging for some more time, but I'd rather drop it if possible.
>
> Thanks for clarification. Okay, so Unicam + bcm2835-isp are able to
> handle the old camera (OV5647)?

Yes, but only for applications using libcamera, directly or indirectly
(through GStreamer for instance, or the V4L2 LD_PRELOAD-ed compatibility
later), not natively through V4L2 only.

> > It would be reasonable to drop it at the point that Libcamera can work
> > to a similar level with at least the following list of applications:
> > - FFmpeg
> > - Gstreamer
> > - Chromium
> > - Firefox
> > - Motion
> > And that still leaves a huge number of existing V4L2 apps out in the cold.
> >
> > Do you wish to make any predictions as to when that would be
> > achievable? Or even when a v1.0 release of libcamera is going to
> > happen?
> > Dropping anything prior to those points would be rather premature in my book.
> >
> >
> > The TODOs on bcm2835-camera are:
> > 1) Zero copy. That comes almost for free as bcm2835-isp already does
> > this, but it does rely on vcsm-cma.
> > The main reason I haven't pushed it is that it then requires
> > reasonable amounts of CMA heap for all the buffers, which until
> > recently haven't been present in the default configurations. With the
> > vc4 DRM driver now being default (at least for the vendor kernel) and
> > also requiring CMA, making the change makes more sense.
> > AFAIK there is no easy way to have one driver choosing between using
> > vb2_vmalloc_memops and vb2_dma_contig_memops at runtime, but I may be
> > wrong.
> > Actually bcm2835_defconfig appears to only allocate a 32MB CMA heap,
> > so perhaps we don't get very far.
>
> CMA configuration should actually happen in device tree or kernel
> cmdline. The bcm2835_defconfig is limited to make it work even with the
> original Pi.
>
> > 2) This isn't workable within the current V4L2 frameworks. The
> > multi-planar V4L2 pixel formats are currently allocated as independent
> > buffers for each plane, whereas the firmware needs a single buffer
> > with (currently) specific offsets for the chroma planes. The
> > V4L2/videobuf2 core changes required to implement that are going to be
> > significant, and have minimal gain.
> > The specific stride handling is already dealt with (set bytesperline
> > appropriately), it's the padding of the height to a multiple of 16
> > before the chroma planes on YUV420 and NV12 formats that require the
> > firmware to do a small amount of repacking. The performance hit is
> > actually minimal anyway.
> >
> > If bcm2835-camera is the only thing holding back vc04_services, then I
> > can have a look at it.
>
> No, it's the vchiq interface which needs the work.
>
> >>> [1]: https://lore.kernel.org/linux-media/[email protected]/
> >>>
> >>>>>>> I see two TODO files in vc04_services:
> >>>>>>> ./bcm2835-camera/TODO
> >>>>>>> ./interface/TODO
> >>>>>>>
> >>>>>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
> >>>>>>> itself. So that's address in the series. The other remaining one -
> >>>>>>> I will need to take a deeper look before commenting on it.
> >>>>>>>
> >>>>>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
> >>>>>>> a cursory reading of them suggests that these apply to *all*
> >>>>>>> vc04_services components? Am I right?
> >>>>>>
> >>>>>> Actually these applies just for the interfaces directory. Some of
> >>>>>> them could apply to the services, but this is no priority.
> >>>>>
> >>>>> By no priority, you mean this doesn't affect the criteria required to
> >>>>> ful-fill to get these out of staging?
> >>>>
> >>>> Correct
> >>>>
> >>>>>>> Are these are the specific bits of cleanup you are referring to in
> >>>>>>> your comment?
> >>>>>>
> >>>>>> You mean about bcm2835-isp? There were too many changes to vchiq
> >>>>>> that i don't remember them all. The first that come to my mind was
> >>>>>> those fancy comment sections which is not kernel coding style. It
> >>>>>> has been removed.
> >>>>>
> >>>>> No, I don't mean the bcm2835-isp changes (those are upcoming /
> >>>>> out-of-tree still so...). I mean what are the specific bits / points
> >>>>> that needs to be addressed to get vc04_services out of the staging.
> >>>>
> >>>> These were the points which i mentioned in my last email. They came
> >>>> from interface/TODO.
> >>>>
> >>>>> You have mentioned it above now, so I'll follow up on those.
> >>>>
> >>>> That would be great :)
> >>>>
> >>>>> The many vchiq changes you referred to above comment (that you don't
> >>>>> remember) are from [1] as well or some other series ?
> >>>> Sorry, for the confusing. The many changes i refer were the dozens of
> >>>> clean up patches for vc04_interfaces in mainline staging since the
> >>>> last years. [1] was just a single patch which has been accepted yet.
> >>>
> >>> Ah I see. There are many others that I've to dig out then. Thanks for
> >>> clarifying!
> >>>
> >>>>>> [1] -
> >>>>>> https://lore.kernel.org/linux-staging/[email protected]/
> >>>>>>
> >>>>>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
> >>>>>>>>
> >>>>>>>> Just my two cents
> >>>>>>>> Stefan

--
Regards,

Laurent Pinchart

2022-12-02 14:20:17

[permalink] [raw]

Subject: Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support

Hi Stefan

On Fri, 2 Dec 2022 at 12:41, Stefan Wahren <[email protected]> wrote:
>
> Hi Dave,
>
> Am 02.12.22 um 12:23 schrieb Dave Stevenson:
> > Hi Laurent, Umang, and Stefan.
> >
> > On Fri, 2 Dec 2022 at 09:17, Laurent Pinchart
> > <[email protected]> wrote:
> >> Hi Umang,
> >>
> >> On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
> >>> On 12/2/22 6:45 AM, Stefan Wahren wrote:
> >>>> Am 30.11.22 um 11:58 schrieb Umang Jain:
> >>>>> On 11/27/22 6:56 AM, Stefan Wahren wrote:
> >>>>>> Am 26.11.22 um 17:26 schrieb Umang Jain:
> >>>>>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
> >>>>>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
> >>>>>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> >>>>>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> >>>>>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> >>>>>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
> >>>>>>>>> with multistream support to get into mainline. Hence only the ISP driver
> >>>>>>>>> will remain the primary area of this series.
> >>>>>>>> thanks for working on this. But honestly i would prefer that vchiq
> >>>>>>>> comes out of staging before adding more features. As Greg said
> >>>>>>>> some time ago staging is not a place to "dump code and run away".
> >>>>>>>> These new files are in the same bad shape as the rest of vc04
> >>>>>>>> before the clean-up here in staging started.
> >>>>>>> Certainly, I am not here to do that - but I am still learning the ropes.
> >>>>>> no problem.
> >>>>>>
> >>>>>>> If the staging issue is becoming a blocker for bcm2835-isp going
> >>>>>>> upstream, I would be happy to help here! Though I must mention that
> >>>>>>> I still have limited visibility so my aim would be to chart out a
> >>>>>>> plan of things needed to be done to get vc04_services out of staging!
> >>>>>> The vchiq driver is in staging since 2016, so every step forwards is
> >>>>>> good. Unfortunately all of the low hanging fruits has been gathered.
> >>>>>>
> >>>>>> For me the most important, but not to tricky steps to get vchiq out
> >>>>>> of staging would be:
> >>>>>>
> >>>>>> * Cleanup logging mechanism
> >>>>>>
> >>>>>> * Get rid of custom function return values
> >>>>>>
> >>>>>> There was already an attempt for this [1]
> >>>>>>
> >>>>>> * Get rid of all non essential global structures and create a proper per
> >>>>>> device structure
> >>>>>>
> >>>>>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
> >>>>>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
> >>>>>>>> staging. It just binds more resources on a new feature.
> >>>>> bcm2835-camera is the legacy camera stack which probably need to be
> >>>>> dropped from hereon...
> >>>> I don't not know if there any users left, so i would be careful here.
> >>>> Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
> >>>> dumb question but i'm not expert here.
> >>> I am careful too here and probably need Input from RaspberryPi in order
> >>> to proceed to drop it. But from my perspective - bcm2835-camera is _not_
> >>> going out of staging - it'll either sit here (or probably dropped) as
> >>> statied from [1]
> >>>
> >>> ```
> >>> + * There are two camera drivers in the kernel for BCM283x - this one
> >>> + * and bcm2835-camera (currently in staging).
> >>> ```
> >>>
> >>> The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
> >>> (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
> >>> mentioned in my cover the testing of bcm2835-isp happened on top of
> >>> unicam patches.
> >> To be accurate, the bcm2835-camera driver supports the VC4
> >> firmware-based camera stack. In that setup, the camera sensors (OV5647
> >> or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
> >> firmware, which provides a high-level interface towards the kernel. This
> >> architecture has been replaced by Linux-side control of the camera
> >> sensors (through existing drivers in drivers/media/i2c/), Unicam
> >> (through the driver from [1]) and ISP (through this driver). Moving
> >> control to the Linux side requires complex processing in userspace,
> >> handled by libcamera.
> >>
> >> bcm2835-camera is thus replaced by multiple drivers combined with
> >> libcamera, and that is the camera stack that is shipped by Raspberry Pi
> >> these days. While this may affect some userspace use cases), we will not
> >> work on destaging bcm2835-camera, and as far as I'm aware, nobody else
> >> is planning to do so either. I don't mind much if the driver stays in
> >> staging for some more time, but I'd rather drop it if possible.
> Thanks for clarification. Okay, so Unicam + bcm2835-isp are able to
> handle the old camera (OV5647)?

There are kernel drivers in mainline for ov5647 (v1) and imx219 (v2)
cameras, with imx477 (HQ camera) due to be done. Connected to
bcm2835-unicam that gives you raw images from these sensors.

However you have to use libcamera to get useful images with these
drivers - raw Bayer isn't easy to consume.
Libcamera has support for all three (and more) sensors in the RPi IPA
(image processing algorithms), and will make use of bcm2835-isp to
process the images appropriately.

> > It would be reasonable to drop it at the point that Libcamera can work
> > to a similar level with at least the following list of applications:
> > - FFmpeg
> > - Gstreamer
> > - Chromium
> > - Firefox
> > - Motion
> > And that still leaves a huge number of existing V4L2 apps out in the cold.
> >
> > Do you wish to make any predictions as to when that would be
> > achievable? Or even when a v1.0 release of libcamera is going to
> > happen?
> > Dropping anything prior to those points would be rather premature in my book.
> >
> >
> > The TODOs on bcm2835-camera are:
> > 1) Zero copy. That comes almost for free as bcm2835-isp already does
> > this, but it does rely on vcsm-cma.
> > The main reason I haven't pushed it is that it then requires
> > reasonable amounts of CMA heap for all the buffers, which until
> > recently haven't been present in the default configurations. With the
> > vc4 DRM driver now being default (at least for the vendor kernel) and
> > also requiring CMA, making the change makes more sense.
> > AFAIK there is no easy way to have one driver choosing between using
> > vb2_vmalloc_memops and vb2_dma_contig_memops at runtime, but I may be
> > wrong.
> > Actually bcm2835_defconfig appears to only allocate a 32MB CMA heap,
> > so perhaps we don't get very far.
> CMA configuration should actually happen in device tree or kernel
> cmdline. The bcm2835_defconfig is limited to make it work even with the
> original Pi.

It's the original 256MB Pis that are the issue as memory is so tight.
Losing a bigger chunk to CMA is a problem there.

I see that bcm283x.dtsi sets it to 64MB, which is probably sufficient
for most bcm2835-camera use cases, but leaves me worried that you'll
steal it from other things.

> >
> > 2) This isn't workable within the current V4L2 frameworks. The
> > multi-planar V4L2 pixel formats are currently allocated as independent
> > buffers for each plane, whereas the firmware needs a single buffer
> > with (currently) specific offsets for the chroma planes. The
> > V4L2/videobuf2 core changes required to implement that are going to be
> > significant, and have minimal gain.
> > The specific stride handling is already dealt with (set bytesperline
> > appropriately), it's the padding of the height to a multiple of 16
> > before the chroma planes on YUV420 and NV12 formats that require the
> > firmware to do a small amount of repacking. The performance hit is
> > actually minimal anyway.
> >
> > If bcm2835-camera is the only thing holding back vc04_services, then I
> > can have a look at it.
>
> No, it's the vchiq interface which needs the work.

OK, I'll liaise with Umang and may look to deal with a couple of them.
Things like documenting the memory barriers probably falls to Phil.

Dave

> Thanks Stefan
>
> >
> > Dave
> >
> >>> [1]: https://lore.kernel.org/linux-media/[email protected]/
> >>>
> >>>>>>> I see two TODO files in vc04_services:
> >>>>>>> ./bcm2835-camera/TODO
> >>>>>>> ./interface/TODO
> >>>>>>>
> >>>>>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
> >>>>>>> itself. So that's address in the series. The other remaining one -
> >>>>>>> I will need to take a deeper look before commenting on it.
> >>>>>>>
> >>>>>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
> >>>>>>> a cursory reading of them suggests that these apply to *all*
> >>>>>>> vc04_services components? Am I right?
> >>>>>> Actually these applies just for the interfaces directory. Some of
> >>>>>> them could apply to the services, but this is no priority.
> >>>>> By no priority, you mean this doesn't affect the criteria required to
> >>>>> ful-fill to get these out of staging?
> >>>> Correct
> >>>>
> >>>>>>> Are these are the specific bits of cleanup you are referring to in
> >>>>>>> your comment?
> >>>>>> You mean about bcm2835-isp? There were too many changes to vchiq
> >>>>>> that i don't remember them all. The first that come to my mind was
> >>>>>> those fancy comment sections which is not kernel coding style. It
> >>>>>> has been removed.
> >>>>> No, I don't mean the bcm2835-isp changes (those are upcoming /
> >>>>> out-of-tree still so...). I mean what are the specific bits / points
> >>>>> that needs to be addressed to get vc04_services out of the staging.
> >>>> These were the points which i mentioned in my last email. They came
> >>>> from interface/TODO.
> >>>>
> >>>>> You have mentioned it above now, so I'll follow up on those.
> >>>> That would be great :)
> >>>>
> >>>>> The many vchiq changes you referred to above comment (that you don't
> >>>>> remember) are from [1] as well or some other series ?
> >>>> Sorry, for the confusing. The many changes i refer were the dozens of
> >>>> clean up patches for vc04_interfaces in mainline staging since the
> >>>> last years. [1] was just a single patch which has been accepted yet.
> >>> Ah I see. There are many others that I've to dig out then. Thanks for
> >>> clarifying!
> >>>
> >>>>>> [1] -
> >>>>>> https://lore.kernel.org/linux-staging/[email protected]/
> >>>>>>
> >>>>>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
> >>>>>>>>
> >>>>>>>> Just my two cents
> >>>>>>>> Stefan
> >> --
> >> Regards,
> >>
> >> Laurent Pinchart
> > _______________________________________________
> > linux-arm-kernel mailing list
> > [email protected]
> > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

2022-12-03 13:57:12