LinuxLists.cc - [PATCH v9 00/13] media: atmel: atmel-isc: implement media controller

[permalink] [raw]

Subject: [PATCH v9 08/13] media: atmel: atmel-isc: change format propagation to subdev into only verification

As a top MC video driver, the atmel-isc should not propagate the format to the
subdevice, it should rather check at start_streaming() time if the subdev is properly
configured with a compatible format.
Removed the whole format finding logic, and reworked the format verification
at start_streaming time, such that the ISC will return an error if the subdevice
is not properly configured. To achieve this, media_pipeline_start
is called and a link_validate callback is created to check the formats.
With this being done, the module parameter 'sensor_preferred' makes no sense
anymore. The ISC should not decide which format the sensor is using. The
ISC should only cope with the situation and inform userspace if the streaming
is possible in the current configuration.
The redesign of the format propagation has also risen the question of the
enumfmt callback. If enumfmt is called with an mbus_code, the enumfmt handler
should only return the formats that are supported for this mbus_code.
Otherwise, the enumfmt will report all the formats that the ISC could output.
With this rework, the dynamic list of user formats is removed. It makes no
more sense to identify at complete time which formats the sensor could emit,
and add those into a separate dynamic list.
The ISC will start with a simple preconfigured default format, and at
link validate time, decide whether it can use the format that is configured
on the sink or not.

Signed-off-by: Eugen Hristev <[email protected]>
Reviewed-by: Jacopo Mondi <[email protected]>
---
Changes in v9:
- isc_link_validate now static

Changes in v7:
- minor typos as suggested by Jacopo
- small changes, reduce some indentation, modified an index, as suggested by
Jacopo

Changes in v6:
- reworked a bit enum_fmt as suggested by Jacopo

Changes in v5:
- removed user_formats dynamic list as it is now pointless
- greatly simplified the enum_fmt function
- removed some init code that was useless now

Changes in v4:
- moved validation code into link_validate and used media_pipeline_start
- merged this patch with the enum_fmt patch which was previously in v3 of
the series

Changes in v3:
- clamp to maximum resolution once the frame size from the subdev is found
drivers/media/platform/atmel/atmel-isc-base.c | 412 ++++++++----------
.../media/platform/atmel/atmel-isc-scaler.c | 5 +
drivers/media/platform/atmel/atmel-isc.h | 13 +-
.../media/platform/atmel/atmel-sama5d2-isc.c | 20 +
.../media/platform/atmel/atmel-sama7g5-isc.c | 20 +
5 files changed, 236 insertions(+), 234 deletions(-)

diff --git a/drivers/media/platform/atmel/atmel-isc-base.c b/drivers/media/platform/atmel/atmel-isc-base.c
index ee1dda6707a0..fe2c0af58060 100644
--- a/drivers/media/platform/atmel/atmel-isc-base.c
+++ b/drivers/media/platform/atmel/atmel-isc-base.c
@@ -36,11 +36,6 @@ static unsigned int debug;
module_param(debug, int, 0644);
MODULE_PARM_DESC(debug, "debug level (0-2)");

-static unsigned int sensor_preferred = 1;
-module_param(sensor_preferred, uint, 0644);
-MODULE_PARM_DESC(sensor_preferred,
- "Sensor is preferred to output the specified format (1-on 0-off), default 1");
-
#define ISC_IS_FORMAT_RAW(mbus_code) \
(((mbus_code) & 0xf000) == 0x3000)

@@ -337,6 +332,10 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)
unsigned long flags;
int ret;

+ ret = media_pipeline_start(&isc->video_dev.entity, &isc->mpipe);
+ if (ret)
+ goto err_pipeline_start;
+
/* Enable stream on the sub device */
ret = v4l2_subdev_call(isc->current_subdev->sd, video, s_stream, 1);
if (ret && ret != -ENOIOCTLCMD) {
@@ -385,6 +384,9 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)
v4l2_subdev_call(isc->current_subdev->sd, video, s_stream, 0);

err_start_stream:
+ media_pipeline_stop(&isc->video_dev.entity);
+
+err_pipeline_start:
spin_lock_irqsave(&isc->dma_queue_lock, flags);
list_for_each_entry(buf, &isc->dma_queue, list)
vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_QUEUED);
@@ -423,6 +425,9 @@ static void isc_stop_streaming(struct vb2_queue *vq)
if (ret && ret != -ENOIOCTLCMD)
v4l2_err(&isc->v4l2_dev, "stream off failed in subdev\n");

+ /* Stop media pipeline */
+ media_pipeline_stop(&isc->video_dev.entity);
+
/* Release all active buffers */
spin_lock_irqsave(&isc->dma_queue_lock, flags);
if (unlikely(isc->cur_frm)) {
@@ -453,22 +458,6 @@ static void isc_buffer_queue(struct vb2_buffer *vb)
spin_unlock_irqrestore(&isc->dma_queue_lock, flags);
}

-static struct isc_format *find_format_by_fourcc(struct isc_device *isc,
- unsigned int fourcc)
-{
- unsigned int num_formats = isc->num_user_formats;
- struct isc_format *fmt;
- unsigned int i;
-
- for (i = 0; i < num_formats; i++) {
- fmt = isc->user_formats[i];
- if (fmt->fourcc == fourcc)
- return fmt;
- }
-
- return NULL;
-}
-
static const struct vb2_ops isc_vb2_ops = {
.queue_setup = isc_queue_setup,
.wait_prepare = vb2_ops_wait_prepare,
@@ -497,23 +486,57 @@ static int isc_enum_fmt_vid_cap(struct file *file, void *priv,
{
struct isc_device *isc = video_drvdata(file);
u32 index = f->index;
- u32 i, supported_index;
+ u32 i, supported_index = 0;
+ struct isc_format *fmt;
+
+ /*
+ * If we are not asked a specific mbus_code, we have to report all
+ * the formats that we can output.
+ */
+ if (!f->mbus_code) {
+ if (index >= isc->controller_formats_size)
+ return -EINVAL;

- if (index < isc->controller_formats_size) {
f->pixelformat = isc->controller_formats[index].fourcc;
+
+ return 0;
+ }
+
+ /*
+ * If a specific mbus_code is requested, check if we support
+ * this mbus_code as input for the ISC.
+ * If it's supported, then we report the corresponding pixelformat
+ * as first possible option for the ISC.
+ * E.g. mbus MEDIA_BUS_FMT_YUYV8_2X8 and report
+ * 'YUYV' (YUYV 4:2:2)
+ */
+ fmt = isc_find_format_by_code(isc, f->mbus_code, &i);
+ if (!fmt)
+ return -EINVAL;
+
+ if (!index) {
+ f->pixelformat = fmt->fourcc;
+
return 0;
}

- index -= isc->controller_formats_size;
+ supported_index++;

- supported_index = 0;
+ /* If the index is not raw, we don't have anymore formats to report */
+ if (!ISC_IS_FORMAT_RAW(f->mbus_code))
+ return -EINVAL;

- for (i = 0; i < isc->formats_list_size; i++) {
- if (!ISC_IS_FORMAT_RAW(isc->formats_list[i].mbus_code) ||
- !isc->formats_list[i].sd_support)
+ /*
+ * We are asked for a specific mbus code, which is raw.
+ * We have to search through the formats we can convert to.
+ * We have to skip the raw formats, we cannot convert to raw.
+ * E.g. 'AR12' (16-bit ARGB 4-4-4-4), 'AR15' (16-bit ARGB 1-5-5-5), etc.
+ */
+ for (i = 0; i < isc->controller_formats_size; i++) {
+ if (isc->controller_formats[i].raw)
continue;
- if (supported_index == index) {
- f->pixelformat = isc->formats_list[i].fourcc;
+ if (index == supported_index) {
+ f->pixelformat = isc->controller_formats[i].fourcc;
return 0;
}
supported_index++;
@@ -584,20 +607,30 @@ static int isc_try_validate_formats(struct isc_device *isc)
break;
default:
/* any other different formats are not supported */
+ v4l2_err(&isc->v4l2_dev, "Requested unsupported format.\n");
ret = -EINVAL;
}
v4l2_dbg(1, debug, &isc->v4l2_dev,
"Format validation, requested rgb=%u, yuv=%u, grey=%u, bayer=%u\n",
rgb, yuv, grey, bayer);

- /* we cannot output RAW if we do not receive RAW */
- if ((bayer) && !ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code))
+ if (bayer &&
+ !ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code)) {
+ v4l2_err(&isc->v4l2_dev, "Cannot output RAW if we do not receive RAW.\n");
return -EINVAL;
+ }

- /* we cannot output GREY if we do not receive RAW/GREY */
if (grey && !ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code) &&
- !ISC_IS_FORMAT_GREY(isc->try_config.sd_format->mbus_code))
+ !ISC_IS_FORMAT_GREY(isc->try_config.sd_format->mbus_code)) {
+ v4l2_err(&isc->v4l2_dev, "Cannot output GREY if we do not receive RAW/GREY.\n");
return -EINVAL;
+ }
+
+ if ((rgb || bayer || yuv) &&
+ ISC_IS_FORMAT_GREY(isc->try_config.sd_format->mbus_code)) {
+ v4l2_err(&isc->v4l2_dev, "Cannot convert GREY to another format.\n");
+ return -EINVAL;
+ }

return ret;
}
@@ -825,7 +858,7 @@ static void isc_try_fse(struct isc_device *isc,
* If we do not know yet which format the subdev is using, we cannot
* do anything.
*/
- if (!isc->try_config.sd_format)
+ if (!isc->config.sd_format)
return;

fse.code = isc->try_config.sd_format->mbus_code;
@@ -846,180 +879,140 @@ static void isc_try_fse(struct isc_device *isc,
}
}

-static int isc_try_fmt(struct isc_device *isc, struct v4l2_format *f,
- u32 *code)
+static int isc_try_fmt(struct isc_device *isc, struct v4l2_format *f)
{
- int i;
- struct isc_format *sd_fmt = NULL, *direct_fmt = NULL;
struct v4l2_pix_format *pixfmt = &f->fmt.pix;
- struct v4l2_subdev_pad_config pad_cfg = {};
- struct v4l2_subdev_state pad_state = {
- .pads = &pad_cfg
- };
- struct v4l2_subdev_format format = {
- .which = V4L2_SUBDEV_FORMAT_TRY,
- };
- u32 mbus_code;
- int ret;
- bool rlp_dma_direct_dump = false;
+ unsigned int i;

if (f->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
return -EINVAL;

- /* Step 1: find a RAW format that is supported */
- for (i = 0; i < isc->num_user_formats; i++) {
- if (ISC_IS_FORMAT_RAW(isc->user_formats[i]->mbus_code)) {
- sd_fmt = isc->user_formats[i];
+ isc->try_config.fourcc = isc->controller_formats[0].fourcc;
+
+ /* find if the format requested is supported */
+ for (i = 0; i < isc->controller_formats_size; i++)
+ if (isc->controller_formats[i].fourcc == pixfmt->pixelformat) {
+ isc->try_config.fourcc = pixfmt->pixelformat;
break;
}
- }
- /* Step 2: We can continue with this RAW format, or we can look
- * for better: maybe sensor supports directly what we need.
- */
- direct_fmt = find_format_by_fourcc(isc, pixfmt->pixelformat);
-
- /* Step 3: We have both. We decide given the module parameter which
- * one to use.
- */
- if (direct_fmt && sd_fmt && sensor_preferred)
- sd_fmt = direct_fmt;
-
- /* Step 4: we do not have RAW but we have a direct format. Use it. */
- if (direct_fmt && !sd_fmt)
- sd_fmt = direct_fmt;
-
- /* Step 5: if we are using a direct format, we need to package
- * everything as 8 bit data and just dump it
- */
- if (sd_fmt == direct_fmt)
- rlp_dma_direct_dump = true;
-
- /* Step 6: We have no format. This can happen if the userspace
- * requests some weird/invalid format.
- * In this case, default to whatever we have
- */
- if (!sd_fmt && !direct_fmt) {
- sd_fmt = isc->user_formats[isc->num_user_formats - 1];
- v4l2_dbg(1, debug, &isc->v4l2_dev,
- "Sensor not supporting %.4s, using %.4s\n",
- (char *)&pixfmt->pixelformat, (char *)&sd_fmt->fourcc);
- }
-
- if (!sd_fmt) {
- ret = -EINVAL;
- goto isc_try_fmt_err;
- }
-
- /* Step 7: Print out what we decided for debugging */
- v4l2_dbg(1, debug, &isc->v4l2_dev,
- "Preferring to have sensor using format %.4s\n",
- (char *)&sd_fmt->fourcc);

- /* Step 8: at this moment we decided which format the subdev will use */
- isc->try_config.sd_format = sd_fmt;
+ isc_try_configure_rlp_dma(isc, false);

/* Limit to Atmel ISC hardware capabilities */
- if (pixfmt->width > isc->max_width)
- pixfmt->width = isc->max_width;
- if (pixfmt->height > isc->max_height)
- pixfmt->height = isc->max_height;
-
- /*
- * The mbus format is the one the subdev outputs.
- * The pixels will be transferred in this format Sensor -> ISC
- */
- mbus_code = sd_fmt->mbus_code;
-
- /*
- * Validate formats. If the required format is not OK, default to raw.
- */
-
- isc->try_config.fourcc = pixfmt->pixelformat;
-
- if (isc_try_validate_formats(isc)) {
- pixfmt->pixelformat = isc->try_config.fourcc = sd_fmt->fourcc;
- /* Re-try to validate the new format */
- ret = isc_try_validate_formats(isc);
- if (ret)
- goto isc_try_fmt_err;
- }
-
- ret = isc_try_configure_rlp_dma(isc, rlp_dma_direct_dump);
- if (ret)
- goto isc_try_fmt_err;
-
- ret = isc_try_configure_pipeline(isc);
- if (ret)
- goto isc_try_fmt_err;
-
- /* Obtain frame sizes if possible to have crop requirements ready */
- isc_try_fse(isc, &pad_state);
-
- v4l2_fill_mbus_format(&format.format, pixfmt, mbus_code);
- ret = v4l2_subdev_call(isc->current_subdev->sd, pad, set_fmt,
- &pad_state, &format);
- if (ret < 0)
- goto isc_try_fmt_subdev_err;
-
- v4l2_fill_pix_format(pixfmt, &format.format);
+ v4l_bound_align_image(&pixfmt->width, 16, isc->max_width, 0,
+ &pixfmt->height, 16, isc->max_height, 0, 0);
+ /* If we did not find the requested format, we will fallback here */
+ pixfmt->pixelformat = isc->try_config.fourcc;
+ pixfmt->colorspace = V4L2_COLORSPACE_SRGB;
+ pixfmt->field = V4L2_FIELD_NONE;

- /* Limit to Atmel ISC hardware capabilities */
- if (pixfmt->width > isc->max_width)
- pixfmt->width = isc->max_width;
- if (pixfmt->height > isc->max_height)
- pixfmt->height = isc->max_height;

- pixfmt->field = V4L2_FIELD_NONE;
pixfmt->bytesperline = (pixfmt->width * isc->try_config.bpp_v4l2) >> 3;
pixfmt->sizeimage = ((pixfmt->width * isc->try_config.bpp) >> 3) *
pixfmt->height;

- if (code)
- *code = mbus_code;
+ isc->try_fmt = *f;

return 0;
+}

-isc_try_fmt_err:
- v4l2_err(&isc->v4l2_dev, "Could not find any possible format for a working pipeline\n");
-isc_try_fmt_subdev_err:
- memset(&isc->try_config, 0, sizeof(isc->try_config));
+static int isc_set_fmt(struct isc_device *isc, struct v4l2_format *f)
+{
+ isc_try_fmt(isc, f);

- return ret;
+ /* make the try configuration active */
+ isc->config = isc->try_config;
+ isc->fmt = isc->try_fmt;
+
+ v4l2_dbg(1, debug, &isc->v4l2_dev, "ISC set_fmt to %.4s @%dx%d\n",
+ (char *)&f->fmt.pix.pixelformat,
+ f->fmt.pix.width, f->fmt.pix.height);
+
+ return 0;
}

-static int isc_set_fmt(struct isc_device *isc, struct v4l2_format *f)
+static int isc_validate(struct isc_device *isc)
{
+ int ret;
+ int i;
+ struct isc_format *sd_fmt = NULL;
+ struct v4l2_pix_format *pixfmt = &isc->fmt.fmt.pix;
struct v4l2_subdev_format format = {
.which = V4L2_SUBDEV_FORMAT_ACTIVE,
+ .pad = isc->remote_pad,
+ };
+ struct v4l2_subdev_pad_config pad_cfg = {};
+ struct v4l2_subdev_state pad_state = {
+ .pads = &pad_cfg,
};
- u32 mbus_code = 0;
- int ret;

- ret = isc_try_fmt(isc, f, &mbus_code);
+ /* Get current format from subdev */
+ ret = v4l2_subdev_call(isc->current_subdev->sd, pad, get_fmt, NULL,
+ &format);
if (ret)
return ret;

- v4l2_fill_mbus_format(&format.format, &f->fmt.pix, mbus_code);
- ret = v4l2_subdev_call(isc->current_subdev->sd, pad,
- set_fmt, NULL, &format);
- if (ret < 0)
- return ret;
+ /* Identify the subdev's format configuration */
+ for (i = 0; i < isc->formats_list_size; i++)
+ if (isc->formats_list[i].mbus_code == format.format.code) {
+ sd_fmt = &isc->formats_list[i];
+ break;
+ }
+
+ /* Check if the format is not supported */
+ if (!sd_fmt) {
+ v4l2_err(&isc->v4l2_dev,
+ "Current subdevice is streaming a media bus code that is not supported 0x%x\n",
+ format.format.code);
+ return -EPIPE;
+ }
+
+ /* At this moment we know which format the subdev will use */
+ isc->try_config.sd_format = sd_fmt;
+
+ /* If the sensor is not RAW, we can only do a direct dump */
+ if (!ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code))
+ isc_try_configure_rlp_dma(isc, true);

/* Limit to Atmel ISC hardware capabilities */
- if (f->fmt.pix.width > isc->max_width)
- f->fmt.pix.width = isc->max_width;
- if (f->fmt.pix.height > isc->max_height)
- f->fmt.pix.height = isc->max_height;
+ v4l_bound_align_image(&format.format.width, 16, isc->max_width, 0,
+ &format.format.height, 16, isc->max_height, 0, 0);

- isc->fmt = *f;
+ /* Check if the frame size is the same. Otherwise we may overflow */
+ if (pixfmt->height != format.format.height ||
+ pixfmt->width != format.format.width) {
+ v4l2_err(&isc->v4l2_dev,
+ "ISC not configured with the proper frame size: %dx%d\n",
+ format.format.width, format.format.height);
+ return -EPIPE;
+ }

+ v4l2_dbg(1, debug, &isc->v4l2_dev,
+ "Identified subdev using format %.4s with %dx%d %d bpp\n",
+ (char *)&sd_fmt->fourcc, pixfmt->width, pixfmt->height,
+ isc->try_config.bpp);
+
+ /* Reset and restart AWB if the subdevice changed the format */
if (isc->try_config.sd_format && isc->config.sd_format &&
isc->try_config.sd_format != isc->config.sd_format) {
isc->ctrls.hist_stat = HIST_INIT;
isc_reset_awb_ctrls(isc);
isc_update_v4l2_ctrls(isc);
}
- /* make the try configuration active */
+
+ /* Validate formats */
+ ret = isc_try_validate_formats(isc);
+ if (ret)
+ return ret;
+
+ /* Obtain frame sizes if possible to have crop requirements ready */
+ isc_try_fse(isc, &pad_state);
+
+ /* Configure ISC pipeline for the config */
+ ret = isc_try_configure_pipeline(isc);
+ if (ret)
+ return ret;
+
isc->config = isc->try_config;

v4l2_dbg(1, debug, &isc->v4l2_dev, "New ISC configuration in place\n");
@@ -1043,7 +1036,7 @@ static int isc_try_fmt_vid_cap(struct file *file, void *priv,
{
struct isc_device *isc = video_drvdata(file);

- return isc_try_fmt(isc, f, NULL);
+ return isc_try_fmt(isc, f);
}

static int isc_enum_input(struct file *file, void *priv,
@@ -1098,10 +1091,6 @@ static int isc_enum_framesizes(struct file *file, void *fh,
if (fsize->index)
return -EINVAL;

- for (i = 0; i < isc->num_user_formats; i++)
- if (isc->user_formats[i]->fourcc == fsize->pixel_format)
- ret = 0;
-
for (i = 0; i < isc->controller_formats_size; i++)
if (isc->controller_formats[i].fourcc == fsize->pixel_format)
ret = 0;
@@ -1782,52 +1771,6 @@ struct isc_format *isc_find_format_by_code(struct isc_device *isc,
}
EXPORT_SYMBOL_GPL(isc_find_format_by_code);

-static int isc_formats_init(struct isc_device *isc)
-{
- struct isc_format *fmt;
- struct v4l2_subdev *subdev = isc->current_subdev->sd;
- unsigned int num_fmts, i, j;
- u32 list_size = isc->formats_list_size;
- struct v4l2_subdev_mbus_code_enum mbus_code = {
- .which = V4L2_SUBDEV_FORMAT_ACTIVE,
- };
-
- num_fmts = 0;
- while (!v4l2_subdev_call(subdev, pad, enum_mbus_code,
- NULL, &mbus_code)) {
- mbus_code.index++;
-
- fmt = isc_find_format_by_code(isc, mbus_code.code, &i);
- if (!fmt) {
- v4l2_warn(&isc->v4l2_dev, "Mbus code %x not supported\n",
- mbus_code.code);
- continue;
- }
-
- fmt->sd_support = true;
- num_fmts++;
- }
-
- if (!num_fmts)
- return -ENXIO;
-
- isc->num_user_formats = num_fmts;
- isc->user_formats = devm_kcalloc(isc->dev,
- num_fmts, sizeof(*isc->user_formats),
- GFP_KERNEL);
- if (!isc->user_formats)
- return -ENOMEM;
-
- fmt = &isc->formats_list[0];
- for (i = 0, j = 0; i < list_size; i++) {
- if (fmt->sd_support)
- isc->user_formats[j++] = fmt;
- fmt++;
- }
-
- return 0;
-}
-
static int isc_set_default_fmt(struct isc_device *isc)
{
struct v4l2_format f = {
@@ -1836,12 +1779,12 @@ static int isc_set_default_fmt(struct isc_device *isc)
.width = VGA_WIDTH,
.height = VGA_HEIGHT,
.field = V4L2_FIELD_NONE,
- .pixelformat = isc->user_formats[0]->fourcc,
+ .pixelformat = isc->controller_formats[0].fourcc,
},
};
int ret;

- ret = isc_try_fmt(isc, &f, NULL);
+ ret = isc_try_fmt(isc, &f);
if (ret)
return ret;

@@ -1896,13 +1839,6 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier)
spin_lock_init(&isc->dma_queue_lock);
spin_lock_init(&isc->awb_lock);

- ret = isc_formats_init(isc);
- if (ret < 0) {
- v4l2_err(&isc->v4l2_dev,
- "Init format failed: %d\n", ret);
- goto isc_async_complete_err;
- }
-
ret = isc_set_default_fmt(isc);
if (ret) {
v4l2_err(&isc->v4l2_dev, "Could not set default format\n");
@@ -2016,6 +1952,24 @@ int isc_pipeline_init(struct isc_device *isc)
}
EXPORT_SYMBOL_GPL(isc_pipeline_init);

+static int isc_link_validate(struct media_link *link)
+{
+ struct video_device *vdev =
+ media_entity_to_video_device(link->sink->entity);
+ struct isc_device *isc = video_get_drvdata(vdev);
+ int ret;
+
+ ret = v4l2_subdev_link_validate(link);
+ if (ret)
+ return ret;
+
+ return isc_validate(isc);
+}
+
+static const struct media_entity_operations isc_entity_operations = {
+ .link_validate = isc_link_validate,
+};
+
int isc_mc_init(struct isc_device *isc, u32 ver)
{
const struct of_device_id *match;
@@ -2023,6 +1977,8 @@ int isc_mc_init(struct isc_device *isc, u32 ver)

isc->video_dev.entity.function = MEDIA_ENT_F_IO_V4L;
isc->video_dev.entity.flags = MEDIA_ENT_FL_DEFAULT;
+ isc->video_dev.entity.ops = &isc_entity_operations;
+
isc->pads[ISC_PAD_SINK].flags = MEDIA_PAD_FL_SINK;

ret = media_entity_pads_init(&isc->video_dev.entity, ISC_PADS_NUM,
diff --git a/drivers/media/platform/atmel/atmel-isc-scaler.c b/drivers/media/platform/atmel/atmel-isc-scaler.c
index a1ca4633787c..2b0621297dfe 100644
--- a/drivers/media/platform/atmel/atmel-isc-scaler.c
+++ b/drivers/media/platform/atmel/atmel-isc-scaler.c
@@ -173,6 +173,10 @@ static const struct v4l2_subdev_pad_ops isc_scaler_pad_ops = {
.init_cfg = isc_scaler_init_cfg,
};

+static const struct media_entity_operations isc_scaler_entity_ops = {
+ .link_validate = v4l2_subdev_link_validate,
+};
+
static const struct v4l2_subdev_ops xisc_scaler_subdev_ops = {
.pad = &isc_scaler_pad_ops,
};
@@ -190,6 +194,7 @@ int isc_scaler_init(struct isc_device *isc)

isc->scaler_sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
isc->scaler_sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
+ isc->scaler_sd.entity.ops = &isc_scaler_entity_ops;
isc->scaler_pads[ISC_SCALER_PAD_SINK].flags = MEDIA_PAD_FL_SINK;
isc->scaler_pads[ISC_SCALER_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;

diff --git a/drivers/media/platform/atmel/atmel-isc.h b/drivers/media/platform/atmel/atmel-isc.h
index f98f25a55e73..5b0100c8c245 100644
--- a/drivers/media/platform/atmel/atmel-isc.h
+++ b/drivers/media/platform/atmel/atmel-isc.h
@@ -63,15 +63,16 @@ struct isc_subdev_entity {
* @cfa_baycfg: If this format is RAW BAYER, indicate the type of bayer.
this is either BGBG, RGRG, etc.
* @pfe_cfg0_bps: Number of hardware data lines connected to the ISC
+ * @raw: If the format is raw bayer.
*/

struct isc_format {
u32 fourcc;
u32 mbus_code;
u32 cfa_baycfg;
-
- bool sd_support;
u32 pfe_cfg0_bps;
+
+ bool raw;
};

/* Pipeline bitmap */
@@ -216,8 +217,7 @@ enum isc_scaler_pads {
* @comp: completion reference that signals frame completion
*
* @fmt: current v42l format
- * @user_formats: list of formats that are supported and agreed with sd
- * @num_user_formats: how many formats are in user_formats
+ * @try_fmt: current v4l2 try format
*
* @config: current ISC format configuration
* @try_config: the current ISC try format , not yet activated
@@ -272,6 +272,7 @@ enum isc_scaler_pads {
* @formats_list_size: size of formats_list array
* @pads: media controller pads for isc video entity
* @mdev: media device that is registered by the isc
+ * @mpipe: media device pipeline used by the isc
* @remote_pad: remote pad on the connected subdevice
* @scaler_sd: subdevice for the scaler that isc registers
* @scaler_pads: media controller pads for the scaler subdevice
@@ -298,8 +299,7 @@ struct isc_device {
struct completion comp;

struct v4l2_format fmt;
- struct isc_format **user_formats;
- unsigned int num_user_formats;
+ struct v4l2_format try_fmt;

struct fmt_config config;
struct fmt_config try_config;
@@ -369,6 +369,7 @@ struct isc_device {
struct {
struct media_pad pads[ISC_PADS_NUM];
struct media_device mdev;
+ struct media_pipeline mpipe;

u32 remote_pad;
};
diff --git a/drivers/media/platform/atmel/atmel-sama5d2-isc.c b/drivers/media/platform/atmel/atmel-sama5d2-isc.c
index d96ee3373889..2a651e237193 100644
--- a/drivers/media/platform/atmel/atmel-sama5d2-isc.c
+++ b/drivers/media/platform/atmel/atmel-sama5d2-isc.c
@@ -80,20 +80,40 @@ static const struct isc_format sama5d2_controller_formats[] = {
.fourcc = V4L2_PIX_FMT_Y10,
}, {
.fourcc = V4L2_PIX_FMT_SBGGR8,
+ .raw = true,
}, {
.fourcc = V4L2_PIX_FMT_SGBRG8,
+ .raw = true,
}, {
.fourcc = V4L2_PIX_FMT_SGRBG8,
+ .raw = true,
}, {
.fourcc = V4L2_PIX_FMT_SRGGB8,
+ .raw = true,
}, {
.fourcc = V4L2_PIX_FMT_SBGGR10,
+ .raw = true,
}, {
.fourcc = V4L2_PIX_FMT_SGBRG10,
+ .raw = true,
}, {
.fourcc = V4L2_PIX_FMT_SGRBG10,
+ .raw = true,
}, {
.fourcc = V4L2_PIX_FMT_SRGGB10,
+ .raw = true,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR12,
+ .raw = true,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG12,
+ .raw = true,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG12,
+ .raw = true,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SRGGB12,
+ .raw = true,
},
};

diff --git a/drivers/media/platform/atmel/atmel-sama7g5-isc.c b/drivers/media/platform/atmel/atmel-sama7g5-isc.c
index e07ae188c15f..a0d60cfdba7b 100644
--- a/drivers/media/platform/atmel/atmel-sama7g5-isc.c
+++ b/drivers/media/platform/atmel/atmel-sama7g5-isc.c
@@ -89,20 +89,40 @@ static const struct isc_format sama7g5_controller_formats[] = {
.fourcc = V4L2_PIX_FMT_Y16,
}, {
.fourcc = V4L2_PIX_FMT_SBGGR8,
+ .raw = true,
}, {
.fourcc = V4L2_PIX_FMT_SGBRG8,
+ .raw = true,
}, {
.fourcc = V4L2_PIX_FMT_SGRBG8,
+ .raw = true,
}, {
.fourcc = V4L2_PIX_FMT_SRGGB8,
+ .raw = true,
}, {
.fourcc = V4L2_PIX_FMT_SBGGR10,
+ .raw = true,
}, {
.fourcc = V4L2_PIX_FMT_SGBRG10,
+ .raw = true,
}, {
.fourcc = V4L2_PIX_FMT_SGRBG10,
+ .raw = true,
}, {
.fourcc = V4L2_PIX_FMT_SRGGB10,
+ .raw = true,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR12,
+ .raw = true,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG12,
+ .raw = true,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG12,
+ .raw = true,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SRGGB12,
+ .raw = true,
},
};

--
2.25.1

2022-03-10 14:53:21

[permalink] [raw]

Subject: [PATCH v9 01/13] media: atmel: atmel-isc-base: use streaming status when queueing buffers

During experiments with libcamera, it looks like vb2_is_streaming returns
true before our start streaming is called.
Order of operations is streamon -> queue -> start_streaming
ISC would have started the DMA immediately when a buffer is being added
to the vbqueue if the queue is streaming.
It is more safe to start the DMA after the start streaming of the driver is
called.
Thus, even if vb2queue is streaming, add the buffer to the dma queue of the
driver instead of actually starting the DMA process, if the start streaming
has not been called yet.
Tho achieve this, we have to use vb2_start_streaming_called instead of
vb2_is_streaming.

Signed-off-by: Eugen Hristev <[email protected]>
Reviewed-by: Jacopo Mondi <[email protected]>
---
drivers/media/platform/atmel/atmel-isc-base.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/media/platform/atmel/atmel-isc-base.c b/drivers/media/platform/atmel/atmel-isc-base.c
index db15770d5b88..d2cc6c99984f 100644
--- a/drivers/media/platform/atmel/atmel-isc-base.c
+++ b/drivers/media/platform/atmel/atmel-isc-base.c
@@ -442,7 +442,7 @@ static void isc_buffer_queue(struct vb2_buffer *vb)

spin_lock_irqsave(&isc->dma_queue_lock, flags);
if (!isc->cur_frm && list_empty(&isc->dma_queue) &&
- vb2_is_streaming(vb->vb2_queue)) {
+ vb2_start_streaming_called(vb->vb2_queue)) {
isc->cur_frm = buf;
isc_start_dma(isc);
} else
--
2.25.1

2022-03-10 14:56:24

[permalink] [raw]

Subject: [PATCH v9 05/13] media: atmel: atmel-sama5d2-isc: fix wrong mask in YUYV format check

While this does not happen in production, this check should be done
versus the mask, as checking with the YCYC value may not include
some bits that may be set.
Is it correct and safe to check the whole mask.

Fixes: 123aaf816b95 ("media: atmel: atmel-sama5d2-isc: fix YUYV format")
Signed-off-by: Eugen Hristev <[email protected]>
Reviewed-by: Jacopo Mondi <[email protected]>
---
drivers/media/platform/atmel/atmel-sama5d2-isc.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/media/platform/atmel/atmel-sama5d2-isc.c b/drivers/media/platform/atmel/atmel-sama5d2-isc.c
index c244682ea22f..025c3e8a7e95 100644
--- a/drivers/media/platform/atmel/atmel-sama5d2-isc.c
+++ b/drivers/media/platform/atmel/atmel-sama5d2-isc.c
@@ -291,7 +291,7 @@ static void isc_sama5d2_config_rlp(struct isc_device *isc)
* Thus, if the YCYC mode is selected, replace it with the
* sama5d2-compliant mode which is YYCC .
*/
- if ((rlp_mode & ISC_RLP_CFG_MODE_YCYC) == ISC_RLP_CFG_MODE_YCYC) {
+ if ((rlp_mode & ISC_RLP_CFG_MODE_MASK) == ISC_RLP_CFG_MODE_YCYC) {
rlp_mode &= ~ISC_RLP_CFG_MODE_MASK;
rlp_mode |= ISC_RLP_CFG_MODE_YYCC;
}
--
2.25.1

2022-03-10 15:01:13

[permalink] [raw]

Subject: [PATCH v9 03/13] media: atmel: atmel-isc: remove redundant comments

Remove duplicate comments which are already in place before the struct
definition.

Signed-off-by: Eugen Hristev <[email protected]>
Reviewed-by: Jacopo Mondi <[email protected]>
---
drivers/media/platform/atmel/atmel-isc.h | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/media/platform/atmel/atmel-isc.h b/drivers/media/platform/atmel/atmel-isc.h
index 07fa6dbf8460..f9ad7ec6bd13 100644
--- a/drivers/media/platform/atmel/atmel-isc.h
+++ b/drivers/media/platform/atmel/atmel-isc.h
@@ -272,7 +272,7 @@ struct isc_device {
struct video_device video_dev;

struct vb2_queue vb2_vidq;
- spinlock_t dma_queue_lock; /* serialize access to dma queue */
+ spinlock_t dma_queue_lock;
struct list_head dma_queue;
struct isc_buffer *cur_frm;
unsigned int sequence;
@@ -289,8 +289,8 @@ struct isc_device {
struct isc_ctrls ctrls;
struct work_struct awb_work;

- struct mutex lock; /* serialize access to file operations */
- spinlock_t awb_lock; /* serialize access to DMA buffers from awb work queue */
+ struct mutex lock;
+ spinlock_t awb_lock;

struct regmap_field *pipeline[ISC_PIPE_LINE_NODE_NUM];

--
2.25.1

2022-03-10 16:20:34

[permalink] [raw]

Subject: [PATCH v9 10/13] dt-bindings: media: microchip,xisc: add bus-width of 14

The Microchip XISC supports a bus width of 14 bits.
Add it to the supported bus widths.

Signed-off-by: Eugen Hristev <[email protected]>
Acked-by: Rob Herring <[email protected]>
---
Documentation/devicetree/bindings/media/microchip,xisc.yaml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Documentation/devicetree/bindings/media/microchip,xisc.yaml b/Documentation/devicetree/bindings/media/microchip,xisc.yaml
index 086e1430af4f..3be8f64c3e21 100644
--- a/Documentation/devicetree/bindings/media/microchip,xisc.yaml
+++ b/Documentation/devicetree/bindings/media/microchip,xisc.yaml
@@ -67,7 +67,7 @@ properties:
remote-endpoint: true

bus-width:
- enum: [8, 9, 10, 11, 12]
+ enum: [8, 9, 10, 11, 12, 14]
default: 12

hsync-active:
--
2.25.1

2022-03-11 16:28:46

[permalink] [raw]

Subject: [PATCH v9 13/13] ARM: multi_v7_defconfig: add atmel video pipeline modules

Add drivers for the atmel video capture pipeline: atmel isc, xisc and
microchip csi2dc.

Signed-off-by: Eugen Hristev <[email protected]>
---
arch/arm/configs/multi_v7_defconfig | 3 +++
1 file changed, 3 insertions(+)

diff --git a/arch/arm/configs/multi_v7_defconfig b/arch/arm/configs/multi_v7_defconfig
index 8863fa969ede..b768abad8df0 100644
--- a/arch/arm/configs/multi_v7_defconfig
+++ b/arch/arm/configs/multi_v7_defconfig
@@ -639,7 +639,10 @@ CONFIG_VIDEO_S5P_MIPI_CSIS=m
CONFIG_VIDEO_EXYNOS_FIMC_LITE=m
CONFIG_VIDEO_EXYNOS4_FIMC_IS=m
CONFIG_VIDEO_RCAR_VIN=m
+CONFIG_VIDEO_ATMEL_ISC=m
+CONFIG_VIDEO_ATMEL_XISC=m
CONFIG_VIDEO_ATMEL_ISI=m
+CONFIG_VIDEO_MICROCHIP_CSI2DC=m
CONFIG_V4L_MEM2MEM_DRIVERS=y
CONFIG_VIDEO_SAMSUNG_S5P_JPEG=m
CONFIG_VIDEO_SAMSUNG_S5P_MFC=m
--
2.25.1

2022-04-29 12:21:11

[permalink] [raw]

Subject: Re: [PATCH v9 08/13] media: atmel: atmel-isc: change format propagation to subdev into only verification

On Fri, Apr 29, 2022 at 11:58:48AM +0200, Jacopo Mondi wrote:
> On Fri, Apr 29, 2022 at 10:43:09AM +0200, Hans Verkuil wrote:
> > On 29/04/2022 10:28, [email protected] wrote:
> > > On 4/29/22 11:17 AM, Hans Verkuil wrote:
> > >> On 10/03/2022 10:51, Eugen Hristev wrote:
> > >>> As a top MC video driver, the atmel-isc should not propagate the format to the
> > >>> subdevice, it should rather check at start_streaming() time if the subdev is properly
> > >>> configured with a compatible format.
> > >>> Removed the whole format finding logic, and reworked the format verification
> > >>> at start_streaming time, such that the ISC will return an error if the subdevice
> > >>> is not properly configured. To achieve this, media_pipeline_start
> > >>> is called and a link_validate callback is created to check the formats.
> > >>> With this being done, the module parameter 'sensor_preferred' makes no sense
> > >>> anymore. The ISC should not decide which format the sensor is using. The
> > >>> ISC should only cope with the situation and inform userspace if the streaming
> > >>> is possible in the current configuration.
> > >>> The redesign of the format propagation has also risen the question of the
> > >>> enumfmt callback. If enumfmt is called with an mbus_code, the enumfmt handler
> > >>> should only return the formats that are supported for this mbus_code.
> > >>> Otherwise, the enumfmt will report all the formats that the ISC could output.
> > >>> With this rework, the dynamic list of user formats is removed. It makes no
> > >>> more sense to identify at complete time which formats the sensor could emit,
> > >>> and add those into a separate dynamic list.
> > >>> The ISC will start with a simple preconfigured default format, and at
> > >>> link validate time, decide whether it can use the format that is configured
> > >>> on the sink or not.
> > >>>
> > >>> Signed-off-by: Eugen Hristev <[email protected]>
> > >>> Reviewed-by: Jacopo Mondi <[email protected]>
> > >>> ---
> > >>> Changes in v9:
> > >>> - isc_link_validate now static
> > >>>
> > >>> Changes in v7:
> > >>> - minor typos as suggested by Jacopo
> > >>> - small changes, reduce some indentation, modified an index, as suggested by
> > >>> Jacopo
> > >>>
> > >>> Changes in v6:
> > >>> - reworked a bit enum_fmt as suggested by Jacopo
> > >>>
> > >>> Changes in v5:
> > >>> - removed user_formats dynamic list as it is now pointless
> > >>> - greatly simplified the enum_fmt function
> > >>> - removed some init code that was useless now
> > >>>
> > >>> Changes in v4:
> > >>> - moved validation code into link_validate and used media_pipeline_start
> > >>> - merged this patch with the enum_fmt patch which was previously in v3 of
> > >>> the series
> > >>>
> > >>> Changes in v3:
> > >>> - clamp to maximum resolution once the frame size from the subdev is found
> > >>> drivers/media/platform/atmel/atmel-isc-base.c | 412 ++++++++----------
> > >>> .../media/platform/atmel/atmel-isc-scaler.c | 5 +
> > >>> drivers/media/platform/atmel/atmel-isc.h | 13 +-
> > >>> .../media/platform/atmel/atmel-sama5d2-isc.c | 20 +
> > >>> .../media/platform/atmel/atmel-sama7g5-isc.c | 20 +
> > >>> 5 files changed, 236 insertions(+), 234 deletions(-)
> > >>>
> > >>> diff --git a/drivers/media/platform/atmel/atmel-isc-base.c b/drivers/media/platform/atmel/atmel-isc-base.c
> > >>> index ee1dda6707a0..fe2c0af58060 100644
> > >>> --- a/drivers/media/platform/atmel/atmel-isc-base.c
> > >>> +++ b/drivers/media/platform/atmel/atmel-isc-base.c
> > >>> @@ -36,11 +36,6 @@ static unsigned int debug;
> > >>> module_param(debug, int, 0644);
> > >>> MODULE_PARM_DESC(debug, "debug level (0-2)");
> > >>>
> > >>> -static unsigned int sensor_preferred = 1;
> > >>> -module_param(sensor_preferred, uint, 0644);
> > >>> -MODULE_PARM_DESC(sensor_preferred,
> > >>> - "Sensor is preferred to output the specified format (1-on 0-off), default 1");
> > >>> -
> > >>> #define ISC_IS_FORMAT_RAW(mbus_code) \
> > >>> (((mbus_code) & 0xf000) == 0x3000)
> > >>>
> > >>> @@ -337,6 +332,10 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)
> > >>> unsigned long flags;
> > >>> int ret;
> > >>>
> > >>> + ret = media_pipeline_start(&isc->video_dev.entity, &isc->mpipe);
> > >>
> > >> The pipeline validation is done in start_streaming, but I don't think that
> > >> is the best place: if STREAMON is called before buffers are queued, then
> > >> an invalid pipeline isn't discovered until enough buffers are queued to
> > >> kick off start_streaming.
> > >>
> > >> Drivers like vsp1, omap3isp and the samsung drivers all do this in streamon().
> > >>
> > >> I think that is the correct time to do this.
> > >
> > > Hello Hans,
> > >
> > > Initially (v2, v3) I had this in streamon(). The problem that I faced at
> > > that time was that streamoff was never called, so I could not call
> > > media_pipeline_stop(). Then Jacopo told me to move it to start_streaming
> > > (see change log for v4) , and I did not face any more problems.
>
> Yes indeed, seems I suggested to use media_pipeline_handler in a
> comment on your v3
>
> "at s_stream time your top driver calls media_pipeline_start()"
>
> sorry about that, I should have looked around a bit more carefully and
> notice most drivers do so at vb2 streamon
>
> However I don't see media_pipeline_start being called at all in v3 of
> the patch
>
> > It's a mess. Looking at some drivers I see that omap3isp calls media_pipeline_stop
> > in streamoff (so will have the same problem as you described if VIDIOC_STREAMOFF
> > isn't called), exynos4-is does the same, but it also checks the streaming state in
> > the release() fop callback, so that would fix this problem. And vimc does this
> > in stop_streaming.
> >
> > I'm in favor of fixing this in vb2, that framework knows exactly when this needs
> > to be called.
>
> Are you suggesting to have vb2 to call media_pipeline_start() or is it
> more complex than this ?

I think Hans meant adding a .validate() operation to vb2.

vb2 is already quite complex, I don't think adding more features is a
good idea. I'd rather have vb2 focus on buffer management only
(.start_streaming() and .stop_streaming() shouldn't have been in there
in my opinion), and handle validation in the .streamon() handler. I'd
expect most drivers that deal with media pipelines to do more work in
.streamon() anyway.

> > I'd like to know what others think.
> >
> > >> Laurent, Jacopo, do you agree with that?
> > >>
> > >> I notice that quite a few other drivers do this in start_streaming as well,
> > >> but I think it just weird that a QBUF can return EPIPE while STREAMON might
> > >> succeed.
> > >>
> > >> Perhaps we need an optional vb2 validate() callback that is called by
> > >> vb2_core_streamon() to validate the pipeline before streaming. That makes
> > >> it easier to implement these checks rather than having to implement your
> > >> own streamon ioctl.
> > >>
> > >>> + if (ret)
> > >>> + goto err_pipeline_start;
> > >>> +
> > >>> /* Enable stream on the sub device */
> > >>> ret = v4l2_subdev_call(isc->current_subdev->sd, video, s_stream, 1);
> > >>> if (ret && ret != -ENOIOCTLCMD) {
> > >>> @@ -385,6 +384,9 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)
> > >>> v4l2_subdev_call(isc->current_subdev->sd, video, s_stream, 0);
> > >>>
> > >>> err_start_stream:
> > >>> + media_pipeline_stop(&isc->video_dev.entity);
> > >>> +
> > >>> +err_pipeline_start:
> > >>> spin_lock_irqsave(&isc->dma_queue_lock, flags);
> > >>> list_for_each_entry(buf, &isc->dma_queue, list)
> > >>> vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_QUEUED);
> > >>> @@ -423,6 +425,9 @@ static void isc_stop_streaming(struct vb2_queue *vq)
> > >>> if (ret && ret != -ENOIOCTLCMD)
> > >>> v4l2_err(&isc->v4l2_dev, "stream off failed in subdev\n");
> > >>>
> > >>> + /* Stop media pipeline */
> > >>> + media_pipeline_stop(&isc->video_dev.entity);
> > >>> +
> > >>> /* Release all active buffers */
> > >>> spin_lock_irqsave(&isc->dma_queue_lock, flags);
> > >>> if (unlikely(isc->cur_frm)) {
> > >>> @@ -453,22 +458,6 @@ static void isc_buffer_queue(struct vb2_buffer *vb)
> > >>> spin_unlock_irqrestore(&isc->dma_queue_lock, flags);
> > >>> }
> > >>>
> > >>> -static struct isc_format *find_format_by_fourcc(struct isc_device *isc,
> > >>> - unsigned int fourcc)
> > >>> -{
> > >>> - unsigned int num_formats = isc->num_user_formats;
> > >>> - struct isc_format *fmt;
> > >>> - unsigned int i;
> > >>> -
> > >>> - for (i = 0; i < num_formats; i++) {
> > >>> - fmt = isc->user_formats[i];
> > >>> - if (fmt->fourcc == fourcc)
> > >>> - return fmt;
> > >>> - }
> > >>> -
> > >>> - return NULL;
> > >>> -}
> > >>> -
> > >>> static const struct vb2_ops isc_vb2_ops = {
> > >>> .queue_setup = isc_queue_setup,
> > >>> .wait_prepare = vb2_ops_wait_prepare,
> > >>> @@ -497,23 +486,57 @@ static int isc_enum_fmt_vid_cap(struct file *file, void *priv,
> > >>> {
> > >>> struct isc_device *isc = video_drvdata(file);
> > >>> u32 index = f->index;
> > >>> - u32 i, supported_index;
> > >>> + u32 i, supported_index = 0;
> > >>> + struct isc_format *fmt;
> > >>> +
> > >>> + /*
> > >>> + * If we are not asked a specific mbus_code, we have to report all
> > >>> + * the formats that we can output.
> > >>> + */
> > >>> + if (!f->mbus_code) {
> > >>> + if (index >= isc->controller_formats_size)
> > >>> + return -EINVAL;
> > >>>
> > >>> - if (index < isc->controller_formats_size) {
> > >>> f->pixelformat = isc->controller_formats[index].fourcc;
> > >>> +
> > >>> + return 0;
> > >>> + }
> > >>> +
> > >>> + /*
> > >>> + * If a specific mbus_code is requested, check if we support
> > >>> + * this mbus_code as input for the ISC.
> > >>> + * If it's supported, then we report the corresponding pixelformat
> > >>> + * as first possible option for the ISC.
> > >>> + * E.g. mbus MEDIA_BUS_FMT_YUYV8_2X8 and report
> > >>> + * 'YUYV' (YUYV 4:2:2)
> > >>> + */
> > >>> + fmt = isc_find_format_by_code(isc, f->mbus_code, &i);
> > >>> + if (!fmt)
> > >>> + return -EINVAL;
> > >>> +
> > >>> + if (!index) {
> > >>> + f->pixelformat = fmt->fourcc;
> > >>> +
> > >>> return 0;
> > >>> }
> > >>>
> > >>> - index -= isc->controller_formats_size;
> > >>> + supported_index++;
> > >>>
> > >>> - supported_index = 0;
> > >>> + /* If the index is not raw, we don't have anymore formats to report */
> > >>> + if (!ISC_IS_FORMAT_RAW(f->mbus_code))
> > >>> + return -EINVAL;
> > >>>
> > >>> - for (i = 0; i < isc->formats_list_size; i++) {
> > >>> - if (!ISC_IS_FORMAT_RAW(isc->formats_list[i].mbus_code) ||
> > >>> - !isc->formats_list[i].sd_support)
> > >>> + /*
> > >>> + * We are asked for a specific mbus code, which is raw.
> > >>> + * We have to search through the formats we can convert to.
> > >>> + * We have to skip the raw formats, we cannot convert to raw.
> > >>> + * E.g. 'AR12' (16-bit ARGB 4-4-4-4), 'AR15' (16-bit ARGB 1-5-5-5), etc.
> > >>> + */
> > >>> + for (i = 0; i < isc->controller_formats_size; i++) {
> > >>> + if (isc->controller_formats[i].raw)
> > >>> continue;
> > >>> - if (supported_index == index) {
> > >>> - f->pixelformat = isc->formats_list[i].fourcc;
> > >>> + if (index == supported_index) {
> > >>> + f->pixelformat = isc->controller_formats[i].fourcc;
> > >>> return 0;
> > >>> }
> > >>> supported_index++;
> > >>> @@ -584,20 +607,30 @@ static int isc_try_validate_formats(struct isc_device *isc)
> > >>> break;
> > >>> default:
> > >>> /* any other different formats are not supported */
> > >>> + v4l2_err(&isc->v4l2_dev, "Requested unsupported format.\n");
> > >>> ret = -EINVAL;
> > >>> }
> > >>> v4l2_dbg(1, debug, &isc->v4l2_dev,
> > >>> "Format validation, requested rgb=%u, yuv=%u, grey=%u, bayer=%u\n",
> > >>> rgb, yuv, grey, bayer);
> > >>>
> > >>> - /* we cannot output RAW if we do not receive RAW */
> > >>> - if ((bayer) && !ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code))
> > >>> + if (bayer &&
> > >>> + !ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code)) {
> > >>> + v4l2_err(&isc->v4l2_dev, "Cannot output RAW if we do not receive RAW.\n");
> > >>> return -EINVAL;
> > >>> + }
> > >>>
> > >>> - /* we cannot output GREY if we do not receive RAW/GREY */
> > >>> if (grey && !ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code) &&
> > >>> - !ISC_IS_FORMAT_GREY(isc->try_config.sd_format->mbus_code))
> > >>> + !ISC_IS_FORMAT_GREY(isc->try_config.sd_format->mbus_code)) {
> > >>> + v4l2_err(&isc->v4l2_dev, "Cannot output GREY if we do not receive RAW/GREY.\n");
> > >>> return -EINVAL;
> > >>> + }
> > >>> +
> > >>> + if ((rgb || bayer || yuv) &&
> > >>> + ISC_IS_FORMAT_GREY(isc->try_config.sd_format->mbus_code)) {
> > >>> + v4l2_err(&isc->v4l2_dev, "Cannot convert GREY to another format.\n");
> > >>> + return -EINVAL;
> > >>> + }
> > >>>
> > >>> return ret;
> > >>> }
> > >>> @@ -825,7 +858,7 @@ static void isc_try_fse(struct isc_device *isc,
> > >>> * If we do not know yet which format the subdev is using, we cannot
> > >>> * do anything.
> > >>> */
> > >>> - if (!isc->try_config.sd_format)
> > >>> + if (!isc->config.sd_format)
> > >>> return;
> > >>>
> > >>> fse.code = isc->try_config.sd_format->mbus_code;
> > >>> @@ -846,180 +879,140 @@ static void isc_try_fse(struct isc_device *isc,
> > >>> }
> > >>> }
> > >>>
> > >>> -static int isc_try_fmt(struct isc_device *isc, struct v4l2_format *f,
> > >>> - u32 *code)
> > >>> +static int isc_try_fmt(struct isc_device *isc, struct v4l2_format *f)
> > >>> {
> > >>> - int i;
> > >>> - struct isc_format *sd_fmt = NULL, *direct_fmt = NULL;
> > >>> struct v4l2_pix_format *pixfmt = &f->fmt.pix;
> > >>> - struct v4l2_subdev_pad_config pad_cfg = {};
> > >>> - struct v4l2_subdev_state pad_state = {
> > >>> - .pads = &pad_cfg
> > >>> - };
> > >>> - struct v4l2_subdev_format format = {
> > >>> - .which = V4L2_SUBDEV_FORMAT_TRY,
> > >>> - };
> > >>> - u32 mbus_code;
> > >>> - int ret;
> > >>> - bool rlp_dma_direct_dump = false;
> > >>> + unsigned int i;
> > >>>
> > >>> if (f->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
> > >>> return -EINVAL;
> > >>>
> > >>> - /* Step 1: find a RAW format that is supported */
> > >>> - for (i = 0; i < isc->num_user_formats; i++) {
> > >>> - if (ISC_IS_FORMAT_RAW(isc->user_formats[i]->mbus_code)) {
> > >>> - sd_fmt = isc->user_formats[i];
> > >>> + isc->try_config.fourcc = isc->controller_formats[0].fourcc;
> > >>> +
> > >>> + /* find if the format requested is supported */
> > >>> + for (i = 0; i < isc->controller_formats_size; i++)
> > >>> + if (isc->controller_formats[i].fourcc == pixfmt->pixelformat) {
> > >>> + isc->try_config.fourcc = pixfmt->pixelformat;
> > >>> break;
> > >>> }
> > >>> - }
> > >>> - /* Step 2: We can continue with this RAW format, or we can look
> > >>> - * for better: maybe sensor supports directly what we need.
> > >>> - */
> > >>> - direct_fmt = find_format_by_fourcc(isc, pixfmt->pixelformat);
> > >>> -
> > >>> - /* Step 3: We have both. We decide given the module parameter which
> > >>> - * one to use.
> > >>> - */
> > >>> - if (direct_fmt && sd_fmt && sensor_preferred)
> > >>> - sd_fmt = direct_fmt;
> > >>> -
> > >>> - /* Step 4: we do not have RAW but we have a direct format. Use it. */
> > >>> - if (direct_fmt && !sd_fmt)
> > >>> - sd_fmt = direct_fmt;
> > >>> -
> > >>> - /* Step 5: if we are using a direct format, we need to package
> > >>> - * everything as 8 bit data and just dump it
> > >>> - */
> > >>> - if (sd_fmt == direct_fmt)
> > >>> - rlp_dma_direct_dump = true;
> > >>> -
> > >>> - /* Step 6: We have no format. This can happen if the userspace
> > >>> - * requests some weird/invalid format.
> > >>> - * In this case, default to whatever we have
> > >>> - */
> > >>> - if (!sd_fmt && !direct_fmt) {
> > >>> - sd_fmt = isc->user_formats[isc->num_user_formats - 1];
> > >>> - v4l2_dbg(1, debug, &isc->v4l2_dev,
> > >>> - "Sensor not supporting %.4s, using %.4s\n",
> > >>> - (char *)&pixfmt->pixelformat, (char *)&sd_fmt->fourcc);
> > >>> - }
> > >>> -
> > >>> - if (!sd_fmt) {
> > >>> - ret = -EINVAL;
> > >>> - goto isc_try_fmt_err;
> > >>> - }
> > >>> -
> > >>> - /* Step 7: Print out what we decided for debugging */
> > >>> - v4l2_dbg(1, debug, &isc->v4l2_dev,
> > >>> - "Preferring to have sensor using format %.4s\n",
> > >>> - (char *)&sd_fmt->fourcc);
> > >>>
> > >>> - /* Step 8: at this moment we decided which format the subdev will use */
> > >>> - isc->try_config.sd_format = sd_fmt;
> > >>> + isc_try_configure_rlp_dma(isc, false);
> > >>>
> > >>> /* Limit to Atmel ISC hardware capabilities */
> > >>> - if (pixfmt->width > isc->max_width)
> > >>> - pixfmt->width = isc->max_width;
> > >>> - if (pixfmt->height > isc->max_height)
> > >>> - pixfmt->height = isc->max_height;
> > >>> -
> > >>> - /*
> > >>> - * The mbus format is the one the subdev outputs.
> > >>> - * The pixels will be transferred in this format Sensor -> ISC
> > >>> - */
> > >>> - mbus_code = sd_fmt->mbus_code;
> > >>> -
> > >>> - /*
> > >>> - * Validate formats. If the required format is not OK, default to raw.
> > >>> - */
> > >>> -
> > >>> - isc->try_config.fourcc = pixfmt->pixelformat;
> > >>> -
> > >>> - if (isc_try_validate_formats(isc)) {
> > >>> - pixfmt->pixelformat = isc->try_config.fourcc = sd_fmt->fourcc;
> > >>> - /* Re-try to validate the new format */
> > >>> - ret = isc_try_validate_formats(isc);
> > >>> - if (ret)
> > >>> - goto isc_try_fmt_err;
> > >>> - }
> > >>> -
> > >>> - ret = isc_try_configure_rlp_dma(isc, rlp_dma_direct_dump);
> > >>> - if (ret)
> > >>> - goto isc_try_fmt_err;
> > >>> -
> > >>> - ret = isc_try_configure_pipeline(isc);
> > >>> - if (ret)
> > >>> - goto isc_try_fmt_err;
> > >>> -
> > >>> - /* Obtain frame sizes if possible to have crop requirements ready */
> > >>> - isc_try_fse(isc, &pad_state);
> > >>> -
> > >>> - v4l2_fill_mbus_format(&format.format, pixfmt, mbus_code);
> > >>> - ret = v4l2_subdev_call(isc->current_subdev->sd, pad, set_fmt,
> > >>> - &pad_state, &format);
> > >>> - if (ret < 0)
> > >>> - goto isc_try_fmt_subdev_err;
> > >>> -
> > >>> - v4l2_fill_pix_format(pixfmt, &format.format);
> > >>> + v4l_bound_align_image(&pixfmt->width, 16, isc->max_width, 0,
> > >>> + &pixfmt->height, 16, isc->max_height, 0, 0);
> > >>> + /* If we did not find the requested format, we will fallback here */
> > >>> + pixfmt->pixelformat = isc->try_config.fourcc;
> > >>> + pixfmt->colorspace = V4L2_COLORSPACE_SRGB;
> > >>> + pixfmt->field = V4L2_FIELD_NONE;
> > >>>
> > >>> - /* Limit to Atmel ISC hardware capabilities */
> > >>> - if (pixfmt->width > isc->max_width)
> > >>> - pixfmt->width = isc->max_width;
> > >>> - if (pixfmt->height > isc->max_height)
> > >>> - pixfmt->height = isc->max_height;
> > >>>
> > >>> - pixfmt->field = V4L2_FIELD_NONE;
> > >>> pixfmt->bytesperline = (pixfmt->width * isc->try_config.bpp_v4l2) >> 3;
> > >>> pixfmt->sizeimage = ((pixfmt->width * isc->try_config.bpp) >> 3) *
> > >>> pixfmt->height;
> > >>>
> > >>> - if (code)
> > >>> - *code = mbus_code;
> > >>> + isc->try_fmt = *f;
> > >>>
> > >>> return 0;
> > >>> +}
> > >>>
> > >>> -isc_try_fmt_err:
> > >>> - v4l2_err(&isc->v4l2_dev, "Could not find any possible format for a working pipeline\n");
> > >>> -isc_try_fmt_subdev_err:
> > >>> - memset(&isc->try_config, 0, sizeof(isc->try_config));
> > >>> +static int isc_set_fmt(struct isc_device *isc, struct v4l2_format *f)
> > >>> +{
> > >>> + isc_try_fmt(isc, f);
> > >>>
> > >>> - return ret;
> > >>> + /* make the try configuration active */
> > >>> + isc->config = isc->try_config;
> > >>> + isc->fmt = isc->try_fmt;
> > >>> +
> > >>> + v4l2_dbg(1, debug, &isc->v4l2_dev, "ISC set_fmt to %.4s @%dx%d\n",
> > >>> + (char *)&f->fmt.pix.pixelformat,
> > >>> + f->fmt.pix.width, f->fmt.pix.height);
> > >>> +
> > >>> + return 0;
> > >>> }
> > >>>
> > >>> -static int isc_set_fmt(struct isc_device *isc, struct v4l2_format *f)
> > >>> +static int isc_validate(struct isc_device *isc)
> > >>> {
> > >>> + int ret;
> > >>> + int i;
> > >>> + struct isc_format *sd_fmt = NULL;
> > >>> + struct v4l2_pix_format *pixfmt = &isc->fmt.fmt.pix;
> > >>> struct v4l2_subdev_format format = {
> > >>> .which = V4L2_SUBDEV_FORMAT_ACTIVE,
> > >>> + .pad = isc->remote_pad,
> > >>> + };
> > >>> + struct v4l2_subdev_pad_config pad_cfg = {};
> > >>> + struct v4l2_subdev_state pad_state = {
> > >>> + .pads = &pad_cfg,
> > >>> };
> > >>> - u32 mbus_code = 0;
> > >>> - int ret;
> > >>>
> > >>> - ret = isc_try_fmt(isc, f, &mbus_code);
> > >>> + /* Get current format from subdev */
> > >>> + ret = v4l2_subdev_call(isc->current_subdev->sd, pad, get_fmt, NULL,
> > >>> + &format);
> > >>> if (ret)
> > >>> return ret;
> > >>>
> > >>> - v4l2_fill_mbus_format(&format.format, &f->fmt.pix, mbus_code);
> > >>> - ret = v4l2_subdev_call(isc->current_subdev->sd, pad,
> > >>> - set_fmt, NULL, &format);
> > >>> - if (ret < 0)
> > >>> - return ret;
> > >>> + /* Identify the subdev's format configuration */
> > >>> + for (i = 0; i < isc->formats_list_size; i++)
> > >>> + if (isc->formats_list[i].mbus_code == format.format.code) {
> > >>> + sd_fmt = &isc->formats_list[i];
> > >>> + break;
> > >>> + }
> > >>> +
> > >>> + /* Check if the format is not supported */
> > >>> + if (!sd_fmt) {
> > >>> + v4l2_err(&isc->v4l2_dev,
> > >>> + "Current subdevice is streaming a media bus code that is not supported 0x%x\n",
> > >>> + format.format.code);
> > >>> + return -EPIPE;
> > >>> + }
> > >>> +
> > >>> + /* At this moment we know which format the subdev will use */
> > >>> + isc->try_config.sd_format = sd_fmt;
> > >>> +
> > >>> + /* If the sensor is not RAW, we can only do a direct dump */
> > >>> + if (!ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code))
> > >>> + isc_try_configure_rlp_dma(isc, true);
> > >>>
> > >>> /* Limit to Atmel ISC hardware capabilities */
> > >>> - if (f->fmt.pix.width > isc->max_width)
> > >>> - f->fmt.pix.width = isc->max_width;
> > >>> - if (f->fmt.pix.height > isc->max_height)
> > >>> - f->fmt.pix.height = isc->max_height;
> > >>> + v4l_bound_align_image(&format.format.width, 16, isc->max_width, 0,
> > >>> + &format.format.height, 16, isc->max_height, 0, 0);
> > >>>
> > >>> - isc->fmt = *f;
> > >>> + /* Check if the frame size is the same. Otherwise we may overflow */
> > >>> + if (pixfmt->height != format.format.height ||
> > >>> + pixfmt->width != format.format.width) {
> > >>> + v4l2_err(&isc->v4l2_dev,
> > >>> + "ISC not configured with the proper frame size: %dx%d\n",
> > >>> + format.format.width, format.format.height);
> > >>> + return -EPIPE;
> > >>> + }
> > >>>
> > >>> + v4l2_dbg(1, debug, &isc->v4l2_dev,
> > >>> + "Identified subdev using format %.4s with %dx%d %d bpp\n",
> > >>> + (char *)&sd_fmt->fourcc, pixfmt->width, pixfmt->height,
> > >>> + isc->try_config.bpp);
> > >>> +
> > >>> + /* Reset and restart AWB if the subdevice changed the format */
> > >>> if (isc->try_config.sd_format && isc->config.sd_format &&
> > >>> isc->try_config.sd_format != isc->config.sd_format) {
> > >>> isc->ctrls.hist_stat = HIST_INIT;
> > >>> isc_reset_awb_ctrls(isc);
> > >>> isc_update_v4l2_ctrls(isc);
> > >>> }
> > >>> - /* make the try configuration active */
> > >>> +
> > >>> + /* Validate formats */
> > >>> + ret = isc_try_validate_formats(isc);
> > >>> + if (ret)
> > >>> + return ret;
> > >>> +
> > >>> + /* Obtain frame sizes if possible to have crop requirements ready */
> > >>> + isc_try_fse(isc, &pad_state);
> > >>> +
> > >>> + /* Configure ISC pipeline for the config */
> > >>> + ret = isc_try_configure_pipeline(isc);
> > >>> + if (ret)
> > >>> + return ret;
> > >>> +
> > >>> isc->config = isc->try_config;
> > >>>
> > >>> v4l2_dbg(1, debug, &isc->v4l2_dev, "New ISC configuration in place\n");
> > >>> @@ -1043,7 +1036,7 @@ static int isc_try_fmt_vid_cap(struct file *file, void *priv,
> > >>> {
> > >>> struct isc_device *isc = video_drvdata(file);
> > >>>
> > >>> - return isc_try_fmt(isc, f, NULL);
> > >>> + return isc_try_fmt(isc, f);
> > >>> }
> > >>>
> > >>> static int isc_enum_input(struct file *file, void *priv,
> > >>> @@ -1098,10 +1091,6 @@ static int isc_enum_framesizes(struct file *file, void *fh,
> > >>> if (fsize->index)
> > >>> return -EINVAL;
> > >>>
> > >>> - for (i = 0; i < isc->num_user_formats; i++)
> > >>> - if (isc->user_formats[i]->fourcc == fsize->pixel_format)
> > >>> - ret = 0;
> > >>> -
> > >>> for (i = 0; i < isc->controller_formats_size; i++)
> > >>> if (isc->controller_formats[i].fourcc == fsize->pixel_format)
> > >>> ret = 0;
> > >>> @@ -1782,52 +1771,6 @@ struct isc_format *isc_find_format_by_code(struct isc_device *isc,
> > >>> }
> > >>> EXPORT_SYMBOL_GPL(isc_find_format_by_code);
> > >>>
> > >>> -static int isc_formats_init(struct isc_device *isc)
> > >>> -{
> > >>> - struct isc_format *fmt;
> > >>> - struct v4l2_subdev *subdev = isc->current_subdev->sd;
> > >>> - unsigned int num_fmts, i, j;
> > >>> - u32 list_size = isc->formats_list_size;
> > >>> - struct v4l2_subdev_mbus_code_enum mbus_code = {
> > >>> - .which = V4L2_SUBDEV_FORMAT_ACTIVE,
> > >>> - };
> > >>> -
> > >>> - num_fmts = 0;
> > >>> - while (!v4l2_subdev_call(subdev, pad, enum_mbus_code,
> > >>> - NULL, &mbus_code)) {
> > >>> - mbus_code.index++;
> > >>> -
> > >>> - fmt = isc_find_format_by_code(isc, mbus_code.code, &i);
> > >>> - if (!fmt) {
> > >>> - v4l2_warn(&isc->v4l2_dev, "Mbus code %x not supported\n",
> > >>> - mbus_code.code);
> > >>> - continue;
> > >>> - }
> > >>> -
> > >>> - fmt->sd_support = true;
> > >>> - num_fmts++;
> > >>> - }
> > >>> -
> > >>> - if (!num_fmts)
> > >>> - return -ENXIO;
> > >>> -
> > >>> - isc->num_user_formats = num_fmts;
> > >>> - isc->user_formats = devm_kcalloc(isc->dev,
> > >>> - num_fmts, sizeof(*isc->user_formats),
> > >>> - GFP_KERNEL);
> > >>> - if (!isc->user_formats)
> > >>> - return -ENOMEM;
> > >>> -
> > >>> - fmt = &isc->formats_list[0];
> > >>> - for (i = 0, j = 0; i < list_size; i++) {
> > >>> - if (fmt->sd_support)
> > >>> - isc->user_formats[j++] = fmt;
> > >>> - fmt++;
> > >>> - }
> > >>> -
> > >>> - return 0;
> > >>> -}
> > >>> -
> > >>> static int isc_set_default_fmt(struct isc_device *isc)
> > >>> {
> > >>> struct v4l2_format f = {
> > >>> @@ -1836,12 +1779,12 @@ static int isc_set_default_fmt(struct isc_device *isc)
> > >>> .width = VGA_WIDTH,
> > >>> .height = VGA_HEIGHT,
> > >>> .field = V4L2_FIELD_NONE,
> > >>> - .pixelformat = isc->user_formats[0]->fourcc,
> > >>> + .pixelformat = isc->controller_formats[0].fourcc,
> > >>> },
> > >>> };
> > >>> int ret;
> > >>>
> > >>> - ret = isc_try_fmt(isc, &f, NULL);
> > >>> + ret = isc_try_fmt(isc, &f);
> > >>> if (ret)
> > >>> return ret;
> > >>>
> > >>> @@ -1896,13 +1839,6 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier)
> > >>> spin_lock_init(&isc->dma_queue_lock);
> > >>> spin_lock_init(&isc->awb_lock);
> > >>>
> > >>> - ret = isc_formats_init(isc);
> > >>> - if (ret < 0) {
> > >>> - v4l2_err(&isc->v4l2_dev,
> > >>> - "Init format failed: %d\n", ret);
> > >>> - goto isc_async_complete_err;
> > >>> - }
> > >>> -
> > >>> ret = isc_set_default_fmt(isc);
> > >>> if (ret) {
> > >>> v4l2_err(&isc->v4l2_dev, "Could not set default format\n");
> > >>> @@ -2016,6 +1952,24 @@ int isc_pipeline_init(struct isc_device *isc)
> > >>> }
> > >>> EXPORT_SYMBOL_GPL(isc_pipeline_init);
> > >>>
> > >>> +static int isc_link_validate(struct media_link *link)
> > >>> +{
> > >>> + struct video_device *vdev =
> > >>> + media_entity_to_video_device(link->sink->entity);
> > >>> + struct isc_device *isc = video_get_drvdata(vdev);
> > >>> + int ret;
> > >>> +
> > >>> + ret = v4l2_subdev_link_validate(link);
> > >>> + if (ret)
> > >>> + return ret;
> > >>> +
> > >>> + return isc_validate(isc);
> > >>> +}
> > >>> +
> > >>> +static const struct media_entity_operations isc_entity_operations = {
> > >>> + .link_validate = isc_link_validate,
> > >>> +};
> > >>> +
> > >>> int isc_mc_init(struct isc_device *isc, u32 ver)
> > >>> {
> > >>> const struct of_device_id *match;
> > >>> @@ -2023,6 +1977,8 @@ int isc_mc_init(struct isc_device *isc, u32 ver)
> > >>>
> > >>> isc->video_dev.entity.function = MEDIA_ENT_F_IO_V4L;
> > >>> isc->video_dev.entity.flags = MEDIA_ENT_FL_DEFAULT;
> > >>> + isc->video_dev.entity.ops = &isc_entity_operations;
> > >>> +
> > >>> isc->pads[ISC_PAD_SINK].flags = MEDIA_PAD_FL_SINK;
> > >>>
> > >>> ret = media_entity_pads_init(&isc->video_dev.entity, ISC_PADS_NUM,
> > >>> diff --git a/drivers/media/platform/atmel/atmel-isc-scaler.c b/drivers/media/platform/atmel/atmel-isc-scaler.c
> > >>> index a1ca4633787c..2b0621297dfe 100644
> > >>> --- a/drivers/media/platform/atmel/atmel-isc-scaler.c
> > >>> +++ b/drivers/media/platform/atmel/atmel-isc-scaler.c
> > >>> @@ -173,6 +173,10 @@ static const struct v4l2_subdev_pad_ops isc_scaler_pad_ops = {
> > >>> .init_cfg = isc_scaler_init_cfg,
> > >>> };
> > >>>
> > >>> +static const struct media_entity_operations isc_scaler_entity_ops = {
> > >>> + .link_validate = v4l2_subdev_link_validate,
> > >>> +};
> > >>> +
> > >>> static const struct v4l2_subdev_ops xisc_scaler_subdev_ops = {
> > >>> .pad = &isc_scaler_pad_ops,
> > >>> };
> > >>> @@ -190,6 +194,7 @@ int isc_scaler_init(struct isc_device *isc)
> > >>>
> > >>> isc->scaler_sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
> > >>> isc->scaler_sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
> > >>> + isc->scaler_sd.entity.ops = &isc_scaler_entity_ops;
> > >>> isc->scaler_pads[ISC_SCALER_PAD_SINK].flags = MEDIA_PAD_FL_SINK;
> > >>> isc->scaler_pads[ISC_SCALER_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
> > >>>
> > >>> diff --git a/drivers/media/platform/atmel/atmel-isc.h b/drivers/media/platform/atmel/atmel-isc.h
> > >>> index f98f25a55e73..5b0100c8c245 100644
> > >>> --- a/drivers/media/platform/atmel/atmel-isc.h
> > >>> +++ b/drivers/media/platform/atmel/atmel-isc.h
> > >>> @@ -63,15 +63,16 @@ struct isc_subdev_entity {
> > >>> * @cfa_baycfg: If this format is RAW BAYER, indicate the type of bayer.
> > >>> this is either BGBG, RGRG, etc.
> > >>> * @pfe_cfg0_bps: Number of hardware data lines connected to the ISC
> > >>> + * @raw: If the format is raw bayer.
> > >>> */
> > >>>
> > >>> struct isc_format {
> > >>> u32 fourcc;
> > >>> u32 mbus_code;
> > >>> u32 cfa_baycfg;
> > >>> -
> > >>> - bool sd_support;
> > >>> u32 pfe_cfg0_bps;
> > >>> +
> > >>> + bool raw;
> > >>> };
> > >>>
> > >>> /* Pipeline bitmap */
> > >>> @@ -216,8 +217,7 @@ enum isc_scaler_pads {
> > >>> * @comp: completion reference that signals frame completion
> > >>> *
> > >>> * @fmt: current v42l format
> > >>> - * @user_formats: list of formats that are supported and agreed with sd
> > >>> - * @num_user_formats: how many formats are in user_formats
> > >>> + * @try_fmt: current v4l2 try format
> > >>> *
> > >>> * @config: current ISC format configuration
> > >>> * @try_config: the current ISC try format , not yet activated
> > >>> @@ -272,6 +272,7 @@ enum isc_scaler_pads {
> > >>> * @formats_list_size: size of formats_list array
> > >>> * @pads: media controller pads for isc video entity
> > >>> * @mdev: media device that is registered by the isc
> > >>> + * @mpipe: media device pipeline used by the isc
> > >>> * @remote_pad: remote pad on the connected subdevice
> > >>> * @scaler_sd: subdevice for the scaler that isc registers
> > >>> * @scaler_pads: media controller pads for the scaler subdevice
> > >>> @@ -298,8 +299,7 @@ struct isc_device {
> > >>> struct completion comp;
> > >>>
> > >>> struct v4l2_format fmt;
> > >>> - struct isc_format **user_formats;
> > >>> - unsigned int num_user_formats;
> > >>> + struct v4l2_format try_fmt;
> > >>>
> > >>> struct fmt_config config;
> > >>> struct fmt_config try_config;
> > >>> @@ -369,6 +369,7 @@ struct isc_device {
> > >>> struct {
> > >>> struct media_pad pads[ISC_PADS_NUM];
> > >>> struct media_device mdev;
> > >>> + struct media_pipeline mpipe;
> > >>>
> > >>> u32 remote_pad;
> > >>> };
> > >>> diff --git a/drivers/media/platform/atmel/atmel-sama5d2-isc.c b/drivers/media/platform/atmel/atmel-sama5d2-isc.c
> > >>> index d96ee3373889..2a651e237193 100644
> > >>> --- a/drivers/media/platform/atmel/atmel-sama5d2-isc.c
> > >>> +++ b/drivers/media/platform/atmel/atmel-sama5d2-isc.c
> > >>> @@ -80,20 +80,40 @@ static const struct isc_format sama5d2_controller_formats[] = {
> > >>> .fourcc = V4L2_PIX_FMT_Y10,
> > >>> }, {
> > >>> .fourcc = V4L2_PIX_FMT_SBGGR8,
> > >>> + .raw = true,
> > >>> }, {
> > >>> .fourcc = V4L2_PIX_FMT_SGBRG8,
> > >>> + .raw = true,
> > >>> }, {
> > >>> .fourcc = V4L2_PIX_FMT_SGRBG8,
> > >>> + .raw = true,
> > >>> }, {
> > >>> .fourcc = V4L2_PIX_FMT_SRGGB8,
> > >>> + .raw = true,
> > >>> }, {
> > >>> .fourcc = V4L2_PIX_FMT_SBGGR10,
> > >>> + .raw = true,
> > >>> }, {
> > >>> .fourcc = V4L2_PIX_FMT_SGBRG10,
> > >>> + .raw = true,
> > >>> }, {
> > >>> .fourcc = V4L2_PIX_FMT_SGRBG10,
> > >>> + .raw = true,
> > >>> }, {
> > >>> .fourcc = V4L2_PIX_FMT_SRGGB10,
> > >>> + .raw = true,
> > >>> + }, {
> > >>> + .fourcc = V4L2_PIX_FMT_SBGGR12,
> > >>> + .raw = true,
> > >>> + }, {
> > >>> + .fourcc = V4L2_PIX_FMT_SGBRG12,
> > >>> + .raw = true,
> > >>> + }, {
> > >>> + .fourcc = V4L2_PIX_FMT_SGRBG12,
> > >>> + .raw = true,
> > >>> + }, {
> > >>> + .fourcc = V4L2_PIX_FMT_SRGGB12,
> > >>> + .raw = true,
> > >>> },
> > >>> };
> > >>>
> > >>> diff --git a/drivers/media/platform/atmel/atmel-sama7g5-isc.c b/drivers/media/platform/atmel/atmel-sama7g5-isc.c
> > >>> index e07ae188c15f..a0d60cfdba7b 100644
> > >>> --- a/drivers/media/platform/atmel/atmel-sama7g5-isc.c
> > >>> +++ b/drivers/media/platform/atmel/atmel-sama7g5-isc.c
> > >>> @@ -89,20 +89,40 @@ static const struct isc_format sama7g5_controller_formats[] = {
> > >>> .fourcc = V4L2_PIX_FMT_Y16,
> > >>> }, {
> > >>> .fourcc = V4L2_PIX_FMT_SBGGR8,
> > >>> + .raw = true,
> > >>> }, {
> > >>> .fourcc = V4L2_PIX_FMT_SGBRG8,
> > >>> + .raw = true,
> > >>> }, {
> > >>> .fourcc = V4L2_PIX_FMT_SGRBG8,
> > >>> + .raw = true,
> > >>> }, {
> > >>> .fourcc = V4L2_PIX_FMT_SRGGB8,
> > >>> + .raw = true,
> > >>> }, {
> > >>> .fourcc = V4L2_PIX_FMT_SBGGR10,
> > >>> + .raw = true,
> > >>> }, {
> > >>> .fourcc = V4L2_PIX_FMT_SGBRG10,
> > >>> + .raw = true,
> > >>> }, {
> > >>> .fourcc = V4L2_PIX_FMT_SGRBG10,
> > >>> + .raw = true,
> > >>> }, {
> > >>> .fourcc = V4L2_PIX_FMT_SRGGB10,
> > >>> + .raw = true,
> > >>> + }, {
> > >>> + .fourcc = V4L2_PIX_FMT_SBGGR12,
> > >>> + .raw = true,
> > >>> + }, {
> > >>> + .fourcc = V4L2_PIX_FMT_SGBRG12,
> > >>> + .raw = true,
> > >>> + }, {
> > >>> + .fourcc = V4L2_PIX_FMT_SGRBG12,
> > >>> + .raw = true,
> > >>> + }, {
> > >>> + .fourcc = V4L2_PIX_FMT_SRGGB12,
> > >>> + .raw = true,
> > >>> },
> > >>> };
> > >>>

--
Regards,

Laurent Pinchart

2022-04-29 12:27:15

[permalink] [raw]

Subject: Re: [PATCH v9 08/13] media: atmel: atmel-isc: change format propagation to subdev into only verification

On Fri, Apr 29, 2022 at 12:13:46PM +0200, Hans Verkuil wrote:
> On 29/04/2022 12:07, Laurent Pinchart wrote:
> > On Fri, Apr 29, 2022 at 11:58:48AM +0200, Jacopo Mondi wrote:
> >> On Fri, Apr 29, 2022 at 10:43:09AM +0200, Hans Verkuil wrote:
> >>> On 29/04/2022 10:28, [email protected] wrote:
> >>>> On 4/29/22 11:17 AM, Hans Verkuil wrote:
> >>>>> On 10/03/2022 10:51, Eugen Hristev wrote:
> >>>>>> As a top MC video driver, the atmel-isc should not propagate the format to the
> >>>>>> subdevice, it should rather check at start_streaming() time if the subdev is properly
> >>>>>> configured with a compatible format.
> >>>>>> Removed the whole format finding logic, and reworked the format verification
> >>>>>> at start_streaming time, such that the ISC will return an error if the subdevice
> >>>>>> is not properly configured. To achieve this, media_pipeline_start
> >>>>>> is called and a link_validate callback is created to check the formats.
> >>>>>> With this being done, the module parameter 'sensor_preferred' makes no sense
> >>>>>> anymore. The ISC should not decide which format the sensor is using. The
> >>>>>> ISC should only cope with the situation and inform userspace if the streaming
> >>>>>> is possible in the current configuration.
> >>>>>> The redesign of the format propagation has also risen the question of the
> >>>>>> enumfmt callback. If enumfmt is called with an mbus_code, the enumfmt handler
> >>>>>> should only return the formats that are supported for this mbus_code.
> >>>>>> Otherwise, the enumfmt will report all the formats that the ISC could output.
> >>>>>> With this rework, the dynamic list of user formats is removed. It makes no
> >>>>>> more sense to identify at complete time which formats the sensor could emit,
> >>>>>> and add those into a separate dynamic list.
> >>>>>> The ISC will start with a simple preconfigured default format, and at
> >>>>>> link validate time, decide whether it can use the format that is configured
> >>>>>> on the sink or not.
> >>>>>>
> >>>>>> Signed-off-by: Eugen Hristev <[email protected]>
> >>>>>> Reviewed-by: Jacopo Mondi <[email protected]>
> >>>>>> ---
> >>>>>> Changes in v9:
> >>>>>> - isc_link_validate now static
> >>>>>>
> >>>>>> Changes in v7:
> >>>>>> - minor typos as suggested by Jacopo
> >>>>>> - small changes, reduce some indentation, modified an index, as suggested by
> >>>>>> Jacopo
> >>>>>>
> >>>>>> Changes in v6:
> >>>>>> - reworked a bit enum_fmt as suggested by Jacopo
> >>>>>>
> >>>>>> Changes in v5:
> >>>>>> - removed user_formats dynamic list as it is now pointless
> >>>>>> - greatly simplified the enum_fmt function
> >>>>>> - removed some init code that was useless now
> >>>>>>
> >>>>>> Changes in v4:
> >>>>>> - moved validation code into link_validate and used media_pipeline_start
> >>>>>> - merged this patch with the enum_fmt patch which was previously in v3 of
> >>>>>> the series
> >>>>>>
> >>>>>> Changes in v3:
> >>>>>> - clamp to maximum resolution once the frame size from the subdev is found
> >>>>>> drivers/media/platform/atmel/atmel-isc-base.c | 412 ++++++++----------
> >>>>>> .../media/platform/atmel/atmel-isc-scaler.c | 5 +
> >>>>>> drivers/media/platform/atmel/atmel-isc.h | 13 +-
> >>>>>> .../media/platform/atmel/atmel-sama5d2-isc.c | 20 +
> >>>>>> .../media/platform/atmel/atmel-sama7g5-isc.c | 20 +
> >>>>>> 5 files changed, 236 insertions(+), 234 deletions(-)
> >>>>>>
> >>>>>> diff --git a/drivers/media/platform/atmel/atmel-isc-base.c b/drivers/media/platform/atmel/atmel-isc-base.c
> >>>>>> index ee1dda6707a0..fe2c0af58060 100644
> >>>>>> --- a/drivers/media/platform/atmel/atmel-isc-base.c
> >>>>>> +++ b/drivers/media/platform/atmel/atmel-isc-base.c
> >>>>>> @@ -36,11 +36,6 @@ static unsigned int debug;
> >>>>>> module_param(debug, int, 0644);
> >>>>>> MODULE_PARM_DESC(debug, "debug level (0-2)");
> >>>>>>
> >>>>>> -static unsigned int sensor_preferred = 1;
> >>>>>> -module_param(sensor_preferred, uint, 0644);
> >>>>>> -MODULE_PARM_DESC(sensor_preferred,
> >>>>>> - "Sensor is preferred to output the specified format (1-on 0-off), default 1");
> >>>>>> -
> >>>>>> #define ISC_IS_FORMAT_RAW(mbus_code) \
> >>>>>> (((mbus_code) & 0xf000) == 0x3000)
> >>>>>>
> >>>>>> @@ -337,6 +332,10 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)
> >>>>>> unsigned long flags;
> >>>>>> int ret;
> >>>>>>
> >>>>>> + ret = media_pipeline_start(&isc->video_dev.entity, &isc->mpipe);
> >>>>>
> >>>>> The pipeline validation is done in start_streaming, but I don't think that
> >>>>> is the best place: if STREAMON is called before buffers are queued, then
> >>>>> an invalid pipeline isn't discovered until enough buffers are queued to
> >>>>> kick off start_streaming.
> >>>>>
> >>>>> Drivers like vsp1, omap3isp and the samsung drivers all do this in streamon().
> >>>>>
> >>>>> I think that is the correct time to do this.
> >>>>
> >>>> Hello Hans,
> >>>>
> >>>> Initially (v2, v3) I had this in streamon(). The problem that I faced at
> >>>> that time was that streamoff was never called, so I could not call
> >>>> media_pipeline_stop(). Then Jacopo told me to move it to start_streaming
> >>>> (see change log for v4) , and I did not face any more problems.
> >>
> >> Yes indeed, seems I suggested to use media_pipeline_handler in a
> >> comment on your v3
> >>
> >> "at s_stream time your top driver calls media_pipeline_start()"
> >>
> >> sorry about that, I should have looked around a bit more carefully and
> >> notice most drivers do so at vb2 streamon
> >>
> >> However I don't see media_pipeline_start being called at all in v3 of
> >> the patch
> >>
> >>> It's a mess. Looking at some drivers I see that omap3isp calls media_pipeline_stop
> >>> in streamoff (so will have the same problem as you described if VIDIOC_STREAMOFF
> >>> isn't called), exynos4-is does the same, but it also checks the streaming state in
> >>> the release() fop callback, so that would fix this problem. And vimc does this
> >>> in stop_streaming.
> >>>
> >>> I'm in favor of fixing this in vb2, that framework knows exactly when this needs
> >>> to be called.
> >>
> >> Are you suggesting to have vb2 to call media_pipeline_start() or is it
> >> more complex than this ?
> >
> > I think Hans meant adding a .validate() operation to vb2.
> >
> > vb2 is already quite complex, I don't think adding more features is a
> > good idea. I'd rather have vb2 focus on buffer management only
> > (.start_streaming() and .stop_streaming() shouldn't have been in there
> > in my opinion), and handle validation in the .streamon() handler. I'd
> > expect most drivers that deal with media pipelines to do more work in
> > .streamon() anyway.
>
> I disagree with that :-)
>
> It's vb2 that keeps track of the streaming state and when what actions
> need to be taken. Drivers really shouldn't need to care about the ioctls
> themselves, and just implement the relevant vb2 callbacks. Relying on
> drivers to handle any of the streaming ioctls is asking for problems,
> as this shows: most drivers implement this wrong today.
>
> The vb2 framework knows when e.g. the pipeline needs to be started or
> stopped, and can do this at the best time, without drivers needing to
> keep track of when streamon/off/release is called. Keep that logic in
> vb2.

Pipeline management and buffer management are two different issues.
Don't forget about devices that have multiple video nodes, part of the
same pipeline (possibly a combination of output and capture nodes, or
all of the same type). Forcing drivers to go through vb2 operations to
handle the pipeline will be messy, will result in more bloat in vb2, and
make the result more bug-prone and harder to maintain.

If pipeline management is too complex, let's simplify it, new helpers
can make sense, but not through vb2.

--
Regards,

Laurent Pinchart

2022-04-29 14:47:17

[permalink] [raw]

Subject: Re: [PATCH v9 00/13] media: atmel: atmel-isc: implement media controller

On 4/29/22 11:33 AM, Hans Verkuil wrote:
> On 29/04/2022 10:20, [email protected] wrote:
>> On 4/29/22 11:05 AM, Hans Verkuil wrote:
>>> Hi Eugen,
>>>
>>> On 10/03/2022 10:51, Eugen Hristev wrote:
>>>> This series is the v9 series that attempts to support media controller in the
>>>> atmel ISC and XISC drivers.
>>>> The CSI2DC driver was accepted thus removed from the patch series, together with
>>>> other patches.
>>>>
>>>> Important note: this series applies on top of current media_staging tree, as it
>>>> relies on previous patches in the series which were accepted.
>>>>
>>>> Thanks to everyone who reviewed my work !
>>>>
>>>> Eugen
>>>>
>>>> Changes in v9:
>>>> -> kernel robot reported isc_link_validate is not static, changed to static.
>>>>
>>>> Changes in v8:
>>>> -> scaler: modified crop bounds to have the exact source size
>>>>
>>>> Changes in v7:
>>>> -> scaler: modified crop bounds to have maximum isc size
>>>> -> format propagation: did small changes as per Jacopo review
>>>>
>>>>
>>>> Changes in v6:
>>>> -> worked a bit on scaler, added try crop and other changes as per Jacopo review
>>>> -> worked on isc-base enum_fmt , reworked as per Jacopo review
>>>>
>>>> Changes in v5:
>>>> -> removed patch that removed the 'stop' variable as it was still required
>>>> -> added two new trivial patches
>>>> -> reworked some parts of the scaler and format propagation after discussions with Jacopo
>>>>
>>>>
>>>> Changes in v4:
>>>> -> as reviewed by Hans, added new patch to remove the 'stop' variable and reworked
>>>> one patch that was using it
>>>> -> as reviewed by Jacopo, reworked some parts of the media controller implementation
>>>>
>>>>
>>>> Changes in v3:
>>>> - change in bindings, small fixes in csi2dc driver and conversion to mc
>>>> for the isc-base.
>>>> - removed some MAINTAINERS patches and used patterns in MAINTAINERS
>>>>
>>>> Changes in v2:
>>>> - integrated many changes suggested by Jacopo in the review of the v1 series.
>>>> - add a few new patches
>>>>
>>>>
>>>>
>>>> Eugen Hristev (13):
>>>> media: atmel: atmel-isc-base: use streaming status when queueing
>>>> buffers
>>>> media: atmel: atmel-isc-base: replace is_streaming call in
>>>> s_fmt_vid_cap
>>>> media: atmel: atmel-isc: remove redundant comments
>>>> media: atmel: atmel-isc: implement media controller
>>>> media: atmel: atmel-sama5d2-isc: fix wrong mask in YUYV format check
>>>> media: atmel: atmel-isc-base: use mutex to lock awb workqueue from
>>>> streaming
>>>> media: atmel: atmel-isc: compact the controller formats list
>>>> media: atmel: atmel-isc: change format propagation to subdev into only
>>>> verification
>>>> media: atmel: atmel-sama7g5-isc: remove stray line
>>>> dt-bindings: media: microchip,xisc: add bus-width of 14
>>>
>>> I'm a bit unhappy with the order of these patches. Mostly these are fixes,
>>> except for patches 4 and 8, which are the meat of this series and actually
>>> switching on the MC support.
>>>
>>> Can those be moved to the end? That also makes it easier to merge the earlier
>>> patches if some more work is needed for the MC part.
>>>
>>> I'm also not sure whether patches 4 and 8 shouldn't be a single patch,
>>> since patch 4 leaves the driver in an inconsistent state since it is
>>> missing the link validation code that patch 8 adds. Unless I missed
>>> something?
>>
>> Hello Hans,
>>
>> The difference that patch 8 is making is that the 'old way' of
>> configuring the ISC is no longer possible.
>>
>> Patch 4 makes the ISC 'mc-ready' with all entities, links, but the old
>> way still works (meaning that the top driver will call s_fmt down to the
>> subdev ).
>> After patch 8, the driver no longer has this support at all, and
>> validates links at start_streaming, and no longer sets anything to the
>> subdev, just validates the config that the subdev already has.
>> So one reason that I had things in two patches was that patch 8 makes
>> this big change that also makes userspace behave differently and has to
>> configure all the subdevs and media pipeline.
>>
>> If you feel patch 4 and patch 8 should be squashed, I can do it, definitely.
>>
>> Let me know how to proceed ?
>
> Ah, that was your intention. The problem in patch 4 is that you set
> V4L2_CAP_IO_MC, which indicates to applications that they have to use
> the MC and configure the pipeline correctly. Moving that to patch 8
> should resolve that, I think.

Okay , got it.

Do you wish to apply some of the patches ? like maybe 1,2,3, 5, 7, 9, 10
? I can then make the series smaller for the next iteration. Otherwise I
will resend everything, no problem.

Eugen

>
> Regards,
>
> Hans
>
>>
>> Thanks,
>> Eugen
>>
>>>
>>> Regards,
>>>
>>> Hans
>>>
>>>> ARM: dts: at91: sama7g5: add nodes for video capture
>>>> ARM: configs: at91: sama7: add xisc and csi2dc
>>>> ARM: multi_v7_defconfig: add atmel video pipeline modules
>>>>
>>>> .../bindings/media/microchip,xisc.yaml | 2 +-
>>>> arch/arm/boot/dts/sama7g5.dtsi | 49 ++
>>>> arch/arm/configs/multi_v7_defconfig | 3 +
>>>> arch/arm/configs/sama7_defconfig | 2 +
>>>> drivers/media/platform/atmel/Makefile | 2 +-
>>>> drivers/media/platform/atmel/atmel-isc-base.c | 518 ++++++++++--------
>>>> .../media/platform/atmel/atmel-isc-scaler.c | 267 +++++++++
>>>> drivers/media/platform/atmel/atmel-isc.h | 58 +-
>>>> .../media/platform/atmel/atmel-sama5d2-isc.c | 87 +--
>>>> .../media/platform/atmel/atmel-sama7g5-isc.c | 93 ++--
>>>> 10 files changed, 754 insertions(+), 327 deletions(-)
>>>> create mode 100644 drivers/media/platform/atmel/atmel-isc-scaler.c
>>>>
>>>
>>
>

2022-04-29 15:46:18

[permalink] [raw]

Subject: Re: [PATCH v9 00/13] media: atmel: atmel-isc: implement media controller

On 29/04/2022 10:39, [email protected] wrote:
> On 4/29/22 11:33 AM, Hans Verkuil wrote:
>> On 29/04/2022 10:20, [email protected] wrote:
>>> On 4/29/22 11:05 AM, Hans Verkuil wrote:
>>>> Hi Eugen,
>>>>
>>>> On 10/03/2022 10:51, Eugen Hristev wrote:
>>>>> This series is the v9 series that attempts to support media controller in the
>>>>> atmel ISC and XISC drivers.
>>>>> The CSI2DC driver was accepted thus removed from the patch series, together with
>>>>> other patches.
>>>>>
>>>>> Important note: this series applies on top of current media_staging tree, as it
>>>>> relies on previous patches in the series which were accepted.
>>>>>
>>>>> Thanks to everyone who reviewed my work !
>>>>>
>>>>> Eugen
>>>>>
>>>>> Changes in v9:
>>>>> -> kernel robot reported isc_link_validate is not static, changed to static.
>>>>>
>>>>> Changes in v8:
>>>>> -> scaler: modified crop bounds to have the exact source size
>>>>>
>>>>> Changes in v7:
>>>>> -> scaler: modified crop bounds to have maximum isc size
>>>>> -> format propagation: did small changes as per Jacopo review
>>>>>
>>>>>
>>>>> Changes in v6:
>>>>> -> worked a bit on scaler, added try crop and other changes as per Jacopo review
>>>>> -> worked on isc-base enum_fmt , reworked as per Jacopo review
>>>>>
>>>>> Changes in v5:
>>>>> -> removed patch that removed the 'stop' variable as it was still required
>>>>> -> added two new trivial patches
>>>>> -> reworked some parts of the scaler and format propagation after discussions with Jacopo
>>>>>
>>>>>
>>>>> Changes in v4:
>>>>> -> as reviewed by Hans, added new patch to remove the 'stop' variable and reworked
>>>>> one patch that was using it
>>>>> -> as reviewed by Jacopo, reworked some parts of the media controller implementation
>>>>>
>>>>>
>>>>> Changes in v3:
>>>>> - change in bindings, small fixes in csi2dc driver and conversion to mc
>>>>> for the isc-base.
>>>>> - removed some MAINTAINERS patches and used patterns in MAINTAINERS
>>>>>
>>>>> Changes in v2:
>>>>> - integrated many changes suggested by Jacopo in the review of the v1 series.
>>>>> - add a few new patches
>>>>>
>>>>>
>>>>>
>>>>> Eugen Hristev (13):
>>>>> media: atmel: atmel-isc-base: use streaming status when queueing
>>>>> buffers
>>>>> media: atmel: atmel-isc-base: replace is_streaming call in
>>>>> s_fmt_vid_cap
>>>>> media: atmel: atmel-isc: remove redundant comments
>>>>> media: atmel: atmel-isc: implement media controller
>>>>> media: atmel: atmel-sama5d2-isc: fix wrong mask in YUYV format check
>>>>> media: atmel: atmel-isc-base: use mutex to lock awb workqueue from
>>>>> streaming
>>>>> media: atmel: atmel-isc: compact the controller formats list
>>>>> media: atmel: atmel-isc: change format propagation to subdev into only
>>>>> verification
>>>>> media: atmel: atmel-sama7g5-isc: remove stray line
>>>>> dt-bindings: media: microchip,xisc: add bus-width of 14
>>>>
>>>> I'm a bit unhappy with the order of these patches. Mostly these are fixes,
>>>> except for patches 4 and 8, which are the meat of this series and actually
>>>> switching on the MC support.
>>>>
>>>> Can those be moved to the end? That also makes it easier to merge the earlier
>>>> patches if some more work is needed for the MC part.
>>>>
>>>> I'm also not sure whether patches 4 and 8 shouldn't be a single patch,
>>>> since patch 4 leaves the driver in an inconsistent state since it is
>>>> missing the link validation code that patch 8 adds. Unless I missed
>>>> something?
>>>
>>> Hello Hans,
>>>
>>> The difference that patch 8 is making is that the 'old way' of
>>> configuring the ISC is no longer possible.
>>>
>>> Patch 4 makes the ISC 'mc-ready' with all entities, links, but the old
>>> way still works (meaning that the top driver will call s_fmt down to the
>>> subdev ).
>>> After patch 8, the driver no longer has this support at all, and
>>> validates links at start_streaming, and no longer sets anything to the
>>> subdev, just validates the config that the subdev already has.
>>> So one reason that I had things in two patches was that patch 8 makes
>>> this big change that also makes userspace behave differently and has to
>>> configure all the subdevs and media pipeline.
>>>
>>> If you feel patch 4 and patch 8 should be squashed, I can do it, definitely.
>>>
>>> Let me know how to proceed ?
>>
>> Ah, that was your intention. The problem in patch 4 is that you set
>> V4L2_CAP_IO_MC, which indicates to applications that they have to use
>> the MC and configure the pipeline correctly. Moving that to patch 8
>> should resolve that, I think.
>
> Okay , got it.
>
> Do you wish to apply some of the patches ? like maybe 1,2,3, 5, 7, 9, 10
> ? I can then make the series smaller for the next iteration. Otherwise I
> will resend everything, no problem.

Hmm, since patch 6 needs to be fixed anyway, I prefer a new series that includes
just the fixes. And later a v10 with just the MC patches.

Regards,

Hans

>
> Eugen
>
>>
>> Regards,
>>
>> Hans
>>
>>>
>>> Thanks,
>>> Eugen
>>>
>>>>
>>>> Regards,
>>>>
>>>> Hans
>>>>
>>>>> ARM: dts: at91: sama7g5: add nodes for video capture
>>>>> ARM: configs: at91: sama7: add xisc and csi2dc
>>>>> ARM: multi_v7_defconfig: add atmel video pipeline modules
>>>>>
>>>>> .../bindings/media/microchip,xisc.yaml | 2 +-
>>>>> arch/arm/boot/dts/sama7g5.dtsi | 49 ++
>>>>> arch/arm/configs/multi_v7_defconfig | 3 +
>>>>> arch/arm/configs/sama7_defconfig | 2 +
>>>>> drivers/media/platform/atmel/Makefile | 2 +-
>>>>> drivers/media/platform/atmel/atmel-isc-base.c | 518 ++++++++++--------
>>>>> .../media/platform/atmel/atmel-isc-scaler.c | 267 +++++++++
>>>>> drivers/media/platform/atmel/atmel-isc.h | 58 +-
>>>>> .../media/platform/atmel/atmel-sama5d2-isc.c | 87 +--
>>>>> .../media/platform/atmel/atmel-sama7g5-isc.c | 93 ++--
>>>>> 10 files changed, 754 insertions(+), 327 deletions(-)
>>>>> create mode 100644 drivers/media/platform/atmel/atmel-isc-scaler.c
>>>>>
>>>>
>>>
>>
>

2022-04-29 19:35:10

[permalink] [raw]

Subject: Re: [PATCH v9 08/13] media: atmel: atmel-isc: change format propagation to subdev into only verification

On 4/29/22 11:17 AM, Hans Verkuil wrote:
> On 10/03/2022 10:51, Eugen Hristev wrote:
>> As a top MC video driver, the atmel-isc should not propagate the format to the
>> subdevice, it should rather check at start_streaming() time if the subdev is properly
>> configured with a compatible format.
>> Removed the whole format finding logic, and reworked the format verification
>> at start_streaming time, such that the ISC will return an error if the subdevice
>> is not properly configured. To achieve this, media_pipeline_start
>> is called and a link_validate callback is created to check the formats.
>> With this being done, the module parameter 'sensor_preferred' makes no sense
>> anymore. The ISC should not decide which format the sensor is using. The
>> ISC should only cope with the situation and inform userspace if the streaming
>> is possible in the current configuration.
>> The redesign of the format propagation has also risen the question of the
>> enumfmt callback. If enumfmt is called with an mbus_code, the enumfmt handler
>> should only return the formats that are supported for this mbus_code.
>> Otherwise, the enumfmt will report all the formats that the ISC could output.
>> With this rework, the dynamic list of user formats is removed. It makes no
>> more sense to identify at complete time which formats the sensor could emit,
>> and add those into a separate dynamic list.
>> The ISC will start with a simple preconfigured default format, and at
>> link validate time, decide whether it can use the format that is configured
>> on the sink or not.
>>
>> Signed-off-by: Eugen Hristev <[email protected]>
>> Reviewed-by: Jacopo Mondi <[email protected]>
>> ---
>> Changes in v9:
>> - isc_link_validate now static
>>
>> Changes in v7:
>> - minor typos as suggested by Jacopo
>> - small changes, reduce some indentation, modified an index, as suggested by
>> Jacopo
>>
>> Changes in v6:
>> - reworked a bit enum_fmt as suggested by Jacopo
>>
>> Changes in v5:
>> - removed user_formats dynamic list as it is now pointless
>> - greatly simplified the enum_fmt function
>> - removed some init code that was useless now
>>
>> Changes in v4:
>> - moved validation code into link_validate and used media_pipeline_start
>> - merged this patch with the enum_fmt patch which was previously in v3 of
>> the series
>>
>> Changes in v3:
>> - clamp to maximum resolution once the frame size from the subdev is found
>> drivers/media/platform/atmel/atmel-isc-base.c | 412 ++++++++----------
>> .../media/platform/atmel/atmel-isc-scaler.c | 5 +
>> drivers/media/platform/atmel/atmel-isc.h | 13 +-
>> .../media/platform/atmel/atmel-sama5d2-isc.c | 20 +
>> .../media/platform/atmel/atmel-sama7g5-isc.c | 20 +
>> 5 files changed, 236 insertions(+), 234 deletions(-)
>>
>> diff --git a/drivers/media/platform/atmel/atmel-isc-base.c b/drivers/media/platform/atmel/atmel-isc-base.c
>> index ee1dda6707a0..fe2c0af58060 100644
>> --- a/drivers/media/platform/atmel/atmel-isc-base.c
>> +++ b/drivers/media/platform/atmel/atmel-isc-base.c
>> @@ -36,11 +36,6 @@ static unsigned int debug;
>> module_param(debug, int, 0644);
>> MODULE_PARM_DESC(debug, "debug level (0-2)");
>>
>> -static unsigned int sensor_preferred = 1;
>> -module_param(sensor_preferred, uint, 0644);
>> -MODULE_PARM_DESC(sensor_preferred,
>> - "Sensor is preferred to output the specified format (1-on 0-off), default 1");
>> -
>> #define ISC_IS_FORMAT_RAW(mbus_code) \
>> (((mbus_code) & 0xf000) == 0x3000)
>>
>> @@ -337,6 +332,10 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)
>> unsigned long flags;
>> int ret;
>>
>> + ret = media_pipeline_start(&isc->video_dev.entity, &isc->mpipe);
>
> The pipeline validation is done in start_streaming, but I don't think that
> is the best place: if STREAMON is called before buffers are queued, then
> an invalid pipeline isn't discovered until enough buffers are queued to
> kick off start_streaming.
>
> Drivers like vsp1, omap3isp and the samsung drivers all do this in streamon().
>
> I think that is the correct time to do this.

Hello Hans,

Initially (v2, v3) I had this in streamon(). The problem that I faced at
that time was that streamoff was never called, so I could not call
media_pipeline_stop(). Then Jacopo told me to move it to start_streaming
(see change log for v4) , and I did not face any more problems.

Eugen

>
> Laurent, Jacopo, do you agree with that?
>
> I notice that quite a few other drivers do this in start_streaming as well,
> but I think it just weird that a QBUF can return EPIPE while STREAMON might
> succeed.
>
> Perhaps we need an optional vb2 validate() callback that is called by
> vb2_core_streamon() to validate the pipeline before streaming. That makes
> it easier to implement these checks rather than having to implement your
> own streamon ioctl.
>
> Regards,
>
> Hans
>
>> + if (ret)
>> + goto err_pipeline_start;
>> +
>> /* Enable stream on the sub device */
>> ret = v4l2_subdev_call(isc->current_subdev->sd, video, s_stream, 1);
>> if (ret && ret != -ENOIOCTLCMD) {
>> @@ -385,6 +384,9 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)
>> v4l2_subdev_call(isc->current_subdev->sd, video, s_stream, 0);
>>
>> err_start_stream:
>> + media_pipeline_stop(&isc->video_dev.entity);
>> +
>> +err_pipeline_start:
>> spin_lock_irqsave(&isc->dma_queue_lock, flags);
>> list_for_each_entry(buf, &isc->dma_queue, list)
>> vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_QUEUED);
>> @@ -423,6 +425,9 @@ static void isc_stop_streaming(struct vb2_queue *vq)
>> if (ret && ret != -ENOIOCTLCMD)
>> v4l2_err(&isc->v4l2_dev, "stream off failed in subdev\n");
>>
>> + /* Stop media pipeline */
>> + media_pipeline_stop(&isc->video_dev.entity);
>> +
>> /* Release all active buffers */
>> spin_lock_irqsave(&isc->dma_queue_lock, flags);
>> if (unlikely(isc->cur_frm)) {
>> @@ -453,22 +458,6 @@ static void isc_buffer_queue(struct vb2_buffer *vb)
>> spin_unlock_irqrestore(&isc->dma_queue_lock, flags);
>> }
>>
>> -static struct isc_format *find_format_by_fourcc(struct isc_device *isc,
>> - unsigned int fourcc)
>> -{
>> - unsigned int num_formats = isc->num_user_formats;
>> - struct isc_format *fmt;
>> - unsigned int i;
>> -
>> - for (i = 0; i < num_formats; i++) {
>> - fmt = isc->user_formats[i];
>> - if (fmt->fourcc == fourcc)
>> - return fmt;
>> - }
>> -
>> - return NULL;
>> -}
>> -
>> static const struct vb2_ops isc_vb2_ops = {
>> .queue_setup = isc_queue_setup,
>> .wait_prepare = vb2_ops_wait_prepare,
>> @@ -497,23 +486,57 @@ static int isc_enum_fmt_vid_cap(struct file *file, void *priv,
>> {
>> struct isc_device *isc = video_drvdata(file);
>> u32 index = f->index;
>> - u32 i, supported_index;
>> + u32 i, supported_index = 0;
>> + struct isc_format *fmt;
>> +
>> + /*
>> + * If we are not asked a specific mbus_code, we have to report all
>> + * the formats that we can output.
>> + */
>> + if (!f->mbus_code) {
>> + if (index >= isc->controller_formats_size)
>> + return -EINVAL;
>>
>> - if (index < isc->controller_formats_size) {
>> f->pixelformat = isc->controller_formats[index].fourcc;
>> +
>> + return 0;
>> + }
>> +
>> + /*
>> + * If a specific mbus_code is requested, check if we support
>> + * this mbus_code as input for the ISC.
>> + * If it's supported, then we report the corresponding pixelformat
>> + * as first possible option for the ISC.
>> + * E.g. mbus MEDIA_BUS_FMT_YUYV8_2X8 and report
>> + * 'YUYV' (YUYV 4:2:2)
>> + */
>> + fmt = isc_find_format_by_code(isc, f->mbus_code, &i);
>> + if (!fmt)
>> + return -EINVAL;
>> +
>> + if (!index) {
>> + f->pixelformat = fmt->fourcc;
>> +
>> return 0;
>> }
>>
>> - index -= isc->controller_formats_size;
>> + supported_index++;
>>
>> - supported_index = 0;
>> + /* If the index is not raw, we don't have anymore formats to report */
>> + if (!ISC_IS_FORMAT_RAW(f->mbus_code))
>> + return -EINVAL;
>>
>> - for (i = 0; i < isc->formats_list_size; i++) {
>> - if (!ISC_IS_FORMAT_RAW(isc->formats_list[i].mbus_code) ||
>> - !isc->formats_list[i].sd_support)
>> + /*
>> + * We are asked for a specific mbus code, which is raw.
>> + * We have to search through the formats we can convert to.
>> + * We have to skip the raw formats, we cannot convert to raw.
>> + * E.g. 'AR12' (16-bit ARGB 4-4-4-4), 'AR15' (16-bit ARGB 1-5-5-5), etc.
>> + */
>> + for (i = 0; i < isc->controller_formats_size; i++) {
>> + if (isc->controller_formats[i].raw)
>> continue;
>> - if (supported_index == index) {
>> - f->pixelformat = isc->formats_list[i].fourcc;
>> + if (index == supported_index) {
>> + f->pixelformat = isc->controller_formats[i].fourcc;
>> return 0;
>> }
>> supported_index++;
>> @@ -584,20 +607,30 @@ static int isc_try_validate_formats(struct isc_device *isc)
>> break;
>> default:
>> /* any other different formats are not supported */
>> + v4l2_err(&isc->v4l2_dev, "Requested unsupported format.\n");
>> ret = -EINVAL;
>> }
>> v4l2_dbg(1, debug, &isc->v4l2_dev,
>> "Format validation, requested rgb=%u, yuv=%u, grey=%u, bayer=%u\n",
>> rgb, yuv, grey, bayer);
>>
>> - /* we cannot output RAW if we do not receive RAW */
>> - if ((bayer) && !ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code))
>> + if (bayer &&
>> + !ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code)) {
>> + v4l2_err(&isc->v4l2_dev, "Cannot output RAW if we do not receive RAW.\n");
>> return -EINVAL;
>> + }
>>
>> - /* we cannot output GREY if we do not receive RAW/GREY */
>> if (grey && !ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code) &&
>> - !ISC_IS_FORMAT_GREY(isc->try_config.sd_format->mbus_code))
>> + !ISC_IS_FORMAT_GREY(isc->try_config.sd_format->mbus_code)) {
>> + v4l2_err(&isc->v4l2_dev, "Cannot output GREY if we do not receive RAW/GREY.\n");
>> return -EINVAL;
>> + }
>> +
>> + if ((rgb || bayer || yuv) &&
>> + ISC_IS_FORMAT_GREY(isc->try_config.sd_format->mbus_code)) {
>> + v4l2_err(&isc->v4l2_dev, "Cannot convert GREY to another format.\n");
>> + return -EINVAL;
>> + }
>>
>> return ret;
>> }
>> @@ -825,7 +858,7 @@ static void isc_try_fse(struct isc_device *isc,
>> * If we do not know yet which format the subdev is using, we cannot
>> * do anything.
>> */
>> - if (!isc->try_config.sd_format)
>> + if (!isc->config.sd_format)
>> return;
>>
>> fse.code = isc->try_config.sd_format->mbus_code;
>> @@ -846,180 +879,140 @@ static void isc_try_fse(struct isc_device *isc,
>> }
>> }
>>
>> -static int isc_try_fmt(struct isc_device *isc, struct v4l2_format *f,
>> - u32 *code)
>> +static int isc_try_fmt(struct isc_device *isc, struct v4l2_format *f)
>> {
>> - int i;
>> - struct isc_format *sd_fmt = NULL, *direct_fmt = NULL;
>> struct v4l2_pix_format *pixfmt = &f->fmt.pix;
>> - struct v4l2_subdev_pad_config pad_cfg = {};
>> - struct v4l2_subdev_state pad_state = {
>> - .pads = &pad_cfg
>> - };
>> - struct v4l2_subdev_format format = {
>> - .which = V4L2_SUBDEV_FORMAT_TRY,
>> - };
>> - u32 mbus_code;
>> - int ret;
>> - bool rlp_dma_direct_dump = false;
>> + unsigned int i;
>>
>> if (f->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
>> return -EINVAL;
>>
>> - /* Step 1: find a RAW format that is supported */
>> - for (i = 0; i < isc->num_user_formats; i++) {
>> - if (ISC_IS_FORMAT_RAW(isc->user_formats[i]->mbus_code)) {
>> - sd_fmt = isc->user_formats[i];
>> + isc->try_config.fourcc = isc->controller_formats[0].fourcc;
>> +
>> + /* find if the format requested is supported */
>> + for (i = 0; i < isc->controller_formats_size; i++)
>> + if (isc->controller_formats[i].fourcc == pixfmt->pixelformat) {
>> + isc->try_config.fourcc = pixfmt->pixelformat;
>> break;
>> }
>> - }
>> - /* Step 2: We can continue with this RAW format, or we can look
>> - * for better: maybe sensor supports directly what we need.
>> - */
>> - direct_fmt = find_format_by_fourcc(isc, pixfmt->pixelformat);
>> -
>> - /* Step 3: We have both. We decide given the module parameter which
>> - * one to use.
>> - */
>> - if (direct_fmt && sd_fmt && sensor_preferred)
>> - sd_fmt = direct_fmt;
>> -
>> - /* Step 4: we do not have RAW but we have a direct format. Use it. */
>> - if (direct_fmt && !sd_fmt)
>> - sd_fmt = direct_fmt;
>> -
>> - /* Step 5: if we are using a direct format, we need to package
>> - * everything as 8 bit data and just dump it
>> - */
>> - if (sd_fmt == direct_fmt)
>> - rlp_dma_direct_dump = true;
>> -
>> - /* Step 6: We have no format. This can happen if the userspace
>> - * requests some weird/invalid format.
>> - * In this case, default to whatever we have
>> - */
>> - if (!sd_fmt && !direct_fmt) {
>> - sd_fmt = isc->user_formats[isc->num_user_formats - 1];
>> - v4l2_dbg(1, debug, &isc->v4l2_dev,
>> - "Sensor not supporting %.4s, using %.4s\n",
>> - (char *)&pixfmt->pixelformat, (char *)&sd_fmt->fourcc);
>> - }
>> -
>> - if (!sd_fmt) {
>> - ret = -EINVAL;
>> - goto isc_try_fmt_err;
>> - }
>> -
>> - /* Step 7: Print out what we decided for debugging */
>> - v4l2_dbg(1, debug, &isc->v4l2_dev,
>> - "Preferring to have sensor using format %.4s\n",
>> - (char *)&sd_fmt->fourcc);
>>
>> - /* Step 8: at this moment we decided which format the subdev will use */
>> - isc->try_config.sd_format = sd_fmt;
>> + isc_try_configure_rlp_dma(isc, false);
>>
>> /* Limit to Atmel ISC hardware capabilities */
>> - if (pixfmt->width > isc->max_width)
>> - pixfmt->width = isc->max_width;
>> - if (pixfmt->height > isc->max_height)
>> - pixfmt->height = isc->max_height;
>> -
>> - /*
>> - * The mbus format is the one the subdev outputs.
>> - * The pixels will be transferred in this format Sensor -> ISC
>> - */
>> - mbus_code = sd_fmt->mbus_code;
>> -
>> - /*
>> - * Validate formats. If the required format is not OK, default to raw.
>> - */
>> -
>> - isc->try_config.fourcc = pixfmt->pixelformat;
>> -
>> - if (isc_try_validate_formats(isc)) {
>> - pixfmt->pixelformat = isc->try_config.fourcc = sd_fmt->fourcc;
>> - /* Re-try to validate the new format */
>> - ret = isc_try_validate_formats(isc);
>> - if (ret)
>> - goto isc_try_fmt_err;
>> - }
>> -
>> - ret = isc_try_configure_rlp_dma(isc, rlp_dma_direct_dump);
>> - if (ret)
>> - goto isc_try_fmt_err;
>> -
>> - ret = isc_try_configure_pipeline(isc);
>> - if (ret)
>> - goto isc_try_fmt_err;
>> -
>> - /* Obtain frame sizes if possible to have crop requirements ready */
>> - isc_try_fse(isc, &pad_state);
>> -
>> - v4l2_fill_mbus_format(&format.format, pixfmt, mbus_code);
>> - ret = v4l2_subdev_call(isc->current_subdev->sd, pad, set_fmt,
>> - &pad_state, &format);
>> - if (ret < 0)
>> - goto isc_try_fmt_subdev_err;
>> -
>> - v4l2_fill_pix_format(pixfmt, &format.format);
>> + v4l_bound_align_image(&pixfmt->width, 16, isc->max_width, 0,
>> + &pixfmt->height, 16, isc->max_height, 0, 0);
>> + /* If we did not find the requested format, we will fallback here */
>> + pixfmt->pixelformat = isc->try_config.fourcc;
>> + pixfmt->colorspace = V4L2_COLORSPACE_SRGB;
>> + pixfmt->field = V4L2_FIELD_NONE;
>>
>> - /* Limit to Atmel ISC hardware capabilities */
>> - if (pixfmt->width > isc->max_width)
>> - pixfmt->width = isc->max_width;
>> - if (pixfmt->height > isc->max_height)
>> - pixfmt->height = isc->max_height;
>>
>> - pixfmt->field = V4L2_FIELD_NONE;
>> pixfmt->bytesperline = (pixfmt->width * isc->try_config.bpp_v4l2) >> 3;
>> pixfmt->sizeimage = ((pixfmt->width * isc->try_config.bpp) >> 3) *
>> pixfmt->height;
>>
>> - if (code)
>> - *code = mbus_code;
>> + isc->try_fmt = *f;
>>
>> return 0;
>> +}
>>
>> -isc_try_fmt_err:
>> - v4l2_err(&isc->v4l2_dev, "Could not find any possible format for a working pipeline\n");
>> -isc_try_fmt_subdev_err:
>> - memset(&isc->try_config, 0, sizeof(isc->try_config));
>> +static int isc_set_fmt(struct isc_device *isc, struct v4l2_format *f)
>> +{
>> + isc_try_fmt(isc, f);
>>
>> - return ret;
>> + /* make the try configuration active */
>> + isc->config = isc->try_config;
>> + isc->fmt = isc->try_fmt;
>> +
>> + v4l2_dbg(1, debug, &isc->v4l2_dev, "ISC set_fmt to %.4s @%dx%d\n",
>> + (char *)&f->fmt.pix.pixelformat,
>> + f->fmt.pix.width, f->fmt.pix.height);
>> +
>> + return 0;
>> }
>>
>> -static int isc_set_fmt(struct isc_device *isc, struct v4l2_format *f)
>> +static int isc_validate(struct isc_device *isc)
>> {
>> + int ret;
>> + int i;
>> + struct isc_format *sd_fmt = NULL;
>> + struct v4l2_pix_format *pixfmt = &isc->fmt.fmt.pix;
>> struct v4l2_subdev_format format = {
>> .which = V4L2_SUBDEV_FORMAT_ACTIVE,
>> + .pad = isc->remote_pad,
>> + };
>> + struct v4l2_subdev_pad_config pad_cfg = {};
>> + struct v4l2_subdev_state pad_state = {
>> + .pads = &pad_cfg,
>> };
>> - u32 mbus_code = 0;
>> - int ret;
>>
>> - ret = isc_try_fmt(isc, f, &mbus_code);
>> + /* Get current format from subdev */
>> + ret = v4l2_subdev_call(isc->current_subdev->sd, pad, get_fmt, NULL,
>> + &format);
>> if (ret)
>> return ret;
>>
>> - v4l2_fill_mbus_format(&format.format, &f->fmt.pix, mbus_code);
>> - ret = v4l2_subdev_call(isc->current_subdev->sd, pad,
>> - set_fmt, NULL, &format);
>> - if (ret < 0)
>> - return ret;
>> + /* Identify the subdev's format configuration */
>> + for (i = 0; i < isc->formats_list_size; i++)
>> + if (isc->formats_list[i].mbus_code == format.format.code) {
>> + sd_fmt = &isc->formats_list[i];
>> + break;
>> + }
>> +
>> + /* Check if the format is not supported */
>> + if (!sd_fmt) {
>> + v4l2_err(&isc->v4l2_dev,
>> + "Current subdevice is streaming a media bus code that is not supported 0x%x\n",
>> + format.format.code);
>> + return -EPIPE;
>> + }
>> +
>> + /* At this moment we know which format the subdev will use */
>> + isc->try_config.sd_format = sd_fmt;
>> +
>> + /* If the sensor is not RAW, we can only do a direct dump */
>> + if (!ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code))
>> + isc_try_configure_rlp_dma(isc, true);
>>
>> /* Limit to Atmel ISC hardware capabilities */
>> - if (f->fmt.pix.width > isc->max_width)
>> - f->fmt.pix.width = isc->max_width;
>> - if (f->fmt.pix.height > isc->max_height)
>> - f->fmt.pix.height = isc->max_height;
>> + v4l_bound_align_image(&format.format.width, 16, isc->max_width, 0,
>> + &format.format.height, 16, isc->max_height, 0, 0);
>>
>> - isc->fmt = *f;
>> + /* Check if the frame size is the same. Otherwise we may overflow */
>> + if (pixfmt->height != format.format.height ||
>> + pixfmt->width != format.format.width) {
>> + v4l2_err(&isc->v4l2_dev,
>> + "ISC not configured with the proper frame size: %dx%d\n",
>> + format.format.width, format.format.height);
>> + return -EPIPE;
>> + }
>>
>> + v4l2_dbg(1, debug, &isc->v4l2_dev,
>> + "Identified subdev using format %.4s with %dx%d %d bpp\n",
>> + (char *)&sd_fmt->fourcc, pixfmt->width, pixfmt->height,
>> + isc->try_config.bpp);
>> +
>> + /* Reset and restart AWB if the subdevice changed the format */
>> if (isc->try_config.sd_format && isc->config.sd_format &&
>> isc->try_config.sd_format != isc->config.sd_format) {
>> isc->ctrls.hist_stat = HIST_INIT;
>> isc_reset_awb_ctrls(isc);
>> isc_update_v4l2_ctrls(isc);
>> }
>> - /* make the try configuration active */
>> +
>> + /* Validate formats */
>> + ret = isc_try_validate_formats(isc);
>> + if (ret)
>> + return ret;
>> +
>> + /* Obtain frame sizes if possible to have crop requirements ready */
>> + isc_try_fse(isc, &pad_state);
>> +
>> + /* Configure ISC pipeline for the config */
>> + ret = isc_try_configure_pipeline(isc);
>> + if (ret)
>> + return ret;
>> +
>> isc->config = isc->try_config;
>>
>> v4l2_dbg(1, debug, &isc->v4l2_dev, "New ISC configuration in place\n");
>> @@ -1043,7 +1036,7 @@ static int isc_try_fmt_vid_cap(struct file *file, void *priv,
>> {
>> struct isc_device *isc = video_drvdata(file);
>>
>> - return isc_try_fmt(isc, f, NULL);
>> + return isc_try_fmt(isc, f);
>> }
>>
>> static int isc_enum_input(struct file *file, void *priv,
>> @@ -1098,10 +1091,6 @@ static int isc_enum_framesizes(struct file *file, void *fh,
>> if (fsize->index)
>> return -EINVAL;
>>
>> - for (i = 0; i < isc->num_user_formats; i++)
>> - if (isc->user_formats[i]->fourcc == fsize->pixel_format)
>> - ret = 0;
>> -
>> for (i = 0; i < isc->controller_formats_size; i++)
>> if (isc->controller_formats[i].fourcc == fsize->pixel_format)
>> ret = 0;
>> @@ -1782,52 +1771,6 @@ struct isc_format *isc_find_format_by_code(struct isc_device *isc,
>> }
>> EXPORT_SYMBOL_GPL(isc_find_format_by_code);
>>
>> -static int isc_formats_init(struct isc_device *isc)
>> -{
>> - struct isc_format *fmt;
>> - struct v4l2_subdev *subdev = isc->current_subdev->sd;
>> - unsigned int num_fmts, i, j;
>> - u32 list_size = isc->formats_list_size;
>> - struct v4l2_subdev_mbus_code_enum mbus_code = {
>> - .which = V4L2_SUBDEV_FORMAT_ACTIVE,
>> - };
>> -
>> - num_fmts = 0;
>> - while (!v4l2_subdev_call(subdev, pad, enum_mbus_code,
>> - NULL, &mbus_code)) {
>> - mbus_code.index++;
>> -
>> - fmt = isc_find_format_by_code(isc, mbus_code.code, &i);
>> - if (!fmt) {
>> - v4l2_warn(&isc->v4l2_dev, "Mbus code %x not supported\n",
>> - mbus_code.code);
>> - continue;
>> - }
>> -
>> - fmt->sd_support = true;
>> - num_fmts++;
>> - }
>> -
>> - if (!num_fmts)
>> - return -ENXIO;
>> -
>> - isc->num_user_formats = num_fmts;
>> - isc->user_formats = devm_kcalloc(isc->dev,
>> - num_fmts, sizeof(*isc->user_formats),
>> - GFP_KERNEL);
>> - if (!isc->user_formats)
>> - return -ENOMEM;
>> -
>> - fmt = &isc->formats_list[0];
>> - for (i = 0, j = 0; i < list_size; i++) {
>> - if (fmt->sd_support)
>> - isc->user_formats[j++] = fmt;
>> - fmt++;
>> - }
>> -
>> - return 0;
>> -}
>> -
>> static int isc_set_default_fmt(struct isc_device *isc)
>> {
>> struct v4l2_format f = {
>> @@ -1836,12 +1779,12 @@ static int isc_set_default_fmt(struct isc_device *isc)
>> .width = VGA_WIDTH,
>> .height = VGA_HEIGHT,
>> .field = V4L2_FIELD_NONE,
>> - .pixelformat = isc->user_formats[0]->fourcc,
>> + .pixelformat = isc->controller_formats[0].fourcc,
>> },
>> };
>> int ret;
>>
>> - ret = isc_try_fmt(isc, &f, NULL);
>> + ret = isc_try_fmt(isc, &f);
>> if (ret)
>> return ret;
>>
>> @@ -1896,13 +1839,6 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier)
>> spin_lock_init(&isc->dma_queue_lock);
>> spin_lock_init(&isc->awb_lock);
>>
>> - ret = isc_formats_init(isc);
>> - if (ret < 0) {
>> - v4l2_err(&isc->v4l2_dev,
>> - "Init format failed: %d\n", ret);
>> - goto isc_async_complete_err;
>> - }
>> -
>> ret = isc_set_default_fmt(isc);
>> if (ret) {
>> v4l2_err(&isc->v4l2_dev, "Could not set default format\n");
>> @@ -2016,6 +1952,24 @@ int isc_pipeline_init(struct isc_device *isc)
>> }
>> EXPORT_SYMBOL_GPL(isc_pipeline_init);
>>
>> +static int isc_link_validate(struct media_link *link)
>> +{
>> + struct video_device *vdev =
>> + media_entity_to_video_device(link->sink->entity);
>> + struct isc_device *isc = video_get_drvdata(vdev);
>> + int ret;
>> +
>> + ret = v4l2_subdev_link_validate(link);
>> + if (ret)
>> + return ret;
>> +
>> + return isc_validate(isc);
>> +}
>> +
>> +static const struct media_entity_operations isc_entity_operations = {
>> + .link_validate = isc_link_validate,
>> +};
>> +
>> int isc_mc_init(struct isc_device *isc, u32 ver)
>> {
>> const struct of_device_id *match;
>> @@ -2023,6 +1977,8 @@ int isc_mc_init(struct isc_device *isc, u32 ver)
>>
>> isc->video_dev.entity.function = MEDIA_ENT_F_IO_V4L;
>> isc->video_dev.entity.flags = MEDIA_ENT_FL_DEFAULT;
>> + isc->video_dev.entity.ops = &isc_entity_operations;
>> +
>> isc->pads[ISC_PAD_SINK].flags = MEDIA_PAD_FL_SINK;
>>
>> ret = media_entity_pads_init(&isc->video_dev.entity, ISC_PADS_NUM,
>> diff --git a/drivers/media/platform/atmel/atmel-isc-scaler.c b/drivers/media/platform/atmel/atmel-isc-scaler.c
>> index a1ca4633787c..2b0621297dfe 100644
>> --- a/drivers/media/platform/atmel/atmel-isc-scaler.c
>> +++ b/drivers/media/platform/atmel/atmel-isc-scaler.c
>> @@ -173,6 +173,10 @@ static const struct v4l2_subdev_pad_ops isc_scaler_pad_ops = {
>> .init_cfg = isc_scaler_init_cfg,
>> };
>>
>> +static const struct media_entity_operations isc_scaler_entity_ops = {
>> + .link_validate = v4l2_subdev_link_validate,
>> +};
>> +
>> static const struct v4l2_subdev_ops xisc_scaler_subdev_ops = {
>> .pad = &isc_scaler_pad_ops,
>> };
>> @@ -190,6 +194,7 @@ int isc_scaler_init(struct isc_device *isc)
>>
>> isc->scaler_sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
>> isc->scaler_sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
>> + isc->scaler_sd.entity.ops = &isc_scaler_entity_ops;
>> isc->scaler_pads[ISC_SCALER_PAD_SINK].flags = MEDIA_PAD_FL_SINK;
>> isc->scaler_pads[ISC_SCALER_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
>>
>> diff --git a/drivers/media/platform/atmel/atmel-isc.h b/drivers/media/platform/atmel/atmel-isc.h
>> index f98f25a55e73..5b0100c8c245 100644
>> --- a/drivers/media/platform/atmel/atmel-isc.h
>> +++ b/drivers/media/platform/atmel/atmel-isc.h
>> @@ -63,15 +63,16 @@ struct isc_subdev_entity {
>> * @cfa_baycfg: If this format is RAW BAYER, indicate the type of bayer.
>> this is either BGBG, RGRG, etc.
>> * @pfe_cfg0_bps: Number of hardware data lines connected to the ISC
>> + * @raw: If the format is raw bayer.
>> */
>>
>> struct isc_format {
>> u32 fourcc;
>> u32 mbus_code;
>> u32 cfa_baycfg;
>> -
>> - bool sd_support;
>> u32 pfe_cfg0_bps;
>> +
>> + bool raw;
>> };
>>
>> /* Pipeline bitmap */
>> @@ -216,8 +217,7 @@ enum isc_scaler_pads {
>> * @comp: completion reference that signals frame completion
>> *
>> * @fmt: current v42l format
>> - * @user_formats: list of formats that are supported and agreed with sd
>> - * @num_user_formats: how many formats are in user_formats
>> + * @try_fmt: current v4l2 try format
>> *
>> * @config: current ISC format configuration
>> * @try_config: the current ISC try format , not yet activated
>> @@ -272,6 +272,7 @@ enum isc_scaler_pads {
>> * @formats_list_size: size of formats_list array
>> * @pads: media controller pads for isc video entity
>> * @mdev: media device that is registered by the isc
>> + * @mpipe: media device pipeline used by the isc
>> * @remote_pad: remote pad on the connected subdevice
>> * @scaler_sd: subdevice for the scaler that isc registers
>> * @scaler_pads: media controller pads for the scaler subdevice
>> @@ -298,8 +299,7 @@ struct isc_device {
>> struct completion comp;
>>
>> struct v4l2_format fmt;
>> - struct isc_format **user_formats;
>> - unsigned int num_user_formats;
>> + struct v4l2_format try_fmt;
>>
>> struct fmt_config config;
>> struct fmt_config try_config;
>> @@ -369,6 +369,7 @@ struct isc_device {
>> struct {
>> struct media_pad pads[ISC_PADS_NUM];
>> struct media_device mdev;
>> + struct media_pipeline mpipe;
>>
>> u32 remote_pad;
>> };
>> diff --git a/drivers/media/platform/atmel/atmel-sama5d2-isc.c b/drivers/media/platform/atmel/atmel-sama5d2-isc.c
>> index d96ee3373889..2a651e237193 100644
>> --- a/drivers/media/platform/atmel/atmel-sama5d2-isc.c
>> +++ b/drivers/media/platform/atmel/atmel-sama5d2-isc.c
>> @@ -80,20 +80,40 @@ static const struct isc_format sama5d2_controller_formats[] = {
>> .fourcc = V4L2_PIX_FMT_Y10,
>> }, {
>> .fourcc = V4L2_PIX_FMT_SBGGR8,
>> + .raw = true,
>> }, {
>> .fourcc = V4L2_PIX_FMT_SGBRG8,
>> + .raw = true,
>> }, {
>> .fourcc = V4L2_PIX_FMT_SGRBG8,
>> + .raw = true,
>> }, {
>> .fourcc = V4L2_PIX_FMT_SRGGB8,
>> + .raw = true,
>> }, {
>> .fourcc = V4L2_PIX_FMT_SBGGR10,
>> + .raw = true,
>> }, {
>> .fourcc = V4L2_PIX_FMT_SGBRG10,
>> + .raw = true,
>> }, {
>> .fourcc = V4L2_PIX_FMT_SGRBG10,
>> + .raw = true,
>> }, {
>> .fourcc = V4L2_PIX_FMT_SRGGB10,
>> + .raw = true,
>> + }, {
>> + .fourcc = V4L2_PIX_FMT_SBGGR12,
>> + .raw = true,
>> + }, {
>> + .fourcc = V4L2_PIX_FMT_SGBRG12,
>> + .raw = true,
>> + }, {
>> + .fourcc = V4L2_PIX_FMT_SGRBG12,
>> + .raw = true,
>> + }, {
>> + .fourcc = V4L2_PIX_FMT_SRGGB12,
>> + .raw = true,
>> },
>> };
>>
>> diff --git a/drivers/media/platform/atmel/atmel-sama7g5-isc.c b/drivers/media/platform/atmel/atmel-sama7g5-isc.c
>> index e07ae188c15f..a0d60cfdba7b 100644
>> --- a/drivers/media/platform/atmel/atmel-sama7g5-isc.c
>> +++ b/drivers/media/platform/atmel/atmel-sama7g5-isc.c
>> @@ -89,20 +89,40 @@ static const struct isc_format sama7g5_controller_formats[] = {
>> .fourcc = V4L2_PIX_FMT_Y16,
>> }, {
>> .fourcc = V4L2_PIX_FMT_SBGGR8,
>> + .raw = true,
>> }, {
>> .fourcc = V4L2_PIX_FMT_SGBRG8,
>> + .raw = true,
>> }, {
>> .fourcc = V4L2_PIX_FMT_SGRBG8,
>> + .raw = true,
>> }, {
>> .fourcc = V4L2_PIX_FMT_SRGGB8,
>> + .raw = true,
>> }, {
>> .fourcc = V4L2_PIX_FMT_SBGGR10,
>> + .raw = true,
>> }, {
>> .fourcc = V4L2_PIX_FMT_SGBRG10,
>> + .raw = true,
>> }, {
>> .fourcc = V4L2_PIX_FMT_SGRBG10,
>> + .raw = true,
>> }, {
>> .fourcc = V4L2_PIX_FMT_SRGGB10,
>> + .raw = true,
>> + }, {
>> + .fourcc = V4L2_PIX_FMT_SBGGR12,
>> + .raw = true,
>> + }, {
>> + .fourcc = V4L2_PIX_FMT_SGBRG12,
>> + .raw = true,
>> + }, {
>> + .fourcc = V4L2_PIX_FMT_SGRBG12,
>> + .raw = true,
>> + }, {
>> + .fourcc = V4L2_PIX_FMT_SRGGB12,
>> + .raw = true,
>> },
>> };
>>
>

2022-04-29 21:02:50

[permalink] [raw]

Subject: Re: [PATCH v9 08/13] media: atmel: atmel-isc: change format propagation to subdev into only verification

On 29/04/2022 11:58, Jacopo Mondi wrote:
> Hello
>
> On Fri, Apr 29, 2022 at 10:43:09AM +0200, Hans Verkuil wrote:
>> On 29/04/2022 10:28, [email protected] wrote:
>>> On 4/29/22 11:17 AM, Hans Verkuil wrote:
>>>> On 10/03/2022 10:51, Eugen Hristev wrote:
>>>>> As a top MC video driver, the atmel-isc should not propagate the format to the
>>>>> subdevice, it should rather check at start_streaming() time if the subdev is properly
>>>>> configured with a compatible format.
>>>>> Removed the whole format finding logic, and reworked the format verification
>>>>> at start_streaming time, such that the ISC will return an error if the subdevice
>>>>> is not properly configured. To achieve this, media_pipeline_start
>>>>> is called and a link_validate callback is created to check the formats.
>>>>> With this being done, the module parameter 'sensor_preferred' makes no sense
>>>>> anymore. The ISC should not decide which format the sensor is using. The
>>>>> ISC should only cope with the situation and inform userspace if the streaming
>>>>> is possible in the current configuration.
>>>>> The redesign of the format propagation has also risen the question of the
>>>>> enumfmt callback. If enumfmt is called with an mbus_code, the enumfmt handler
>>>>> should only return the formats that are supported for this mbus_code.
>>>>> Otherwise, the enumfmt will report all the formats that the ISC could output.
>>>>> With this rework, the dynamic list of user formats is removed. It makes no
>>>>> more sense to identify at complete time which formats the sensor could emit,
>>>>> and add those into a separate dynamic list.
>>>>> The ISC will start with a simple preconfigured default format, and at
>>>>> link validate time, decide whether it can use the format that is configured
>>>>> on the sink or not.
>>>>>
>>>>> Signed-off-by: Eugen Hristev <[email protected]>
>>>>> Reviewed-by: Jacopo Mondi <[email protected]>
>>>>> ---
>>>>> Changes in v9:
>>>>> - isc_link_validate now static
>>>>>
>>>>> Changes in v7:
>>>>> - minor typos as suggested by Jacopo
>>>>> - small changes, reduce some indentation, modified an index, as suggested by
>>>>> Jacopo
>>>>>
>>>>> Changes in v6:
>>>>> - reworked a bit enum_fmt as suggested by Jacopo
>>>>>
>>>>> Changes in v5:
>>>>> - removed user_formats dynamic list as it is now pointless
>>>>> - greatly simplified the enum_fmt function
>>>>> - removed some init code that was useless now
>>>>>
>>>>> Changes in v4:
>>>>> - moved validation code into link_validate and used media_pipeline_start
>>>>> - merged this patch with the enum_fmt patch which was previously in v3 of
>>>>> the series
>>>>>
>>>>> Changes in v3:
>>>>> - clamp to maximum resolution once the frame size from the subdev is found
>>>>> drivers/media/platform/atmel/atmel-isc-base.c | 412 ++++++++----------
>>>>> .../media/platform/atmel/atmel-isc-scaler.c | 5 +
>>>>> drivers/media/platform/atmel/atmel-isc.h | 13 +-
>>>>> .../media/platform/atmel/atmel-sama5d2-isc.c | 20 +
>>>>> .../media/platform/atmel/atmel-sama7g5-isc.c | 20 +
>>>>> 5 files changed, 236 insertions(+), 234 deletions(-)
>>>>>
>>>>> diff --git a/drivers/media/platform/atmel/atmel-isc-base.c b/drivers/media/platform/atmel/atmel-isc-base.c
>>>>> index ee1dda6707a0..fe2c0af58060 100644
>>>>> --- a/drivers/media/platform/atmel/atmel-isc-base.c
>>>>> +++ b/drivers/media/platform/atmel/atmel-isc-base.c
>>>>> @@ -36,11 +36,6 @@ static unsigned int debug;
>>>>> module_param(debug, int, 0644);
>>>>> MODULE_PARM_DESC(debug, "debug level (0-2)");
>>>>>
>>>>> -static unsigned int sensor_preferred = 1;
>>>>> -module_param(sensor_preferred, uint, 0644);
>>>>> -MODULE_PARM_DESC(sensor_preferred,
>>>>> - "Sensor is preferred to output the specified format (1-on 0-off), default 1");
>>>>> -
>>>>> #define ISC_IS_FORMAT_RAW(mbus_code) \
>>>>> (((mbus_code) & 0xf000) == 0x3000)
>>>>>
>>>>> @@ -337,6 +332,10 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)
>>>>> unsigned long flags;
>>>>> int ret;
>>>>>
>>>>> + ret = media_pipeline_start(&isc->video_dev.entity, &isc->mpipe);
>>>>
>>>> The pipeline validation is done in start_streaming, but I don't think that
>>>> is the best place: if STREAMON is called before buffers are queued, then
>>>> an invalid pipeline isn't discovered until enough buffers are queued to
>>>> kick off start_streaming.
>>>>
>>>> Drivers like vsp1, omap3isp and the samsung drivers all do this in streamon().
>>>>
>>>> I think that is the correct time to do this.
>>>
>>> Hello Hans,
>>>
>>> Initially (v2, v3) I had this in streamon(). The problem that I faced at
>>> that time was that streamoff was never called, so I could not call
>>> media_pipeline_stop(). Then Jacopo told me to move it to start_streaming
>>> (see change log for v4) , and I did not face any more problems.
>
> Yes indeed, seems I suggested to use media_pipeline_handler in a
> comment on your v3
>
> "at s_stream time your top driver calls media_pipeline_start()"
>
> sorry about that, I should have looked around a bit more carefully and
> notice most drivers do so at vb2 streamon
>
> However I don't see media_pipeline_start being called at all in v3 of
> the patch
>
>>
>> It's a mess. Looking at some drivers I see that omap3isp calls media_pipeline_stop
>> in streamoff (so will have the same problem as you described if VIDIOC_STREAMOFF
>> isn't called), exynos4-is does the same, but it also checks the streaming state in
>> the release() fop callback, so that would fix this problem. And vimc does this
>> in stop_streaming.
>>
>> I'm in favor of fixing this in vb2, that framework knows exactly when this needs
>> to be called.
>
> Are you suggesting to have vb2 to call media_pipeline_start() or is it
> more complex than this ?

No, I don't think vb2 should do that. But it needs two new ops: pipeline_start/stop
(or some other name) that are called at VIDIOC_STREAMON time and when the streaming
stops (due to STREAMOFF or by closing the fh). The driver can then take appropriate
action, which will typically be calling media_pipeline_start/stop, but other checks
may also be needed at that time.

Regards,

Hans

>
>>
>> I'd like to know what others think.
>>
>> Regards,
>>
>> Hans
>>
>>>
>>> Eugen
>>>
>>>>
>>>> Laurent, Jacopo, do you agree with that?
>>>>
>>>> I notice that quite a few other drivers do this in start_streaming as well,
>>>> but I think it just weird that a QBUF can return EPIPE while STREAMON might
>>>> succeed.
>>>>
>>>> Perhaps we need an optional vb2 validate() callback that is called by
>>>> vb2_core_streamon() to validate the pipeline before streaming. That makes
>>>> it easier to implement these checks rather than having to implement your
>>>> own streamon ioctl.
>>>>
>>>> Regards,
>>>>
>>>> Hans
>>>>
>>>>> + if (ret)
>>>>> + goto err_pipeline_start;
>>>>> +
>>>>> /* Enable stream on the sub device */
>>>>> ret = v4l2_subdev_call(isc->current_subdev->sd, video, s_stream, 1);
>>>>> if (ret && ret != -ENOIOCTLCMD) {
>>>>> @@ -385,6 +384,9 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)
>>>>> v4l2_subdev_call(isc->current_subdev->sd, video, s_stream, 0);
>>>>>
>>>>> err_start_stream:
>>>>> + media_pipeline_stop(&isc->video_dev.entity);
>>>>> +
>>>>> +err_pipeline_start:
>>>>> spin_lock_irqsave(&isc->dma_queue_lock, flags);
>>>>> list_for_each_entry(buf, &isc->dma_queue, list)
>>>>> vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_QUEUED);
>>>>> @@ -423,6 +425,9 @@ static void isc_stop_streaming(struct vb2_queue *vq)
>>>>> if (ret && ret != -ENOIOCTLCMD)
>>>>> v4l2_err(&isc->v4l2_dev, "stream off failed in subdev\n");
>>>>>
>>>>> + /* Stop media pipeline */
>>>>> + media_pipeline_stop(&isc->video_dev.entity);
>>>>> +
>>>>> /* Release all active buffers */
>>>>> spin_lock_irqsave(&isc->dma_queue_lock, flags);
>>>>> if (unlikely(isc->cur_frm)) {
>>>>> @@ -453,22 +458,6 @@ static void isc_buffer_queue(struct vb2_buffer *vb)
>>>>> spin_unlock_irqrestore(&isc->dma_queue_lock, flags);
>>>>> }
>>>>>
>>>>> -static struct isc_format *find_format_by_fourcc(struct isc_device *isc,
>>>>> - unsigned int fourcc)
>>>>> -{
>>>>> - unsigned int num_formats = isc->num_user_formats;
>>>>> - struct isc_format *fmt;
>>>>> - unsigned int i;
>>>>> -
>>>>> - for (i = 0; i < num_formats; i++) {
>>>>> - fmt = isc->user_formats[i];
>>>>> - if (fmt->fourcc == fourcc)
>>>>> - return fmt;
>>>>> - }
>>>>> -
>>>>> - return NULL;
>>>>> -}
>>>>> -
>>>>> static const struct vb2_ops isc_vb2_ops = {
>>>>> .queue_setup = isc_queue_setup,
>>>>> .wait_prepare = vb2_ops_wait_prepare,
>>>>> @@ -497,23 +486,57 @@ static int isc_enum_fmt_vid_cap(struct file *file, void *priv,
>>>>> {
>>>>> struct isc_device *isc = video_drvdata(file);
>>>>> u32 index = f->index;
>>>>> - u32 i, supported_index;
>>>>> + u32 i, supported_index = 0;
>>>>> + struct isc_format *fmt;
>>>>> +
>>>>> + /*
>>>>> + * If we are not asked a specific mbus_code, we have to report all
>>>>> + * the formats that we can output.
>>>>> + */
>>>>> + if (!f->mbus_code) {
>>>>> + if (index >= isc->controller_formats_size)
>>>>> + return -EINVAL;
>>>>>
>>>>> - if (index < isc->controller_formats_size) {
>>>>> f->pixelformat = isc->controller_formats[index].fourcc;
>>>>> +
>>>>> + return 0;
>>>>> + }
>>>>> +
>>>>> + /*
>>>>> + * If a specific mbus_code is requested, check if we support
>>>>> + * this mbus_code as input for the ISC.
>>>>> + * If it's supported, then we report the corresponding pixelformat
>>>>> + * as first possible option for the ISC.
>>>>> + * E.g. mbus MEDIA_BUS_FMT_YUYV8_2X8 and report
>>>>> + * 'YUYV' (YUYV 4:2:2)
>>>>> + */
>>>>> + fmt = isc_find_format_by_code(isc, f->mbus_code, &i);
>>>>> + if (!fmt)
>>>>> + return -EINVAL;
>>>>> +
>>>>> + if (!index) {
>>>>> + f->pixelformat = fmt->fourcc;
>>>>> +
>>>>> return 0;
>>>>> }
>>>>>
>>>>> - index -= isc->controller_formats_size;
>>>>> + supported_index++;
>>>>>
>>>>> - supported_index = 0;
>>>>> + /* If the index is not raw, we don't have anymore formats to report */
>>>>> + if (!ISC_IS_FORMAT_RAW(f->mbus_code))
>>>>> + return -EINVAL;
>>>>>
>>>>> - for (i = 0; i < isc->formats_list_size; i++) {
>>>>> - if (!ISC_IS_FORMAT_RAW(isc->formats_list[i].mbus_code) ||
>>>>> - !isc->formats_list[i].sd_support)
>>>>> + /*
>>>>> + * We are asked for a specific mbus code, which is raw.
>>>>> + * We have to search through the formats we can convert to.
>>>>> + * We have to skip the raw formats, we cannot convert to raw.
>>>>> + * E.g. 'AR12' (16-bit ARGB 4-4-4-4), 'AR15' (16-bit ARGB 1-5-5-5), etc.
>>>>> + */
>>>>> + for (i = 0; i < isc->controller_formats_size; i++) {
>>>>> + if (isc->controller_formats[i].raw)
>>>>> continue;
>>>>> - if (supported_index == index) {
>>>>> - f->pixelformat = isc->formats_list[i].fourcc;
>>>>> + if (index == supported_index) {
>>>>> + f->pixelformat = isc->controller_formats[i].fourcc;
>>>>> return 0;
>>>>> }
>>>>> supported_index++;
>>>>> @@ -584,20 +607,30 @@ static int isc_try_validate_formats(struct isc_device *isc)
>>>>> break;
>>>>> default:
>>>>> /* any other different formats are not supported */
>>>>> + v4l2_err(&isc->v4l2_dev, "Requested unsupported format.\n");
>>>>> ret = -EINVAL;
>>>>> }
>>>>> v4l2_dbg(1, debug, &isc->v4l2_dev,
>>>>> "Format validation, requested rgb=%u, yuv=%u, grey=%u, bayer=%u\n",
>>>>> rgb, yuv, grey, bayer);
>>>>>
>>>>> - /* we cannot output RAW if we do not receive RAW */
>>>>> - if ((bayer) && !ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code))
>>>>> + if (bayer &&
>>>>> + !ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code)) {
>>>>> + v4l2_err(&isc->v4l2_dev, "Cannot output RAW if we do not receive RAW.\n");
>>>>> return -EINVAL;
>>>>> + }
>>>>>
>>>>> - /* we cannot output GREY if we do not receive RAW/GREY */
>>>>> if (grey && !ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code) &&
>>>>> - !ISC_IS_FORMAT_GREY(isc->try_config.sd_format->mbus_code))
>>>>> + !ISC_IS_FORMAT_GREY(isc->try_config.sd_format->mbus_code)) {
>>>>> + v4l2_err(&isc->v4l2_dev, "Cannot output GREY if we do not receive RAW/GREY.\n");
>>>>> return -EINVAL;
>>>>> + }
>>>>> +
>>>>> + if ((rgb || bayer || yuv) &&
>>>>> + ISC_IS_FORMAT_GREY(isc->try_config.sd_format->mbus_code)) {
>>>>> + v4l2_err(&isc->v4l2_dev, "Cannot convert GREY to another format.\n");
>>>>> + return -EINVAL;
>>>>> + }
>>>>>
>>>>> return ret;
>>>>> }
>>>>> @@ -825,7 +858,7 @@ static void isc_try_fse(struct isc_device *isc,
>>>>> * If we do not know yet which format the subdev is using, we cannot
>>>>> * do anything.
>>>>> */
>>>>> - if (!isc->try_config.sd_format)
>>>>> + if (!isc->config.sd_format)
>>>>> return;
>>>>>
>>>>> fse.code = isc->try_config.sd_format->mbus_code;
>>>>> @@ -846,180 +879,140 @@ static void isc_try_fse(struct isc_device *isc,
>>>>> }
>>>>> }
>>>>>
>>>>> -static int isc_try_fmt(struct isc_device *isc, struct v4l2_format *f,
>>>>> - u32 *code)
>>>>> +static int isc_try_fmt(struct isc_device *isc, struct v4l2_format *f)
>>>>> {
>>>>> - int i;
>>>>> - struct isc_format *sd_fmt = NULL, *direct_fmt = NULL;
>>>>> struct v4l2_pix_format *pixfmt = &f->fmt.pix;
>>>>> - struct v4l2_subdev_pad_config pad_cfg = {};
>>>>> - struct v4l2_subdev_state pad_state = {
>>>>> - .pads = &pad_cfg
>>>>> - };
>>>>> - struct v4l2_subdev_format format = {
>>>>> - .which = V4L2_SUBDEV_FORMAT_TRY,
>>>>> - };
>>>>> - u32 mbus_code;
>>>>> - int ret;
>>>>> - bool rlp_dma_direct_dump = false;
>>>>> + unsigned int i;
>>>>>
>>>>> if (f->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
>>>>> return -EINVAL;
>>>>>
>>>>> - /* Step 1: find a RAW format that is supported */
>>>>> - for (i = 0; i < isc->num_user_formats; i++) {
>>>>> - if (ISC_IS_FORMAT_RAW(isc->user_formats[i]->mbus_code)) {
>>>>> - sd_fmt = isc->user_formats[i];
>>>>> + isc->try_config.fourcc = isc->controller_formats[0].fourcc;
>>>>> +
>>>>> + /* find if the format requested is supported */
>>>>> + for (i = 0; i < isc->controller_formats_size; i++)
>>>>> + if (isc->controller_formats[i].fourcc == pixfmt->pixelformat) {
>>>>> + isc->try_config.fourcc = pixfmt->pixelformat;
>>>>> break;
>>>>> }
>>>>> - }
>>>>> - /* Step 2: We can continue with this RAW format, or we can look
>>>>> - * for better: maybe sensor supports directly what we need.
>>>>> - */
>>>>> - direct_fmt = find_format_by_fourcc(isc, pixfmt->pixelformat);
>>>>> -
>>>>> - /* Step 3: We have both. We decide given the module parameter which
>>>>> - * one to use.
>>>>> - */
>>>>> - if (direct_fmt && sd_fmt && sensor_preferred)
>>>>> - sd_fmt = direct_fmt;
>>>>> -
>>>>> - /* Step 4: we do not have RAW but we have a direct format. Use it. */
>>>>> - if (direct_fmt && !sd_fmt)
>>>>> - sd_fmt = direct_fmt;
>>>>> -
>>>>> - /* Step 5: if we are using a direct format, we need to package
>>>>> - * everything as 8 bit data and just dump it
>>>>> - */
>>>>> - if (sd_fmt == direct_fmt)
>>>>> - rlp_dma_direct_dump = true;
>>>>> -
>>>>> - /* Step 6: We have no format. This can happen if the userspace
>>>>> - * requests some weird/invalid format.
>>>>> - * In this case, default to whatever we have
>>>>> - */
>>>>> - if (!sd_fmt && !direct_fmt) {
>>>>> - sd_fmt = isc->user_formats[isc->num_user_formats - 1];
>>>>> - v4l2_dbg(1, debug, &isc->v4l2_dev,
>>>>> - "Sensor not supporting %.4s, using %.4s\n",
>>>>> - (char *)&pixfmt->pixelformat, (char *)&sd_fmt->fourcc);
>>>>> - }
>>>>> -
>>>>> - if (!sd_fmt) {
>>>>> - ret = -EINVAL;
>>>>> - goto isc_try_fmt_err;
>>>>> - }
>>>>> -
>>>>> - /* Step 7: Print out what we decided for debugging */
>>>>> - v4l2_dbg(1, debug, &isc->v4l2_dev,
>>>>> - "Preferring to have sensor using format %.4s\n",
>>>>> - (char *)&sd_fmt->fourcc);
>>>>>
>>>>> - /* Step 8: at this moment we decided which format the subdev will use */
>>>>> - isc->try_config.sd_format = sd_fmt;
>>>>> + isc_try_configure_rlp_dma(isc, false);
>>>>>
>>>>> /* Limit to Atmel ISC hardware capabilities */
>>>>> - if (pixfmt->width > isc->max_width)
>>>>> - pixfmt->width = isc->max_width;
>>>>> - if (pixfmt->height > isc->max_height)
>>>>> - pixfmt->height = isc->max_height;
>>>>> -
>>>>> - /*
>>>>> - * The mbus format is the one the subdev outputs.
>>>>> - * The pixels will be transferred in this format Sensor -> ISC
>>>>> - */
>>>>> - mbus_code = sd_fmt->mbus_code;
>>>>> -
>>>>> - /*
>>>>> - * Validate formats. If the required format is not OK, default to raw.
>>>>> - */
>>>>> -
>>>>> - isc->try_config.fourcc = pixfmt->pixelformat;
>>>>> -
>>>>> - if (isc_try_validate_formats(isc)) {
>>>>> - pixfmt->pixelformat = isc->try_config.fourcc = sd_fmt->fourcc;
>>>>> - /* Re-try to validate the new format */
>>>>> - ret = isc_try_validate_formats(isc);
>>>>> - if (ret)
>>>>> - goto isc_try_fmt_err;
>>>>> - }
>>>>> -
>>>>> - ret = isc_try_configure_rlp_dma(isc, rlp_dma_direct_dump);
>>>>> - if (ret)
>>>>> - goto isc_try_fmt_err;
>>>>> -
>>>>> - ret = isc_try_configure_pipeline(isc);
>>>>> - if (ret)
>>>>> - goto isc_try_fmt_err;
>>>>> -
>>>>> - /* Obtain frame sizes if possible to have crop requirements ready */
>>>>> - isc_try_fse(isc, &pad_state);
>>>>> -
>>>>> - v4l2_fill_mbus_format(&format.format, pixfmt, mbus_code);
>>>>> - ret = v4l2_subdev_call(isc->current_subdev->sd, pad, set_fmt,
>>>>> - &pad_state, &format);
>>>>> - if (ret < 0)
>>>>> - goto isc_try_fmt_subdev_err;
>>>>> -
>>>>> - v4l2_fill_pix_format(pixfmt, &format.format);
>>>>> + v4l_bound_align_image(&pixfmt->width, 16, isc->max_width, 0,
>>>>> + &pixfmt->height, 16, isc->max_height, 0, 0);
>>>>> + /* If we did not find the requested format, we will fallback here */
>>>>> + pixfmt->pixelformat = isc->try_config.fourcc;
>>>>> + pixfmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>> + pixfmt->field = V4L2_FIELD_NONE;
>>>>>
>>>>> - /* Limit to Atmel ISC hardware capabilities */
>>>>> - if (pixfmt->width > isc->max_width)
>>>>> - pixfmt->width = isc->max_width;
>>>>> - if (pixfmt->height > isc->max_height)
>>>>> - pixfmt->height = isc->max_height;
>>>>>
>>>>> - pixfmt->field = V4L2_FIELD_NONE;
>>>>> pixfmt->bytesperline = (pixfmt->width * isc->try_config.bpp_v4l2) >> 3;
>>>>> pixfmt->sizeimage = ((pixfmt->width * isc->try_config.bpp) >> 3) *
>>>>> pixfmt->height;
>>>>>
>>>>> - if (code)
>>>>> - *code = mbus_code;
>>>>> + isc->try_fmt = *f;
>>>>>
>>>>> return 0;
>>>>> +}
>>>>>
>>>>> -isc_try_fmt_err:
>>>>> - v4l2_err(&isc->v4l2_dev, "Could not find any possible format for a working pipeline\n");
>>>>> -isc_try_fmt_subdev_err:
>>>>> - memset(&isc->try_config, 0, sizeof(isc->try_config));
>>>>> +static int isc_set_fmt(struct isc_device *isc, struct v4l2_format *f)
>>>>> +{
>>>>> + isc_try_fmt(isc, f);
>>>>>
>>>>> - return ret;
>>>>> + /* make the try configuration active */
>>>>> + isc->config = isc->try_config;
>>>>> + isc->fmt = isc->try_fmt;
>>>>> +
>>>>> + v4l2_dbg(1, debug, &isc->v4l2_dev, "ISC set_fmt to %.4s @%dx%d\n",
>>>>> + (char *)&f->fmt.pix.pixelformat,
>>>>> + f->fmt.pix.width, f->fmt.pix.height);
>>>>> +
>>>>> + return 0;
>>>>> }
>>>>>
>>>>> -static int isc_set_fmt(struct isc_device *isc, struct v4l2_format *f)
>>>>> +static int isc_validate(struct isc_device *isc)
>>>>> {
>>>>> + int ret;
>>>>> + int i;
>>>>> + struct isc_format *sd_fmt = NULL;
>>>>> + struct v4l2_pix_format *pixfmt = &isc->fmt.fmt.pix;
>>>>> struct v4l2_subdev_format format = {
>>>>> .which = V4L2_SUBDEV_FORMAT_ACTIVE,
>>>>> + .pad = isc->remote_pad,
>>>>> + };
>>>>> + struct v4l2_subdev_pad_config pad_cfg = {};
>>>>> + struct v4l2_subdev_state pad_state = {
>>>>> + .pads = &pad_cfg,
>>>>> };
>>>>> - u32 mbus_code = 0;
>>>>> - int ret;
>>>>>
>>>>> - ret = isc_try_fmt(isc, f, &mbus_code);
>>>>> + /* Get current format from subdev */
>>>>> + ret = v4l2_subdev_call(isc->current_subdev->sd, pad, get_fmt, NULL,
>>>>> + &format);
>>>>> if (ret)
>>>>> return ret;
>>>>>
>>>>> - v4l2_fill_mbus_format(&format.format, &f->fmt.pix, mbus_code);
>>>>> - ret = v4l2_subdev_call(isc->current_subdev->sd, pad,
>>>>> - set_fmt, NULL, &format);
>>>>> - if (ret < 0)
>>>>> - return ret;
>>>>> + /* Identify the subdev's format configuration */
>>>>> + for (i = 0; i < isc->formats_list_size; i++)
>>>>> + if (isc->formats_list[i].mbus_code == format.format.code) {
>>>>> + sd_fmt = &isc->formats_list[i];
>>>>> + break;
>>>>> + }
>>>>> +
>>>>> + /* Check if the format is not supported */
>>>>> + if (!sd_fmt) {
>>>>> + v4l2_err(&isc->v4l2_dev,
>>>>> + "Current subdevice is streaming a media bus code that is not supported 0x%x\n",
>>>>> + format.format.code);
>>>>> + return -EPIPE;
>>>>> + }
>>>>> +
>>>>> + /* At this moment we know which format the subdev will use */
>>>>> + isc->try_config.sd_format = sd_fmt;
>>>>> +
>>>>> + /* If the sensor is not RAW, we can only do a direct dump */
>>>>> + if (!ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code))
>>>>> + isc_try_configure_rlp_dma(isc, true);
>>>>>
>>>>> /* Limit to Atmel ISC hardware capabilities */
>>>>> - if (f->fmt.pix.width > isc->max_width)
>>>>> - f->fmt.pix.width = isc->max_width;
>>>>> - if (f->fmt.pix.height > isc->max_height)
>>>>> - f->fmt.pix.height = isc->max_height;
>>>>> + v4l_bound_align_image(&format.format.width, 16, isc->max_width, 0,
>>>>> + &format.format.height, 16, isc->max_height, 0, 0);
>>>>>
>>>>> - isc->fmt = *f;
>>>>> + /* Check if the frame size is the same. Otherwise we may overflow */
>>>>> + if (pixfmt->height != format.format.height ||
>>>>> + pixfmt->width != format.format.width) {
>>>>> + v4l2_err(&isc->v4l2_dev,
>>>>> + "ISC not configured with the proper frame size: %dx%d\n",
>>>>> + format.format.width, format.format.height);
>>>>> + return -EPIPE;
>>>>> + }
>>>>>
>>>>> + v4l2_dbg(1, debug, &isc->v4l2_dev,
>>>>> + "Identified subdev using format %.4s with %dx%d %d bpp\n",
>>>>> + (char *)&sd_fmt->fourcc, pixfmt->width, pixfmt->height,
>>>>> + isc->try_config.bpp);
>>>>> +
>>>>> + /* Reset and restart AWB if the subdevice changed the format */
>>>>> if (isc->try_config.sd_format && isc->config.sd_format &&
>>>>> isc->try_config.sd_format != isc->config.sd_format) {
>>>>> isc->ctrls.hist_stat = HIST_INIT;
>>>>> isc_reset_awb_ctrls(isc);
>>>>> isc_update_v4l2_ctrls(isc);
>>>>> }
>>>>> - /* make the try configuration active */
>>>>> +
>>>>> + /* Validate formats */
>>>>> + ret = isc_try_validate_formats(isc);
>>>>> + if (ret)
>>>>> + return ret;
>>>>> +
>>>>> + /* Obtain frame sizes if possible to have crop requirements ready */
>>>>> + isc_try_fse(isc, &pad_state);
>>>>> +
>>>>> + /* Configure ISC pipeline for the config */
>>>>> + ret = isc_try_configure_pipeline(isc);
>>>>> + if (ret)
>>>>> + return ret;
>>>>> +
>>>>> isc->config = isc->try_config;
>>>>>
>>>>> v4l2_dbg(1, debug, &isc->v4l2_dev, "New ISC configuration in place\n");
>>>>> @@ -1043,7 +1036,7 @@ static int isc_try_fmt_vid_cap(struct file *file, void *priv,
>>>>> {
>>>>> struct isc_device *isc = video_drvdata(file);
>>>>>
>>>>> - return isc_try_fmt(isc, f, NULL);
>>>>> + return isc_try_fmt(isc, f);
>>>>> }
>>>>>
>>>>> static int isc_enum_input(struct file *file, void *priv,
>>>>> @@ -1098,10 +1091,6 @@ static int isc_enum_framesizes(struct file *file, void *fh,
>>>>> if (fsize->index)
>>>>> return -EINVAL;
>>>>>
>>>>> - for (i = 0; i < isc->num_user_formats; i++)
>>>>> - if (isc->user_formats[i]->fourcc == fsize->pixel_format)
>>>>> - ret = 0;
>>>>> -
>>>>> for (i = 0; i < isc->controller_formats_size; i++)
>>>>> if (isc->controller_formats[i].fourcc == fsize->pixel_format)
>>>>> ret = 0;
>>>>> @@ -1782,52 +1771,6 @@ struct isc_format *isc_find_format_by_code(struct isc_device *isc,
>>>>> }
>>>>> EXPORT_SYMBOL_GPL(isc_find_format_by_code);
>>>>>
>>>>> -static int isc_formats_init(struct isc_device *isc)
>>>>> -{
>>>>> - struct isc_format *fmt;
>>>>> - struct v4l2_subdev *subdev = isc->current_subdev->sd;
>>>>> - unsigned int num_fmts, i, j;
>>>>> - u32 list_size = isc->formats_list_size;
>>>>> - struct v4l2_subdev_mbus_code_enum mbus_code = {
>>>>> - .which = V4L2_SUBDEV_FORMAT_ACTIVE,
>>>>> - };
>>>>> -
>>>>> - num_fmts = 0;
>>>>> - while (!v4l2_subdev_call(subdev, pad, enum_mbus_code,
>>>>> - NULL, &mbus_code)) {
>>>>> - mbus_code.index++;
>>>>> -
>>>>> - fmt = isc_find_format_by_code(isc, mbus_code.code, &i);
>>>>> - if (!fmt) {
>>>>> - v4l2_warn(&isc->v4l2_dev, "Mbus code %x not supported\n",
>>>>> - mbus_code.code);
>>>>> - continue;
>>>>> - }
>>>>> -
>>>>> - fmt->sd_support = true;
>>>>> - num_fmts++;
>>>>> - }
>>>>> -
>>>>> - if (!num_fmts)
>>>>> - return -ENXIO;
>>>>> -
>>>>> - isc->num_user_formats = num_fmts;
>>>>> - isc->user_formats = devm_kcalloc(isc->dev,
>>>>> - num_fmts, sizeof(*isc->user_formats),
>>>>> - GFP_KERNEL);
>>>>> - if (!isc->user_formats)
>>>>> - return -ENOMEM;
>>>>> -
>>>>> - fmt = &isc->formats_list[0];
>>>>> - for (i = 0, j = 0; i < list_size; i++) {
>>>>> - if (fmt->sd_support)
>>>>> - isc->user_formats[j++] = fmt;
>>>>> - fmt++;
>>>>> - }
>>>>> -
>>>>> - return 0;
>>>>> -}
>>>>> -
>>>>> static int isc_set_default_fmt(struct isc_device *isc)
>>>>> {
>>>>> struct v4l2_format f = {
>>>>> @@ -1836,12 +1779,12 @@ static int isc_set_default_fmt(struct isc_device *isc)
>>>>> .width = VGA_WIDTH,
>>>>> .height = VGA_HEIGHT,
>>>>> .field = V4L2_FIELD_NONE,
>>>>> - .pixelformat = isc->user_formats[0]->fourcc,
>>>>> + .pixelformat = isc->controller_formats[0].fourcc,
>>>>> },
>>>>> };
>>>>> int ret;
>>>>>
>>>>> - ret = isc_try_fmt(isc, &f, NULL);
>>>>> + ret = isc_try_fmt(isc, &f);
>>>>> if (ret)
>>>>> return ret;
>>>>>
>>>>> @@ -1896,13 +1839,6 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier)
>>>>> spin_lock_init(&isc->dma_queue_lock);
>>>>> spin_lock_init(&isc->awb_lock);
>>>>>
>>>>> - ret = isc_formats_init(isc);
>>>>> - if (ret < 0) {
>>>>> - v4l2_err(&isc->v4l2_dev,
>>>>> - "Init format failed: %d\n", ret);
>>>>> - goto isc_async_complete_err;
>>>>> - }
>>>>> -
>>>>> ret = isc_set_default_fmt(isc);
>>>>> if (ret) {
>>>>> v4l2_err(&isc->v4l2_dev, "Could not set default format\n");
>>>>> @@ -2016,6 +1952,24 @@ int isc_pipeline_init(struct isc_device *isc)
>>>>> }
>>>>> EXPORT_SYMBOL_GPL(isc_pipeline_init);
>>>>>
>>>>> +static int isc_link_validate(struct media_link *link)
>>>>> +{
>>>>> + struct video_device *vdev =
>>>>> + media_entity_to_video_device(link->sink->entity);
>>>>> + struct isc_device *isc = video_get_drvdata(vdev);
>>>>> + int ret;
>>>>> +
>>>>> + ret = v4l2_subdev_link_validate(link);
>>>>> + if (ret)
>>>>> + return ret;
>>>>> +
>>>>> + return isc_validate(isc);
>>>>> +}
>>>>> +
>>>>> +static const struct media_entity_operations isc_entity_operations = {
>>>>> + .link_validate = isc_link_validate,
>>>>> +};
>>>>> +
>>>>> int isc_mc_init(struct isc_device *isc, u32 ver)
>>>>> {
>>>>> const struct of_device_id *match;
>>>>> @@ -2023,6 +1977,8 @@ int isc_mc_init(struct isc_device *isc, u32 ver)
>>>>>
>>>>> isc->video_dev.entity.function = MEDIA_ENT_F_IO_V4L;
>>>>> isc->video_dev.entity.flags = MEDIA_ENT_FL_DEFAULT;
>>>>> + isc->video_dev.entity.ops = &isc_entity_operations;
>>>>> +
>>>>> isc->pads[ISC_PAD_SINK].flags = MEDIA_PAD_FL_SINK;
>>>>>
>>>>> ret = media_entity_pads_init(&isc->video_dev.entity, ISC_PADS_NUM,
>>>>> diff --git a/drivers/media/platform/atmel/atmel-isc-scaler.c b/drivers/media/platform/atmel/atmel-isc-scaler.c
>>>>> index a1ca4633787c..2b0621297dfe 100644
>>>>> --- a/drivers/media/platform/atmel/atmel-isc-scaler.c
>>>>> +++ b/drivers/media/platform/atmel/atmel-isc-scaler.c
>>>>> @@ -173,6 +173,10 @@ static const struct v4l2_subdev_pad_ops isc_scaler_pad_ops = {
>>>>> .init_cfg = isc_scaler_init_cfg,
>>>>> };
>>>>>
>>>>> +static const struct media_entity_operations isc_scaler_entity_ops = {
>>>>> + .link_validate = v4l2_subdev_link_validate,
>>>>> +};
>>>>> +
>>>>> static const struct v4l2_subdev_ops xisc_scaler_subdev_ops = {
>>>>> .pad = &isc_scaler_pad_ops,
>>>>> };
>>>>> @@ -190,6 +194,7 @@ int isc_scaler_init(struct isc_device *isc)
>>>>>
>>>>> isc->scaler_sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
>>>>> isc->scaler_sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
>>>>> + isc->scaler_sd.entity.ops = &isc_scaler_entity_ops;
>>>>> isc->scaler_pads[ISC_SCALER_PAD_SINK].flags = MEDIA_PAD_FL_SINK;
>>>>> isc->scaler_pads[ISC_SCALER_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
>>>>>
>>>>> diff --git a/drivers/media/platform/atmel/atmel-isc.h b/drivers/media/platform/atmel/atmel-isc.h
>>>>> index f98f25a55e73..5b0100c8c245 100644
>>>>> --- a/drivers/media/platform/atmel/atmel-isc.h
>>>>> +++ b/drivers/media/platform/atmel/atmel-isc.h
>>>>> @@ -63,15 +63,16 @@ struct isc_subdev_entity {
>>>>> * @cfa_baycfg: If this format is RAW BAYER, indicate the type of bayer.
>>>>> this is either BGBG, RGRG, etc.
>>>>> * @pfe_cfg0_bps: Number of hardware data lines connected to the ISC
>>>>> + * @raw: If the format is raw bayer.
>>>>> */
>>>>>
>>>>> struct isc_format {
>>>>> u32 fourcc;
>>>>> u32 mbus_code;
>>>>> u32 cfa_baycfg;
>>>>> -
>>>>> - bool sd_support;
>>>>> u32 pfe_cfg0_bps;
>>>>> +
>>>>> + bool raw;
>>>>> };
>>>>>
>>>>> /* Pipeline bitmap */
>>>>> @@ -216,8 +217,7 @@ enum isc_scaler_pads {
>>>>> * @comp: completion reference that signals frame completion
>>>>> *
>>>>> * @fmt: current v42l format
>>>>> - * @user_formats: list of formats that are supported and agreed with sd
>>>>> - * @num_user_formats: how many formats are in user_formats
>>>>> + * @try_fmt: current v4l2 try format
>>>>> *
>>>>> * @config: current ISC format configuration
>>>>> * @try_config: the current ISC try format , not yet activated
>>>>> @@ -272,6 +272,7 @@ enum isc_scaler_pads {
>>>>> * @formats_list_size: size of formats_list array
>>>>> * @pads: media controller pads for isc video entity
>>>>> * @mdev: media device that is registered by the isc
>>>>> + * @mpipe: media device pipeline used by the isc
>>>>> * @remote_pad: remote pad on the connected subdevice
>>>>> * @scaler_sd: subdevice for the scaler that isc registers
>>>>> * @scaler_pads: media controller pads for the scaler subdevice
>>>>> @@ -298,8 +299,7 @@ struct isc_device {
>>>>> struct completion comp;
>>>>>
>>>>> struct v4l2_format fmt;
>>>>> - struct isc_format **user_formats;
>>>>> - unsigned int num_user_formats;
>>>>> + struct v4l2_format try_fmt;
>>>>>
>>>>> struct fmt_config config;
>>>>> struct fmt_config try_config;
>>>>> @@ -369,6 +369,7 @@ struct isc_device {
>>>>> struct {
>>>>> struct media_pad pads[ISC_PADS_NUM];
>>>>> struct media_device mdev;
>>>>> + struct media_pipeline mpipe;
>>>>>
>>>>> u32 remote_pad;
>>>>> };
>>>>> diff --git a/drivers/media/platform/atmel/atmel-sama5d2-isc.c b/drivers/media/platform/atmel/atmel-sama5d2-isc.c
>>>>> index d96ee3373889..2a651e237193 100644
>>>>> --- a/drivers/media/platform/atmel/atmel-sama5d2-isc.c
>>>>> +++ b/drivers/media/platform/atmel/atmel-sama5d2-isc.c
>>>>> @@ -80,20 +80,40 @@ static const struct isc_format sama5d2_controller_formats[] = {
>>>>> .fourcc = V4L2_PIX_FMT_Y10,
>>>>> }, {
>>>>> .fourcc = V4L2_PIX_FMT_SBGGR8,
>>>>> + .raw = true,
>>>>> }, {
>>>>> .fourcc = V4L2_PIX_FMT_SGBRG8,
>>>>> + .raw = true,
>>>>> }, {
>>>>> .fourcc = V4L2_PIX_FMT_SGRBG8,
>>>>> + .raw = true,
>>>>> }, {
>>>>> .fourcc = V4L2_PIX_FMT_SRGGB8,
>>>>> + .raw = true,
>>>>> }, {
>>>>> .fourcc = V4L2_PIX_FMT_SBGGR10,
>>>>> + .raw = true,
>>>>> }, {
>>>>> .fourcc = V4L2_PIX_FMT_SGBRG10,
>>>>> + .raw = true,
>>>>> }, {
>>>>> .fourcc = V4L2_PIX_FMT_SGRBG10,
>>>>> + .raw = true,
>>>>> }, {
>>>>> .fourcc = V4L2_PIX_FMT_SRGGB10,
>>>>> + .raw = true,
>>>>> + }, {
>>>>> + .fourcc = V4L2_PIX_FMT_SBGGR12,
>>>>> + .raw = true,
>>>>> + }, {
>>>>> + .fourcc = V4L2_PIX_FMT_SGBRG12,
>>>>> + .raw = true,
>>>>> + }, {
>>>>> + .fourcc = V4L2_PIX_FMT_SGRBG12,
>>>>> + .raw = true,
>>>>> + }, {
>>>>> + .fourcc = V4L2_PIX_FMT_SRGGB12,
>>>>> + .raw = true,
>>>>> },
>>>>> };
>>>>>
>>>>> diff --git a/drivers/media/platform/atmel/atmel-sama7g5-isc.c b/drivers/media/platform/atmel/atmel-sama7g5-isc.c
>>>>> index e07ae188c15f..a0d60cfdba7b 100644
>>>>> --- a/drivers/media/platform/atmel/atmel-sama7g5-isc.c
>>>>> +++ b/drivers/media/platform/atmel/atmel-sama7g5-isc.c
>>>>> @@ -89,20 +89,40 @@ static const struct isc_format sama7g5_controller_formats[] = {
>>>>> .fourcc = V4L2_PIX_FMT_Y16,
>>>>> }, {
>>>>> .fourcc = V4L2_PIX_FMT_SBGGR8,
>>>>> + .raw = true,
>>>>> }, {
>>>>> .fourcc = V4L2_PIX_FMT_SGBRG8,
>>>>> + .raw = true,
>>>>> }, {
>>>>> .fourcc = V4L2_PIX_FMT_SGRBG8,
>>>>> + .raw = true,
>>>>> }, {
>>>>> .fourcc = V4L2_PIX_FMT_SRGGB8,
>>>>> + .raw = true,
>>>>> }, {
>>>>> .fourcc = V4L2_PIX_FMT_SBGGR10,
>>>>> + .raw = true,
>>>>> }, {
>>>>> .fourcc = V4L2_PIX_FMT_SGBRG10,
>>>>> + .raw = true,
>>>>> }, {
>>>>> .fourcc = V4L2_PIX_FMT_SGRBG10,
>>>>> + .raw = true,
>>>>> }, {
>>>>> .fourcc = V4L2_PIX_FMT_SRGGB10,
>>>>> + .raw = true,
>>>>> + }, {
>>>>> + .fourcc = V4L2_PIX_FMT_SBGGR12,
>>>>> + .raw = true,
>>>>> + }, {
>>>>> + .fourcc = V4L2_PIX_FMT_SGBRG12,
>>>>> + .raw = true,
>>>>> + }, {
>>>>> + .fourcc = V4L2_PIX_FMT_SGRBG12,
>>>>> + .raw = true,
>>>>> + }, {
>>>>> + .fourcc = V4L2_PIX_FMT_SRGGB12,
>>>>> + .raw = true,
>>>>> },
>>>>> };
>>>>>
>>>>
>>>
>>

2022-04-29 21:57:19

[permalink] [raw]

Subject: Re: [PATCH v9 08/13] media: atmel: atmel-isc: change format propagation to subdev into only verification

On 4/29/22 1:03 PM, Hans Verkuil wrote:
> On 29/04/2022 11:58, Jacopo Mondi wrote:
>> Hello
>>
>> On Fri, Apr 29, 2022 at 10:43:09AM +0200, Hans Verkuil wrote:
>>> On 29/04/2022 10:28, [email protected] wrote:
>>>> On 4/29/22 11:17 AM, Hans Verkuil wrote:
>>>>> On 10/03/2022 10:51, Eugen Hristev wrote:
>>>>>> As a top MC video driver, the atmel-isc should not propagate the format to the
>>>>>> subdevice, it should rather check at start_streaming() time if the subdev is properly
>>>>>> configured with a compatible format.
>>>>>> Removed the whole format finding logic, and reworked the format verification
>>>>>> at start_streaming time, such that the ISC will return an error if the subdevice
>>>>>> is not properly configured. To achieve this, media_pipeline_start
>>>>>> is called and a link_validate callback is created to check the formats.
>>>>>> With this being done, the module parameter 'sensor_preferred' makes no sense
>>>>>> anymore. The ISC should not decide which format the sensor is using. The
>>>>>> ISC should only cope with the situation and inform userspace if the streaming
>>>>>> is possible in the current configuration.
>>>>>> The redesign of the format propagation has also risen the question of the
>>>>>> enumfmt callback. If enumfmt is called with an mbus_code, the enumfmt handler
>>>>>> should only return the formats that are supported for this mbus_code.
>>>>>> Otherwise, the enumfmt will report all the formats that the ISC could output.
>>>>>> With this rework, the dynamic list of user formats is removed. It makes no
>>>>>> more sense to identify at complete time which formats the sensor could emit,
>>>>>> and add those into a separate dynamic list.
>>>>>> The ISC will start with a simple preconfigured default format, and at
>>>>>> link validate time, decide whether it can use the format that is configured
>>>>>> on the sink or not.
>>>>>>
>>>>>> Signed-off-by: Eugen Hristev <[email protected]>
>>>>>> Reviewed-by: Jacopo Mondi <[email protected]>
>>>>>> ---
>>>>>> Changes in v9:
>>>>>> - isc_link_validate now static
>>>>>>
>>>>>> Changes in v7:
>>>>>> - minor typos as suggested by Jacopo
>>>>>> - small changes, reduce some indentation, modified an index, as suggested by
>>>>>> Jacopo
>>>>>>
>>>>>> Changes in v6:
>>>>>> - reworked a bit enum_fmt as suggested by Jacopo
>>>>>>
>>>>>> Changes in v5:
>>>>>> - removed user_formats dynamic list as it is now pointless
>>>>>> - greatly simplified the enum_fmt function
>>>>>> - removed some init code that was useless now
>>>>>>
>>>>>> Changes in v4:
>>>>>> - moved validation code into link_validate and used media_pipeline_start
>>>>>> - merged this patch with the enum_fmt patch which was previously in v3 of
>>>>>> the series
>>>>>>
>>>>>> Changes in v3:
>>>>>> - clamp to maximum resolution once the frame size from the subdev is found
>>>>>> drivers/media/platform/atmel/atmel-isc-base.c | 412 ++++++++----------
>>>>>> .../media/platform/atmel/atmel-isc-scaler.c | 5 +
>>>>>> drivers/media/platform/atmel/atmel-isc.h | 13 +-
>>>>>> .../media/platform/atmel/atmel-sama5d2-isc.c | 20 +
>>>>>> .../media/platform/atmel/atmel-sama7g5-isc.c | 20 +
>>>>>> 5 files changed, 236 insertions(+), 234 deletions(-)
>>>>>>
>>>>>> diff --git a/drivers/media/platform/atmel/atmel-isc-base.c b/drivers/media/platform/atmel/atmel-isc-base.c
>>>>>> index ee1dda6707a0..fe2c0af58060 100644
>>>>>> --- a/drivers/media/platform/atmel/atmel-isc-base.c
>>>>>> +++ b/drivers/media/platform/atmel/atmel-isc-base.c
>>>>>> @@ -36,11 +36,6 @@ static unsigned int debug;
>>>>>> module_param(debug, int, 0644);
>>>>>> MODULE_PARM_DESC(debug, "debug level (0-2)");
>>>>>>
>>>>>> -static unsigned int sensor_preferred = 1;
>>>>>> -module_param(sensor_preferred, uint, 0644);
>>>>>> -MODULE_PARM_DESC(sensor_preferred,
>>>>>> - "Sensor is preferred to output the specified format (1-on 0-off), default 1");
>>>>>> -
>>>>>> #define ISC_IS_FORMAT_RAW(mbus_code) \
>>>>>> (((mbus_code) & 0xf000) == 0x3000)
>>>>>>
>>>>>> @@ -337,6 +332,10 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)
>>>>>> unsigned long flags;
>>>>>> int ret;
>>>>>>
>>>>>> + ret = media_pipeline_start(&isc->video_dev.entity, &isc->mpipe);
>>>>>
>>>>> The pipeline validation is done in start_streaming, but I don't think that
>>>>> is the best place: if STREAMON is called before buffers are queued, then
>>>>> an invalid pipeline isn't discovered until enough buffers are queued to
>>>>> kick off start_streaming.
>>>>>
>>>>> Drivers like vsp1, omap3isp and the samsung drivers all do this in streamon().
>>>>>
>>>>> I think that is the correct time to do this.
>>>>
>>>> Hello Hans,
>>>>
>>>> Initially (v2, v3) I had this in streamon(). The problem that I faced at
>>>> that time was that streamoff was never called, so I could not call
>>>> media_pipeline_stop(). Then Jacopo told me to move it to start_streaming
>>>> (see change log for v4) , and I did not face any more problems.
>>
>> Yes indeed, seems I suggested to use media_pipeline_handler in a
>> comment on your v3
>>
>> "at s_stream time your top driver calls media_pipeline_start()"
>>
>> sorry about that, I should have looked around a bit more carefully and
>> notice most drivers do so at vb2 streamon
>>
>> However I don't see media_pipeline_start being called at all in v3 of
>> the patch

No need to be sorry. Your suggestion was good and some other drivers do
it, and it works well. We use this in production at the moment.
So thank you for that.

>>
>>>
>>> It's a mess. Looking at some drivers I see that omap3isp calls media_pipeline_stop
>>> in streamoff (so will have the same problem as you described if VIDIOC_STREAMOFF
>>> isn't called), exynos4-is does the same, but it also checks the streaming state in
>>> the release() fop callback, so that would fix this problem. And vimc does this
>>> in stop_streaming.
>>>
>>> I'm in favor of fixing this in vb2, that framework knows exactly when this needs
>>> to be called.
>>
>> Are you suggesting to have vb2 to call media_pipeline_start() or is it
>> more complex than this ?
>
> No, I don't think vb2 should do that. But it needs two new ops: pipeline_start/stop
> (or some other name) that are called at VIDIOC_STREAMON time and when the streaming
> stops (due to STREAMOFF or by closing the fh). The driver can then take appropriate
> action, which will typically be calling media_pipeline_start/stop, but other checks
> may also be needed at that time.

I can try to implement this and test with my driver, and come up with a
proposal, if that is fine with you.

Eugen
>
> Regards,
>
> Hans
>
>>
>>>
>>> I'd like to know what others think.
>>>
>>> Regards,
>>>
>>> Hans
>>>
>>>>
>>>> Eugen
>>>>
>>>>>
>>>>> Laurent, Jacopo, do you agree with that?
>>>>>
>>>>> I notice that quite a few other drivers do this in start_streaming as well,
>>>>> but I think it just weird that a QBUF can return EPIPE while STREAMON might
>>>>> succeed.
>>>>>
>>>>> Perhaps we need an optional vb2 validate() callback that is called by
>>>>> vb2_core_streamon() to validate the pipeline before streaming. That makes
>>>>> it easier to implement these checks rather than having to implement your
>>>>> own streamon ioctl.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Hans
>>>>>
>>>>>> + if (ret)
>>>>>> + goto err_pipeline_start;
>>>>>> +
>>>>>> /* Enable stream on the sub device */
>>>>>> ret = v4l2_subdev_call(isc->current_subdev->sd, video, s_stream, 1);
>>>>>> if (ret && ret != -ENOIOCTLCMD) {
>>>>>> @@ -385,6 +384,9 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)
>>>>>> v4l2_subdev_call(isc->current_subdev->sd, video, s_stream, 0);
>>>>>>
>>>>>> err_start_stream:
>>>>>> + media_pipeline_stop(&isc->video_dev.entity);
>>>>>> +
>>>>>> +err_pipeline_start:
>>>>>> spin_lock_irqsave(&isc->dma_queue_lock, flags);
>>>>>> list_for_each_entry(buf, &isc->dma_queue, list)
>>>>>> vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_QUEUED);
>>>>>> @@ -423,6 +425,9 @@ static void isc_stop_streaming(struct vb2_queue *vq)
>>>>>> if (ret && ret != -ENOIOCTLCMD)
>>>>>> v4l2_err(&isc->v4l2_dev, "stream off failed in subdev\n");
>>>>>>
>>>>>> + /* Stop media pipeline */
>>>>>> + media_pipeline_stop(&isc->video_dev.entity);
>>>>>> +
>>>>>> /* Release all active buffers */
>>>>>> spin_lock_irqsave(&isc->dma_queue_lock, flags);
>>>>>> if (unlikely(isc->cur_frm)) {
>>>>>> @@ -453,22 +458,6 @@ static void isc_buffer_queue(struct vb2_buffer *vb)
>>>>>> spin_unlock_irqrestore(&isc->dma_queue_lock, flags);
>>>>>> }
>>>>>>
>>>>>> -static struct isc_format *find_format_by_fourcc(struct isc_device *isc,
>>>>>> - unsigned int fourcc)
>>>>>> -{
>>>>>> - unsigned int num_formats = isc->num_user_formats;
>>>>>> - struct isc_format *fmt;
>>>>>> - unsigned int i;
>>>>>> -
>>>>>> - for (i = 0; i < num_formats; i++) {
>>>>>> - fmt = isc->user_formats[i];
>>>>>> - if (fmt->fourcc == fourcc)
>>>>>> - return fmt;
>>>>>> - }
>>>>>> -
>>>>>> - return NULL;
>>>>>> -}
>>>>>> -
>>>>>> static const struct vb2_ops isc_vb2_ops = {
>>>>>> .queue_setup = isc_queue_setup,
>>>>>> .wait_prepare = vb2_ops_wait_prepare,
>>>>>> @@ -497,23 +486,57 @@ static int isc_enum_fmt_vid_cap(struct file *file, void *priv,
>>>>>> {
>>>>>> struct isc_device *isc = video_drvdata(file);
>>>>>> u32 index = f->index;
>>>>>> - u32 i, supported_index;
>>>>>> + u32 i, supported_index = 0;
>>>>>> + struct isc_format *fmt;
>>>>>> +
>>>>>> + /*
>>>>>> + * If we are not asked a specific mbus_code, we have to report all
>>>>>> + * the formats that we can output.
>>>>>> + */
>>>>>> + if (!f->mbus_code) {
>>>>>> + if (index >= isc->controller_formats_size)
>>>>>> + return -EINVAL;
>>>>>>
>>>>>> - if (index < isc->controller_formats_size) {
>>>>>> f->pixelformat = isc->controller_formats[index].fourcc;
>>>>>> +
>>>>>> + return 0;
>>>>>> + }
>>>>>> +
>>>>>> + /*
>>>>>> + * If a specific mbus_code is requested, check if we support
>>>>>> + * this mbus_code as input for the ISC.
>>>>>> + * If it's supported, then we report the corresponding pixelformat
>>>>>> + * as first possible option for the ISC.
>>>>>> + * E.g. mbus MEDIA_BUS_FMT_YUYV8_2X8 and report
>>>>>> + * 'YUYV' (YUYV 4:2:2)
>>>>>> + */
>>>>>> + fmt = isc_find_format_by_code(isc, f->mbus_code, &i);
>>>>>> + if (!fmt)
>>>>>> + return -EINVAL;
>>>>>> +
>>>>>> + if (!index) {
>>>>>> + f->pixelformat = fmt->fourcc;
>>>>>> +
>>>>>> return 0;
>>>>>> }
>>>>>>
>>>>>> - index -= isc->controller_formats_size;
>>>>>> + supported_index++;
>>>>>>
>>>>>> - supported_index = 0;
>>>>>> + /* If the index is not raw, we don't have anymore formats to report */
>>>>>> + if (!ISC_IS_FORMAT_RAW(f->mbus_code))
>>>>>> + return -EINVAL;
>>>>>>
>>>>>> - for (i = 0; i < isc->formats_list_size; i++) {
>>>>>> - if (!ISC_IS_FORMAT_RAW(isc->formats_list[i].mbus_code) ||
>>>>>> - !isc->formats_list[i].sd_support)
>>>>>> + /*
>>>>>> + * We are asked for a specific mbus code, which is raw.
>>>>>> + * We have to search through the formats we can convert to.
>>>>>> + * We have to skip the raw formats, we cannot convert to raw.
>>>>>> + * E.g. 'AR12' (16-bit ARGB 4-4-4-4), 'AR15' (16-bit ARGB 1-5-5-5), etc.
>>>>>> + */
>>>>>> + for (i = 0; i < isc->controller_formats_size; i++) {
>>>>>> + if (isc->controller_formats[i].raw)
>>>>>> continue;
>>>>>> - if (supported_index == index) {
>>>>>> - f->pixelformat = isc->formats_list[i].fourcc;
>>>>>> + if (index == supported_index) {
>>>>>> + f->pixelformat = isc->controller_formats[i].fourcc;
>>>>>> return 0;
>>>>>> }
>>>>>> supported_index++;
>>>>>> @@ -584,20 +607,30 @@ static int isc_try_validate_formats(struct isc_device *isc)
>>>>>> break;
>>>>>> default:
>>>>>> /* any other different formats are not supported */
>>>>>> + v4l2_err(&isc->v4l2_dev, "Requested unsupported format.\n");
>>>>>> ret = -EINVAL;
>>>>>> }
>>>>>> v4l2_dbg(1, debug, &isc->v4l2_dev,
>>>>>> "Format validation, requested rgb=%u, yuv=%u, grey=%u, bayer=%u\n",
>>>>>> rgb, yuv, grey, bayer);
>>>>>>
>>>>>> - /* we cannot output RAW if we do not receive RAW */
>>>>>> - if ((bayer) && !ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code))
>>>>>> + if (bayer &&
>>>>>> + !ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code)) {
>>>>>> + v4l2_err(&isc->v4l2_dev, "Cannot output RAW if we do not receive RAW.\n");
>>>>>> return -EINVAL;
>>>>>> + }
>>>>>>
>>>>>> - /* we cannot output GREY if we do not receive RAW/GREY */
>>>>>> if (grey && !ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code) &&
>>>>>> - !ISC_IS_FORMAT_GREY(isc->try_config.sd_format->mbus_code))
>>>>>> + !ISC_IS_FORMAT_GREY(isc->try_config.sd_format->mbus_code)) {
>>>>>> + v4l2_err(&isc->v4l2_dev, "Cannot output GREY if we do not receive RAW/GREY.\n");
>>>>>> return -EINVAL;
>>>>>> + }
>>>>>> +
>>>>>> + if ((rgb || bayer || yuv) &&
>>>>>> + ISC_IS_FORMAT_GREY(isc->try_config.sd_format->mbus_code)) {
>>>>>> + v4l2_err(&isc->v4l2_dev, "Cannot convert GREY to another format.\n");
>>>>>> + return -EINVAL;
>>>>>> + }
>>>>>>
>>>>>> return ret;
>>>>>> }
>>>>>> @@ -825,7 +858,7 @@ static void isc_try_fse(struct isc_device *isc,
>>>>>> * If we do not know yet which format the subdev is using, we cannot
>>>>>> * do anything.
>>>>>> */
>>>>>> - if (!isc->try_config.sd_format)
>>>>>> + if (!isc->config.sd_format)
>>>>>> return;
>>>>>>
>>>>>> fse.code = isc->try_config.sd_format->mbus_code;
>>>>>> @@ -846,180 +879,140 @@ static void isc_try_fse(struct isc_device *isc,
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> -static int isc_try_fmt(struct isc_device *isc, struct v4l2_format *f,
>>>>>> - u32 *code)
>>>>>> +static int isc_try_fmt(struct isc_device *isc, struct v4l2_format *f)
>>>>>> {
>>>>>> - int i;
>>>>>> - struct isc_format *sd_fmt = NULL, *direct_fmt = NULL;
>>>>>> struct v4l2_pix_format *pixfmt = &f->fmt.pix;
>>>>>> - struct v4l2_subdev_pad_config pad_cfg = {};
>>>>>> - struct v4l2_subdev_state pad_state = {
>>>>>> - .pads = &pad_cfg
>>>>>> - };
>>>>>> - struct v4l2_subdev_format format = {
>>>>>> - .which = V4L2_SUBDEV_FORMAT_TRY,
>>>>>> - };
>>>>>> - u32 mbus_code;
>>>>>> - int ret;
>>>>>> - bool rlp_dma_direct_dump = false;
>>>>>> + unsigned int i;
>>>>>>
>>>>>> if (f->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
>>>>>> return -EINVAL;
>>>>>>
>>>>>> - /* Step 1: find a RAW format that is supported */
>>>>>> - for (i = 0; i < isc->num_user_formats; i++) {
>>>>>> - if (ISC_IS_FORMAT_RAW(isc->user_formats[i]->mbus_code)) {
>>>>>> - sd_fmt = isc->user_formats[i];
>>>>>> + isc->try_config.fourcc = isc->controller_formats[0].fourcc;
>>>>>> +
>>>>>> + /* find if the format requested is supported */
>>>>>> + for (i = 0; i < isc->controller_formats_size; i++)
>>>>>> + if (isc->controller_formats[i].fourcc == pixfmt->pixelformat) {
>>>>>> + isc->try_config.fourcc = pixfmt->pixelformat;
>>>>>> break;
>>>>>> }
>>>>>> - }
>>>>>> - /* Step 2: We can continue with this RAW format, or we can look
>>>>>> - * for better: maybe sensor supports directly what we need.
>>>>>> - */
>>>>>> - direct_fmt = find_format_by_fourcc(isc, pixfmt->pixelformat);
>>>>>> -
>>>>>> - /* Step 3: We have both. We decide given the module parameter which
>>>>>> - * one to use.
>>>>>> - */
>>>>>> - if (direct_fmt && sd_fmt && sensor_preferred)
>>>>>> - sd_fmt = direct_fmt;
>>>>>> -
>>>>>> - /* Step 4: we do not have RAW but we have a direct format. Use it. */
>>>>>> - if (direct_fmt && !sd_fmt)
>>>>>> - sd_fmt = direct_fmt;
>>>>>> -
>>>>>> - /* Step 5: if we are using a direct format, we need to package
>>>>>> - * everything as 8 bit data and just dump it
>>>>>> - */
>>>>>> - if (sd_fmt == direct_fmt)
>>>>>> - rlp_dma_direct_dump = true;
>>>>>> -
>>>>>> - /* Step 6: We have no format. This can happen if the userspace
>>>>>> - * requests some weird/invalid format.
>>>>>> - * In this case, default to whatever we have
>>>>>> - */
>>>>>> - if (!sd_fmt && !direct_fmt) {
>>>>>> - sd_fmt = isc->user_formats[isc->num_user_formats - 1];
>>>>>> - v4l2_dbg(1, debug, &isc->v4l2_dev,
>>>>>> - "Sensor not supporting %.4s, using %.4s\n",
>>>>>> - (char *)&pixfmt->pixelformat, (char *)&sd_fmt->fourcc);
>>>>>> - }
>>>>>> -
>>>>>> - if (!sd_fmt) {
>>>>>> - ret = -EINVAL;
>>>>>> - goto isc_try_fmt_err;
>>>>>> - }
>>>>>> -
>>>>>> - /* Step 7: Print out what we decided for debugging */
>>>>>> - v4l2_dbg(1, debug, &isc->v4l2_dev,
>>>>>> - "Preferring to have sensor using format %.4s\n",
>>>>>> - (char *)&sd_fmt->fourcc);
>>>>>>
>>>>>> - /* Step 8: at this moment we decided which format the subdev will use */
>>>>>> - isc->try_config.sd_format = sd_fmt;
>>>>>> + isc_try_configure_rlp_dma(isc, false);
>>>>>>
>>>>>> /* Limit to Atmel ISC hardware capabilities */
>>>>>> - if (pixfmt->width > isc->max_width)
>>>>>> - pixfmt->width = isc->max_width;
>>>>>> - if (pixfmt->height > isc->max_height)
>>>>>> - pixfmt->height = isc->max_height;
>>>>>> -
>>>>>> - /*
>>>>>> - * The mbus format is the one the subdev outputs.
>>>>>> - * The pixels will be transferred in this format Sensor -> ISC
>>>>>> - */
>>>>>> - mbus_code = sd_fmt->mbus_code;
>>>>>> -
>>>>>> - /*
>>>>>> - * Validate formats. If the required format is not OK, default to raw.
>>>>>> - */
>>>>>> -
>>>>>> - isc->try_config.fourcc = pixfmt->pixelformat;
>>>>>> -
>>>>>> - if (isc_try_validate_formats(isc)) {
>>>>>> - pixfmt->pixelformat = isc->try_config.fourcc = sd_fmt->fourcc;
>>>>>> - /* Re-try to validate the new format */
>>>>>> - ret = isc_try_validate_formats(isc);
>>>>>> - if (ret)
>>>>>> - goto isc_try_fmt_err;
>>>>>> - }
>>>>>> -
>>>>>> - ret = isc_try_configure_rlp_dma(isc, rlp_dma_direct_dump);
>>>>>> - if (ret)
>>>>>> - goto isc_try_fmt_err;
>>>>>> -
>>>>>> - ret = isc_try_configure_pipeline(isc);
>>>>>> - if (ret)
>>>>>> - goto isc_try_fmt_err;
>>>>>> -
>>>>>> - /* Obtain frame sizes if possible to have crop requirements ready */
>>>>>> - isc_try_fse(isc, &pad_state);
>>>>>> -
>>>>>> - v4l2_fill_mbus_format(&format.format, pixfmt, mbus_code);
>>>>>> - ret = v4l2_subdev_call(isc->current_subdev->sd, pad, set_fmt,
>>>>>> - &pad_state, &format);
>>>>>> - if (ret < 0)
>>>>>> - goto isc_try_fmt_subdev_err;
>>>>>> -
>>>>>> - v4l2_fill_pix_format(pixfmt, &format.format);
>>>>>> + v4l_bound_align_image(&pixfmt->width, 16, isc->max_width, 0,
>>>>>> + &pixfmt->height, 16, isc->max_height, 0, 0);
>>>>>> + /* If we did not find the requested format, we will fallback here */
>>>>>> + pixfmt->pixelformat = isc->try_config.fourcc;
>>>>>> + pixfmt->colorspace = V4L2_COLORSPACE_SRGB;
>>>>>> + pixfmt->field = V4L2_FIELD_NONE;
>>>>>>
>>>>>> - /* Limit to Atmel ISC hardware capabilities */
>>>>>> - if (pixfmt->width > isc->max_width)
>>>>>> - pixfmt->width = isc->max_width;
>>>>>> - if (pixfmt->height > isc->max_height)
>>>>>> - pixfmt->height = isc->max_height;
>>>>>>
>>>>>> - pixfmt->field = V4L2_FIELD_NONE;
>>>>>> pixfmt->bytesperline = (pixfmt->width * isc->try_config.bpp_v4l2) >> 3;
>>>>>> pixfmt->sizeimage = ((pixfmt->width * isc->try_config.bpp) >> 3) *
>>>>>> pixfmt->height;
>>>>>>
>>>>>> - if (code)
>>>>>> - *code = mbus_code;
>>>>>> + isc->try_fmt = *f;
>>>>>>
>>>>>> return 0;
>>>>>> +}
>>>>>>
>>>>>> -isc_try_fmt_err:
>>>>>> - v4l2_err(&isc->v4l2_dev, "Could not find any possible format for a working pipeline\n");
>>>>>> -isc_try_fmt_subdev_err:
>>>>>> - memset(&isc->try_config, 0, sizeof(isc->try_config));
>>>>>> +static int isc_set_fmt(struct isc_device *isc, struct v4l2_format *f)
>>>>>> +{
>>>>>> + isc_try_fmt(isc, f);
>>>>>>
>>>>>> - return ret;
>>>>>> + /* make the try configuration active */
>>>>>> + isc->config = isc->try_config;
>>>>>> + isc->fmt = isc->try_fmt;
>>>>>> +
>>>>>> + v4l2_dbg(1, debug, &isc->v4l2_dev, "ISC set_fmt to %.4s @%dx%d\n",
>>>>>> + (char *)&f->fmt.pix.pixelformat,
>>>>>> + f->fmt.pix.width, f->fmt.pix.height);
>>>>>> +
>>>>>> + return 0;
>>>>>> }
>>>>>>
>>>>>> -static int isc_set_fmt(struct isc_device *isc, struct v4l2_format *f)
>>>>>> +static int isc_validate(struct isc_device *isc)
>>>>>> {
>>>>>> + int ret;
>>>>>> + int i;
>>>>>> + struct isc_format *sd_fmt = NULL;
>>>>>> + struct v4l2_pix_format *pixfmt = &isc->fmt.fmt.pix;
>>>>>> struct v4l2_subdev_format format = {
>>>>>> .which = V4L2_SUBDEV_FORMAT_ACTIVE,
>>>>>> + .pad = isc->remote_pad,
>>>>>> + };
>>>>>> + struct v4l2_subdev_pad_config pad_cfg = {};
>>>>>> + struct v4l2_subdev_state pad_state = {
>>>>>> + .pads = &pad_cfg,
>>>>>> };
>>>>>> - u32 mbus_code = 0;
>>>>>> - int ret;
>>>>>>
>>>>>> - ret = isc_try_fmt(isc, f, &mbus_code);
>>>>>> + /* Get current format from subdev */
>>>>>> + ret = v4l2_subdev_call(isc->current_subdev->sd, pad, get_fmt, NULL,
>>>>>> + &format);
>>>>>> if (ret)
>>>>>> return ret;
>>>>>>
>>>>>> - v4l2_fill_mbus_format(&format.format, &f->fmt.pix, mbus_code);
>>>>>> - ret = v4l2_subdev_call(isc->current_subdev->sd, pad,
>>>>>> - set_fmt, NULL, &format);
>>>>>> - if (ret < 0)
>>>>>> - return ret;
>>>>>> + /* Identify the subdev's format configuration */
>>>>>> + for (i = 0; i < isc->formats_list_size; i++)
>>>>>> + if (isc->formats_list[i].mbus_code == format.format.code) {
>>>>>> + sd_fmt = &isc->formats_list[i];
>>>>>> + break;
>>>>>> + }
>>>>>> +
>>>>>> + /* Check if the format is not supported */
>>>>>> + if (!sd_fmt) {
>>>>>> + v4l2_err(&isc->v4l2_dev,
>>>>>> + "Current subdevice is streaming a media bus code that is not supported 0x%x\n",
>>>>>> + format.format.code);
>>>>>> + return -EPIPE;
>>>>>> + }
>>>>>> +
>>>>>> + /* At this moment we know which format the subdev will use */
>>>>>> + isc->try_config.sd_format = sd_fmt;
>>>>>> +
>>>>>> + /* If the sensor is not RAW, we can only do a direct dump */
>>>>>> + if (!ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code))
>>>>>> + isc_try_configure_rlp_dma(isc, true);
>>>>>>
>>>>>> /* Limit to Atmel ISC hardware capabilities */
>>>>>> - if (f->fmt.pix.width > isc->max_width)
>>>>>> - f->fmt.pix.width = isc->max_width;
>>>>>> - if (f->fmt.pix.height > isc->max_height)
>>>>>> - f->fmt.pix.height = isc->max_height;
>>>>>> + v4l_bound_align_image(&format.format.width, 16, isc->max_width, 0,
>>>>>> + &format.format.height, 16, isc->max_height, 0, 0);
>>>>>>
>>>>>> - isc->fmt = *f;
>>>>>> + /* Check if the frame size is the same. Otherwise we may overflow */
>>>>>> + if (pixfmt->height != format.format.height ||
>>>>>> + pixfmt->width != format.format.width) {
>>>>>> + v4l2_err(&isc->v4l2_dev,
>>>>>> + "ISC not configured with the proper frame size: %dx%d\n",
>>>>>> + format.format.width, format.format.height);
>>>>>> + return -EPIPE;
>>>>>> + }
>>>>>>
>>>>>> + v4l2_dbg(1, debug, &isc->v4l2_dev,
>>>>>> + "Identified subdev using format %.4s with %dx%d %d bpp\n",
>>>>>> + (char *)&sd_fmt->fourcc, pixfmt->width, pixfmt->height,
>>>>>> + isc->try_config.bpp);
>>>>>> +
>>>>>> + /* Reset and restart AWB if the subdevice changed the format */
>>>>>> if (isc->try_config.sd_format && isc->config.sd_format &&
>>>>>> isc->try_config.sd_format != isc->config.sd_format) {
>>>>>> isc->ctrls.hist_stat = HIST_INIT;
>>>>>> isc_reset_awb_ctrls(isc);
>>>>>> isc_update_v4l2_ctrls(isc);
>>>>>> }
>>>>>> - /* make the try configuration active */
>>>>>> +
>>>>>> + /* Validate formats */
>>>>>> + ret = isc_try_validate_formats(isc);
>>>>>> + if (ret)
>>>>>> + return ret;
>>>>>> +
>>>>>> + /* Obtain frame sizes if possible to have crop requirements ready */
>>>>>> + isc_try_fse(isc, &pad_state);
>>>>>> +
>>>>>> + /* Configure ISC pipeline for the config */
>>>>>> + ret = isc_try_configure_pipeline(isc);
>>>>>> + if (ret)
>>>>>> + return ret;
>>>>>> +
>>>>>> isc->config = isc->try_config;
>>>>>>
>>>>>> v4l2_dbg(1, debug, &isc->v4l2_dev, "New ISC configuration in place\n");
>>>>>> @@ -1043,7 +1036,7 @@ static int isc_try_fmt_vid_cap(struct file *file, void *priv,
>>>>>> {
>>>>>> struct isc_device *isc = video_drvdata(file);
>>>>>>
>>>>>> - return isc_try_fmt(isc, f, NULL);
>>>>>> + return isc_try_fmt(isc, f);
>>>>>> }
>>>>>>
>>>>>> static int isc_enum_input(struct file *file, void *priv,
>>>>>> @@ -1098,10 +1091,6 @@ static int isc_enum_framesizes(struct file *file, void *fh,
>>>>>> if (fsize->index)
>>>>>> return -EINVAL;
>>>>>>
>>>>>> - for (i = 0; i < isc->num_user_formats; i++)
>>>>>> - if (isc->user_formats[i]->fourcc == fsize->pixel_format)
>>>>>> - ret = 0;
>>>>>> -
>>>>>> for (i = 0; i < isc->controller_formats_size; i++)
>>>>>> if (isc->controller_formats[i].fourcc == fsize->pixel_format)
>>>>>> ret = 0;
>>>>>> @@ -1782,52 +1771,6 @@ struct isc_format *isc_find_format_by_code(struct isc_device *isc,
>>>>>> }
>>>>>> EXPORT_SYMBOL_GPL(isc_find_format_by_code);
>>>>>>
>>>>>> -static int isc_formats_init(struct isc_device *isc)
>>>>>> -{
>>>>>> - struct isc_format *fmt;
>>>>>> - struct v4l2_subdev *subdev = isc->current_subdev->sd;
>>>>>> - unsigned int num_fmts, i, j;
>>>>>> - u32 list_size = isc->formats_list_size;
>>>>>> - struct v4l2_subdev_mbus_code_enum mbus_code = {
>>>>>> - .which = V4L2_SUBDEV_FORMAT_ACTIVE,
>>>>>> - };
>>>>>> -
>>>>>> - num_fmts = 0;
>>>>>> - while (!v4l2_subdev_call(subdev, pad, enum_mbus_code,
>>>>>> - NULL, &mbus_code)) {
>>>>>> - mbus_code.index++;
>>>>>> -
>>>>>> - fmt = isc_find_format_by_code(isc, mbus_code.code, &i);
>>>>>> - if (!fmt) {
>>>>>> - v4l2_warn(&isc->v4l2_dev, "Mbus code %x not supported\n",
>>>>>> - mbus_code.code);
>>>>>> - continue;
>>>>>> - }
>>>>>> -
>>>>>> - fmt->sd_support = true;
>>>>>> - num_fmts++;
>>>>>> - }
>>>>>> -
>>>>>> - if (!num_fmts)
>>>>>> - return -ENXIO;
>>>>>> -
>>>>>> - isc->num_user_formats = num_fmts;
>>>>>> - isc->user_formats = devm_kcalloc(isc->dev,
>>>>>> - num_fmts, sizeof(*isc->user_formats),
>>>>>> - GFP_KERNEL);
>>>>>> - if (!isc->user_formats)
>>>>>> - return -ENOMEM;
>>>>>> -
>>>>>> - fmt = &isc->formats_list[0];
>>>>>> - for (i = 0, j = 0; i < list_size; i++) {
>>>>>> - if (fmt->sd_support)
>>>>>> - isc->user_formats[j++] = fmt;
>>>>>> - fmt++;
>>>>>> - }
>>>>>> -
>>>>>> - return 0;
>>>>>> -}
>>>>>> -
>>>>>> static int isc_set_default_fmt(struct isc_device *isc)
>>>>>> {
>>>>>> struct v4l2_format f = {
>>>>>> @@ -1836,12 +1779,12 @@ static int isc_set_default_fmt(struct isc_device *isc)
>>>>>> .width = VGA_WIDTH,
>>>>>> .height = VGA_HEIGHT,
>>>>>> .field = V4L2_FIELD_NONE,
>>>>>> - .pixelformat = isc->user_formats[0]->fourcc,
>>>>>> + .pixelformat = isc->controller_formats[0].fourcc,
>>>>>> },
>>>>>> };
>>>>>> int ret;
>>>>>>
>>>>>> - ret = isc_try_fmt(isc, &f, NULL);
>>>>>> + ret = isc_try_fmt(isc, &f);
>>>>>> if (ret)
>>>>>> return ret;
>>>>>>
>>>>>> @@ -1896,13 +1839,6 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier)
>>>>>> spin_lock_init(&isc->dma_queue_lock);
>>>>>> spin_lock_init(&isc->awb_lock);
>>>>>>
>>>>>> - ret = isc_formats_init(isc);
>>>>>> - if (ret < 0) {
>>>>>> - v4l2_err(&isc->v4l2_dev,
>>>>>> - "Init format failed: %d\n", ret);
>>>>>> - goto isc_async_complete_err;
>>>>>> - }
>>>>>> -
>>>>>> ret = isc_set_default_fmt(isc);
>>>>>> if (ret) {
>>>>>> v4l2_err(&isc->v4l2_dev, "Could not set default format\n");
>>>>>> @@ -2016,6 +1952,24 @@ int isc_pipeline_init(struct isc_device *isc)
>>>>>> }
>>>>>> EXPORT_SYMBOL_GPL(isc_pipeline_init);
>>>>>>
>>>>>> +static int isc_link_validate(struct media_link *link)
>>>>>> +{
>>>>>> + struct video_device *vdev =
>>>>>> + media_entity_to_video_device(link->sink->entity);
>>>>>> + struct isc_device *isc = video_get_drvdata(vdev);
>>>>>> + int ret;
>>>>>> +
>>>>>> + ret = v4l2_subdev_link_validate(link);
>>>>>> + if (ret)
>>>>>> + return ret;
>>>>>> +
>>>>>> + return isc_validate(isc);
>>>>>> +}
>>>>>> +
>>>>>> +static const struct media_entity_operations isc_entity_operations = {
>>>>>> + .link_validate = isc_link_validate,
>>>>>> +};
>>>>>> +
>>>>>> int isc_mc_init(struct isc_device *isc, u32 ver)
>>>>>> {
>>>>>> const struct of_device_id *match;
>>>>>> @@ -2023,6 +1977,8 @@ int isc_mc_init(struct isc_device *isc, u32 ver)
>>>>>>
>>>>>> isc->video_dev.entity.function = MEDIA_ENT_F_IO_V4L;
>>>>>> isc->video_dev.entity.flags = MEDIA_ENT_FL_DEFAULT;
>>>>>> + isc->video_dev.entity.ops = &isc_entity_operations;
>>>>>> +
>>>>>> isc->pads[ISC_PAD_SINK].flags = MEDIA_PAD_FL_SINK;
>>>>>>
>>>>>> ret = media_entity_pads_init(&isc->video_dev.entity, ISC_PADS_NUM,
>>>>>> diff --git a/drivers/media/platform/atmel/atmel-isc-scaler.c b/drivers/media/platform/atmel/atmel-isc-scaler.c
>>>>>> index a1ca4633787c..2b0621297dfe 100644
>>>>>> --- a/drivers/media/platform/atmel/atmel-isc-scaler.c
>>>>>> +++ b/drivers/media/platform/atmel/atmel-isc-scaler.c
>>>>>> @@ -173,6 +173,10 @@ static const struct v4l2_subdev_pad_ops isc_scaler_pad_ops = {
>>>>>> .init_cfg = isc_scaler_init_cfg,
>>>>>> };
>>>>>>
>>>>>> +static const struct media_entity_operations isc_scaler_entity_ops = {
>>>>>> + .link_validate = v4l2_subdev_link_validate,
>>>>>> +};
>>>>>> +
>>>>>> static const struct v4l2_subdev_ops xisc_scaler_subdev_ops = {
>>>>>> .pad = &isc_scaler_pad_ops,
>>>>>> };
>>>>>> @@ -190,6 +194,7 @@ int isc_scaler_init(struct isc_device *isc)
>>>>>>
>>>>>> isc->scaler_sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
>>>>>> isc->scaler_sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
>>>>>> + isc->scaler_sd.entity.ops = &isc_scaler_entity_ops;
>>>>>> isc->scaler_pads[ISC_SCALER_PAD_SINK].flags = MEDIA_PAD_FL_SINK;
>>>>>> isc->scaler_pads[ISC_SCALER_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
>>>>>>
>>>>>> diff --git a/drivers/media/platform/atmel/atmel-isc.h b/drivers/media/platform/atmel/atmel-isc.h
>>>>>> index f98f25a55e73..5b0100c8c245 100644
>>>>>> --- a/drivers/media/platform/atmel/atmel-isc.h
>>>>>> +++ b/drivers/media/platform/atmel/atmel-isc.h
>>>>>> @@ -63,15 +63,16 @@ struct isc_subdev_entity {
>>>>>> * @cfa_baycfg: If this format is RAW BAYER, indicate the type of bayer.
>>>>>> this is either BGBG, RGRG, etc.
>>>>>> * @pfe_cfg0_bps: Number of hardware data lines connected to the ISC
>>>>>> + * @raw: If the format is raw bayer.
>>>>>> */
>>>>>>
>>>>>> struct isc_format {
>>>>>> u32 fourcc;
>>>>>> u32 mbus_code;
>>>>>> u32 cfa_baycfg;
>>>>>> -
>>>>>> - bool sd_support;
>>>>>> u32 pfe_cfg0_bps;
>>>>>> +
>>>>>> + bool raw;
>>>>>> };
>>>>>>
>>>>>> /* Pipeline bitmap */
>>>>>> @@ -216,8 +217,7 @@ enum isc_scaler_pads {
>>>>>> * @comp: completion reference that signals frame completion
>>>>>> *
>>>>>> * @fmt: current v42l format
>>>>>> - * @user_formats: list of formats that are supported and agreed with sd
>>>>>> - * @num_user_formats: how many formats are in user_formats
>>>>>> + * @try_fmt: current v4l2 try format
>>>>>> *
>>>>>> * @config: current ISC format configuration
>>>>>> * @try_config: the current ISC try format , not yet activated
>>>>>> @@ -272,6 +272,7 @@ enum isc_scaler_pads {
>>>>>> * @formats_list_size: size of formats_list array
>>>>>> * @pads: media controller pads for isc video entity
>>>>>> * @mdev: media device that is registered by the isc
>>>>>> + * @mpipe: media device pipeline used by the isc
>>>>>> * @remote_pad: remote pad on the connected subdevice
>>>>>> * @scaler_sd: subdevice for the scaler that isc registers
>>>>>> * @scaler_pads: media controller pads for the scaler subdevice
>>>>>> @@ -298,8 +299,7 @@ struct isc_device {
>>>>>> struct completion comp;
>>>>>>
>>>>>> struct v4l2_format fmt;
>>>>>> - struct isc_format **user_formats;
>>>>>> - unsigned int num_user_formats;
>>>>>> + struct v4l2_format try_fmt;
>>>>>>
>>>>>> struct fmt_config config;
>>>>>> struct fmt_config try_config;
>>>>>> @@ -369,6 +369,7 @@ struct isc_device {
>>>>>> struct {
>>>>>> struct media_pad pads[ISC_PADS_NUM];
>>>>>> struct media_device mdev;
>>>>>> + struct media_pipeline mpipe;
>>>>>>
>>>>>> u32 remote_pad;
>>>>>> };
>>>>>> diff --git a/drivers/media/platform/atmel/atmel-sama5d2-isc.c b/drivers/media/platform/atmel/atmel-sama5d2-isc.c
>>>>>> index d96ee3373889..2a651e237193 100644
>>>>>> --- a/drivers/media/platform/atmel/atmel-sama5d2-isc.c
>>>>>> +++ b/drivers/media/platform/atmel/atmel-sama5d2-isc.c
>>>>>> @@ -80,20 +80,40 @@ static const struct isc_format sama5d2_controller_formats[] = {
>>>>>> .fourcc = V4L2_PIX_FMT_Y10,
>>>>>> }, {
>>>>>> .fourcc = V4L2_PIX_FMT_SBGGR8,
>>>>>> + .raw = true,
>>>>>> }, {
>>>>>> .fourcc = V4L2_PIX_FMT_SGBRG8,
>>>>>> + .raw = true,
>>>>>> }, {
>>>>>> .fourcc = V4L2_PIX_FMT_SGRBG8,
>>>>>> + .raw = true,
>>>>>> }, {
>>>>>> .fourcc = V4L2_PIX_FMT_SRGGB8,
>>>>>> + .raw = true,
>>>>>> }, {
>>>>>> .fourcc = V4L2_PIX_FMT_SBGGR10,
>>>>>> + .raw = true,
>>>>>> }, {
>>>>>> .fourcc = V4L2_PIX_FMT_SGBRG10,
>>>>>> + .raw = true,
>>>>>> }, {
>>>>>> .fourcc = V4L2_PIX_FMT_SGRBG10,
>>>>>> + .raw = true,
>>>>>> }, {
>>>>>> .fourcc = V4L2_PIX_FMT_SRGGB10,
>>>>>> + .raw = true,
>>>>>> + }, {
>>>>>> + .fourcc = V4L2_PIX_FMT_SBGGR12,
>>>>>> + .raw = true,
>>>>>> + }, {
>>>>>> + .fourcc = V4L2_PIX_FMT_SGBRG12,
>>>>>> + .raw = true,
>>>>>> + }, {
>>>>>> + .fourcc = V4L2_PIX_FMT_SGRBG12,
>>>>>> + .raw = true,
>>>>>> + }, {
>>>>>> + .fourcc = V4L2_PIX_FMT_SRGGB12,
>>>>>> + .raw = true,
>>>>>> },
>>>>>> };
>>>>>>
>>>>>> diff --git a/drivers/media/platform/atmel/atmel-sama7g5-isc.c b/drivers/media/platform/atmel/atmel-sama7g5-isc.c
>>>>>> index e07ae188c15f..a0d60cfdba7b 100644
>>>>>> --- a/drivers/media/platform/atmel/atmel-sama7g5-isc.c
>>>>>> +++ b/drivers/media/platform/atmel/atmel-sama7g5-isc.c
>>>>>> @@ -89,20 +89,40 @@ static const struct isc_format sama7g5_controller_formats[] = {
>>>>>> .fourcc = V4L2_PIX_FMT_Y16,
>>>>>> }, {
>>>>>> .fourcc = V4L2_PIX_FMT_SBGGR8,
>>>>>> + .raw = true,
>>>>>> }, {
>>>>>> .fourcc = V4L2_PIX_FMT_SGBRG8,
>>>>>> + .raw = true,
>>>>>> }, {
>>>>>> .fourcc = V4L2_PIX_FMT_SGRBG8,
>>>>>> + .raw = true,
>>>>>> }, {
>>>>>> .fourcc = V4L2_PIX_FMT_SRGGB8,
>>>>>> + .raw = true,
>>>>>> }, {
>>>>>> .fourcc = V4L2_PIX_FMT_SBGGR10,
>>>>>> + .raw = true,
>>>>>> }, {
>>>>>> .fourcc = V4L2_PIX_FMT_SGBRG10,
>>>>>> + .raw = true,
>>>>>> }, {
>>>>>> .fourcc = V4L2_PIX_FMT_SGRBG10,
>>>>>> + .raw = true,
>>>>>> }, {
>>>>>> .fourcc = V4L2_PIX_FMT_SRGGB10,
>>>>>> + .raw = true,
>>>>>> + }, {
>>>>>> + .fourcc = V4L2_PIX_FMT_SBGGR12,
>>>>>> + .raw = true,
>>>>>> + }, {
>>>>>> + .fourcc = V4L2_PIX_FMT_SGBRG12,
>>>>>> + .raw = true,
>>>>>> + }, {
>>>>>> + .fourcc = V4L2_PIX_FMT_SGRBG12,
>>>>>> + .raw = true,
>>>>>> + }, {
>>>>>> + .fourcc = V4L2_PIX_FMT_SRGGB12,
>>>>>> + .raw = true,
>>>>>> },
>>>>>> };
>>>>>>
>>>>>
>>>>
>>>
>

2022-04-30 11:09:33

[permalink] [raw]

Subject: Re: [PATCH v9 08/13] media: atmel: atmel-isc: change format propagation to subdev into only verification

On Fri, Apr 29, 2022 at 01:17:27PM +0200, Hans Verkuil wrote:
> On 29/04/2022 12:20, Laurent Pinchart wrote:
> > On Fri, Apr 29, 2022 at 12:13:46PM +0200, Hans Verkuil wrote:
> >> On 29/04/2022 12:07, Laurent Pinchart wrote:
> >>> On Fri, Apr 29, 2022 at 11:58:48AM +0200, Jacopo Mondi wrote:
> >>>> On Fri, Apr 29, 2022 at 10:43:09AM +0200, Hans Verkuil wrote:
> >>>>> On 29/04/2022 10:28, [email protected] wrote:
> >>>>>> On 4/29/22 11:17 AM, Hans Verkuil wrote:
> >>>>>>> On 10/03/2022 10:51, Eugen Hristev wrote:
> >>>>>>>> As a top MC video driver, the atmel-isc should not propagate the format to the
> >>>>>>>> subdevice, it should rather check at start_streaming() time if the subdev is properly
> >>>>>>>> configured with a compatible format.
> >>>>>>>> Removed the whole format finding logic, and reworked the format verification
> >>>>>>>> at start_streaming time, such that the ISC will return an error if the subdevice
> >>>>>>>> is not properly configured. To achieve this, media_pipeline_start
> >>>>>>>> is called and a link_validate callback is created to check the formats.
> >>>>>>>> With this being done, the module parameter 'sensor_preferred' makes no sense
> >>>>>>>> anymore. The ISC should not decide which format the sensor is using. The
> >>>>>>>> ISC should only cope with the situation and inform userspace if the streaming
> >>>>>>>> is possible in the current configuration.
> >>>>>>>> The redesign of the format propagation has also risen the question of the
> >>>>>>>> enumfmt callback. If enumfmt is called with an mbus_code, the enumfmt handler
> >>>>>>>> should only return the formats that are supported for this mbus_code.
> >>>>>>>> Otherwise, the enumfmt will report all the formats that the ISC could output.
> >>>>>>>> With this rework, the dynamic list of user formats is removed. It makes no
> >>>>>>>> more sense to identify at complete time which formats the sensor could emit,
> >>>>>>>> and add those into a separate dynamic list.
> >>>>>>>> The ISC will start with a simple preconfigured default format, and at
> >>>>>>>> link validate time, decide whether it can use the format that is configured
> >>>>>>>> on the sink or not.
> >>>>>>>>
> >>>>>>>> Signed-off-by: Eugen Hristev <[email protected]>
> >>>>>>>> Reviewed-by: Jacopo Mondi <[email protected]>
> >>>>>>>> ---
> >>>>>>>> Changes in v9:
> >>>>>>>> - isc_link_validate now static
> >>>>>>>>
> >>>>>>>> Changes in v7:
> >>>>>>>> - minor typos as suggested by Jacopo
> >>>>>>>> - small changes, reduce some indentation, modified an index, as suggested by
> >>>>>>>> Jacopo
> >>>>>>>>
> >>>>>>>> Changes in v6:
> >>>>>>>> - reworked a bit enum_fmt as suggested by Jacopo
> >>>>>>>>
> >>>>>>>> Changes in v5:
> >>>>>>>> - removed user_formats dynamic list as it is now pointless
> >>>>>>>> - greatly simplified the enum_fmt function
> >>>>>>>> - removed some init code that was useless now
> >>>>>>>>
> >>>>>>>> Changes in v4:
> >>>>>>>> - moved validation code into link_validate and used media_pipeline_start
> >>>>>>>> - merged this patch with the enum_fmt patch which was previously in v3 of
> >>>>>>>> the series
> >>>>>>>>
> >>>>>>>> Changes in v3:
> >>>>>>>> - clamp to maximum resolution once the frame size from the subdev is found
> >>>>>>>> drivers/media/platform/atmel/atmel-isc-base.c | 412 ++++++++----------
> >>>>>>>> .../media/platform/atmel/atmel-isc-scaler.c | 5 +
> >>>>>>>> drivers/media/platform/atmel/atmel-isc.h | 13 +-
> >>>>>>>> .../media/platform/atmel/atmel-sama5d2-isc.c | 20 +
> >>>>>>>> .../media/platform/atmel/atmel-sama7g5-isc.c | 20 +
> >>>>>>>> 5 files changed, 236 insertions(+), 234 deletions(-)
> >>>>>>>>
> >>>>>>>> diff --git a/drivers/media/platform/atmel/atmel-isc-base.c b/drivers/media/platform/atmel/atmel-isc-base.c
> >>>>>>>> index ee1dda6707a0..fe2c0af58060 100644
> >>>>>>>> --- a/drivers/media/platform/atmel/atmel-isc-base.c
> >>>>>>>> +++ b/drivers/media/platform/atmel/atmel-isc-base.c
> >>>>>>>> @@ -36,11 +36,6 @@ static unsigned int debug;
> >>>>>>>> module_param(debug, int, 0644);
> >>>>>>>> MODULE_PARM_DESC(debug, "debug level (0-2)");
> >>>>>>>>
> >>>>>>>> -static unsigned int sensor_preferred = 1;
> >>>>>>>> -module_param(sensor_preferred, uint, 0644);
> >>>>>>>> -MODULE_PARM_DESC(sensor_preferred,
> >>>>>>>> - "Sensor is preferred to output the specified format (1-on 0-off), default 1");
> >>>>>>>> -
> >>>>>>>> #define ISC_IS_FORMAT_RAW(mbus_code) \
> >>>>>>>> (((mbus_code) & 0xf000) == 0x3000)
> >>>>>>>>
> >>>>>>>> @@ -337,6 +332,10 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)
> >>>>>>>> unsigned long flags;
> >>>>>>>> int ret;
> >>>>>>>>
> >>>>>>>> + ret = media_pipeline_start(&isc->video_dev.entity, &isc->mpipe);
> >>>>>>>
> >>>>>>> The pipeline validation is done in start_streaming, but I don't think that
> >>>>>>> is the best place: if STREAMON is called before buffers are queued, then
> >>>>>>> an invalid pipeline isn't discovered until enough buffers are queued to
> >>>>>>> kick off start_streaming.
> >>>>>>>
> >>>>>>> Drivers like vsp1, omap3isp and the samsung drivers all do this in streamon().
> >>>>>>>
> >>>>>>> I think that is the correct time to do this.
> >>>>>>
> >>>>>> Hello Hans,
> >>>>>>
> >>>>>> Initially (v2, v3) I had this in streamon(). The problem that I faced at
> >>>>>> that time was that streamoff was never called, so I could not call
> >>>>>> media_pipeline_stop(). Then Jacopo told me to move it to start_streaming
> >>>>>> (see change log for v4) , and I did not face any more problems.
> >>>>
> >>>> Yes indeed, seems I suggested to use media_pipeline_handler in a
> >>>> comment on your v3
> >>>>
> >>>> "at s_stream time your top driver calls media_pipeline_start()"
> >>>>
> >>>> sorry about that, I should have looked around a bit more carefully and
> >>>> notice most drivers do so at vb2 streamon
> >>>>
> >>>> However I don't see media_pipeline_start being called at all in v3 of
> >>>> the patch
> >>>>
> >>>>> It's a mess. Looking at some drivers I see that omap3isp calls media_pipeline_stop
> >>>>> in streamoff (so will have the same problem as you described if VIDIOC_STREAMOFF
> >>>>> isn't called), exynos4-is does the same, but it also checks the streaming state in
> >>>>> the release() fop callback, so that would fix this problem. And vimc does this
> >>>>> in stop_streaming.
> >>>>>
> >>>>> I'm in favor of fixing this in vb2, that framework knows exactly when this needs
> >>>>> to be called.
> >>>>
> >>>> Are you suggesting to have vb2 to call media_pipeline_start() or is it
> >>>> more complex than this ?
> >>>
> >>> I think Hans meant adding a .validate() operation to vb2.
> >>>
> >>> vb2 is already quite complex, I don't think adding more features is a
> >>> good idea. I'd rather have vb2 focus on buffer management only
> >>> (.start_streaming() and .stop_streaming() shouldn't have been in there
> >>> in my opinion), and handle validation in the .streamon() handler. I'd
> >>> expect most drivers that deal with media pipelines to do more work in
> >>> .streamon() anyway.
> >>
> >> I disagree with that :-)
> >>
> >> It's vb2 that keeps track of the streaming state and when what actions
> >> need to be taken. Drivers really shouldn't need to care about the ioctls
> >> themselves, and just implement the relevant vb2 callbacks. Relying on
> >> drivers to handle any of the streaming ioctls is asking for problems,
> >> as this shows: most drivers implement this wrong today.
> >>
> >> The vb2 framework knows when e.g. the pipeline needs to be started or
> >> stopped, and can do this at the best time, without drivers needing to
> >> keep track of when streamon/off/release is called. Keep that logic in
> >> vb2.
> >
> > Pipeline management and buffer management are two different issues.
> > Don't forget about devices that have multiple video nodes, part of the
> > same pipeline (possibly a combination of output and capture nodes, or
> > all of the same type). Forcing drivers to go through vb2 operations to
> > handle the pipeline will be messy, will result in more bloat in vb2, and
> > make the result more bug-prone and harder to maintain.
> >
> > If pipeline management is too complex, let's simplify it, new helpers
> > can make sense, but not through vb2.
> >
>
> But it is vb2 that knows when streaming starts and stops.

That's right, but pipeline start (which includes validation and resource
reservation) needs to be performed synchronously with VIDIOC_STREAMON.
The streaming state managed by vb2 is not relevant,
media_pipeline_start() must not be delayed the same way
.start_streaming() is.

> The driver just
> needs to be informed (e.g. prepare_streaming and unprepare_streaming ops).
>
> vb2 deals with buffer management and it keeps track of the streaming state
> and makes the streaming state transitions. That *is* an integral part of
> vb2. What is missing at the moment are callbacks done at streamon time and
> when the streaming stops (streamoff, or close() when is_streaming is true).
>
> If you want to implement stream validation in a driver, then there are a
> lot of things you need to do:
>
> - override streamon, make sure you call vb2_queue_is_busy(), validate the
> pipeline, then call vb2_streamon, if that fails, remember to stop the
> pipeline.
>
> - override streamoff, make sure you call vb2_queue_is_busy(), stop the
> pipeline and call vb2_streamoff.
>
> - in the release() function when the fh is closed, you have to check
> vb2_is_streaming(), check that you are the owner of the queue, and if true,
> stop the pipeline.

I'm not opposed to helper functions to implement that, they can bundle
vb2 calls and pipeline management.

> By moving this to vb2 ops all you need to implement are the prepare and
> unprepare ops.
>
> Esp. the release() implementation is tricky. I'm pretty sure that
> drivers/media/platform/samsung/exynos4-is/fimc-lite.c is wrong, since it
> should only call media_pipeline_stop() for the owner of the queue. Instead
> it calls it for the last user of the queue.
>
> I see that fimc_lite_streamoff() is wrong too: you can safely call
> VIDIOC_STREAMOFF twice: the second streamoff just returns 0 without
> doing anything. Instead media_pipeline_stop is called without testing if
> the queue is streaming.
>
> And yes, this is in part because V4L2 has quite some history and certainly
> API choice were made in the past that we wouldn't make today. But vb2
> shields you from that, and behaves much more like a proper state machine.
>
> I know you prefer to give a lot more control to driver developers, but
> in my experience very few developers can do things like this right. And
> it is really hard as a reviewer to check if all the corner cases are handled
> correctly in a driver. If vb2 is used, then I know things are called at the
> right time, and that makes my life as reviewer so much easier.

It's not just about giving more control to drivers, it's about
organizing the software layers in a way that keeps them maintainable,
with layered abstractions and not midlayers.

We are extensively reworking the media pipeline management as part of
the stream series, and there will be more work on top of that that will
make even more fundamental changes. I would like to at least postpone
any work on vb2 until then, to be able to evaluate the impact.

> There may still be a few drivers that really need to do this manually, and
> that's OK, but a driver like the atmel-isc doesn't need that at all.

--
Regards,

Laurent Pinchart

2022-04-30 11:33:25

by jacopo mondi

[permalink] [raw]

Subject: Re: [PATCH v9 08/13] media: atmel: atmel-isc: change format propagation to subdev into only verification

Hello

On Fri, Apr 29, 2022 at 10:43:09AM +0200, Hans Verkuil wrote:
> On 29/04/2022 10:28, [email protected] wrote:
> > On 4/29/22 11:17 AM, Hans Verkuil wrote:
> >> On 10/03/2022 10:51, Eugen Hristev wrote:
> >>> As a top MC video driver, the atmel-isc should not propagate the format to the
> >>> subdevice, it should rather check at start_streaming() time if the subdev is properly
> >>> configured with a compatible format.
> >>> Removed the whole format finding logic, and reworked the format verification
> >>> at start_streaming time, such that the ISC will return an error if the subdevice
> >>> is not properly configured. To achieve this, media_pipeline_start
> >>> is called and a link_validate callback is created to check the formats.
> >>> With this being done, the module parameter 'sensor_preferred' makes no sense
> >>> anymore. The ISC should not decide which format the sensor is using. The
> >>> ISC should only cope with the situation and inform userspace if the streaming
> >>> is possible in the current configuration.
> >>> The redesign of the format propagation has also risen the question of the
> >>> enumfmt callback. If enumfmt is called with an mbus_code, the enumfmt handler
> >>> should only return the formats that are supported for this mbus_code.
> >>> Otherwise, the enumfmt will report all the formats that the ISC could output.
> >>> With this rework, the dynamic list of user formats is removed. It makes no
> >>> more sense to identify at complete time which formats the sensor could emit,
> >>> and add those into a separate dynamic list.
> >>> The ISC will start with a simple preconfigured default format, and at
> >>> link validate time, decide whether it can use the format that is configured
> >>> on the sink or not.
> >>>
> >>> Signed-off-by: Eugen Hristev <[email protected]>
> >>> Reviewed-by: Jacopo Mondi <[email protected]>
> >>> ---
> >>> Changes in v9:
> >>> - isc_link_validate now static
> >>>
> >>> Changes in v7:
> >>> - minor typos as suggested by Jacopo
> >>> - small changes, reduce some indentation, modified an index, as suggested by
> >>> Jacopo
> >>>
> >>> Changes in v6:
> >>> - reworked a bit enum_fmt as suggested by Jacopo
> >>>
> >>> Changes in v5:
> >>> - removed user_formats dynamic list as it is now pointless
> >>> - greatly simplified the enum_fmt function
> >>> - removed some init code that was useless now
> >>>
> >>> Changes in v4:
> >>> - moved validation code into link_validate and used media_pipeline_start
> >>> - merged this patch with the enum_fmt patch which was previously in v3 of
> >>> the series
> >>>
> >>> Changes in v3:
> >>> - clamp to maximum resolution once the frame size from the subdev is found
> >>> drivers/media/platform/atmel/atmel-isc-base.c | 412 ++++++++----------
> >>> .../media/platform/atmel/atmel-isc-scaler.c | 5 +
> >>> drivers/media/platform/atmel/atmel-isc.h | 13 +-
> >>> .../media/platform/atmel/atmel-sama5d2-isc.c | 20 +
> >>> .../media/platform/atmel/atmel-sama7g5-isc.c | 20 +
> >>> 5 files changed, 236 insertions(+), 234 deletions(-)
> >>>
> >>> diff --git a/drivers/media/platform/atmel/atmel-isc-base.c b/drivers/media/platform/atmel/atmel-isc-base.c
> >>> index ee1dda6707a0..fe2c0af58060 100644
> >>> --- a/drivers/media/platform/atmel/atmel-isc-base.c
> >>> +++ b/drivers/media/platform/atmel/atmel-isc-base.c
> >>> @@ -36,11 +36,6 @@ static unsigned int debug;
> >>> module_param(debug, int, 0644);
> >>> MODULE_PARM_DESC(debug, "debug level (0-2)");
> >>>
> >>> -static unsigned int sensor_preferred = 1;
> >>> -module_param(sensor_preferred, uint, 0644);
> >>> -MODULE_PARM_DESC(sensor_preferred,
> >>> - "Sensor is preferred to output the specified format (1-on 0-off), default 1");
> >>> -
> >>> #define ISC_IS_FORMAT_RAW(mbus_code) \
> >>> (((mbus_code) & 0xf000) == 0x3000)
> >>>
> >>> @@ -337,6 +332,10 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)
> >>> unsigned long flags;
> >>> int ret;
> >>>
> >>> + ret = media_pipeline_start(&isc->video_dev.entity, &isc->mpipe);
> >>
> >> The pipeline validation is done in start_streaming, but I don't think that
> >> is the best place: if STREAMON is called before buffers are queued, then
> >> an invalid pipeline isn't discovered until enough buffers are queued to
> >> kick off start_streaming.
> >>
> >> Drivers like vsp1, omap3isp and the samsung drivers all do this in streamon().
> >>
> >> I think that is the correct time to do this.
> >
> > Hello Hans,
> >
> > Initially (v2, v3) I had this in streamon(). The problem that I faced at
> > that time was that streamoff was never called, so I could not call
> > media_pipeline_stop(). Then Jacopo told me to move it to start_streaming
> > (see change log for v4) , and I did not face any more problems.

Yes indeed, seems I suggested to use media_pipeline_handler in a
comment on your v3

"at s_stream time your top driver calls media_pipeline_start()"

sorry about that, I should have looked around a bit more carefully and
notice most drivers do so at vb2 streamon

However I don't see media_pipeline_start being called at all in v3 of
the patch

>
> It's a mess. Looking at some drivers I see that omap3isp calls media_pipeline_stop
> in streamoff (so will have the same problem as you described if VIDIOC_STREAMOFF
> isn't called), exynos4-is does the same, but it also checks the streaming state in
> the release() fop callback, so that would fix this problem. And vimc does this
> in stop_streaming.
>
> I'm in favor of fixing this in vb2, that framework knows exactly when this needs
> to be called.

Are you suggesting to have vb2 to call media_pipeline_start() or is it
more complex than this ?

>
> I'd like to know what others think.
>
> Regards,
>
> Hans
>
> >
> > Eugen
> >
> >>
> >> Laurent, Jacopo, do you agree with that?
> >>
> >> I notice that quite a few other drivers do this in start_streaming as well,
> >> but I think it just weird that a QBUF can return EPIPE while STREAMON might
> >> succeed.
> >>
> >> Perhaps we need an optional vb2 validate() callback that is called by
> >> vb2_core_streamon() to validate the pipeline before streaming. That makes
> >> it easier to implement these checks rather than having to implement your
> >> own streamon ioctl.
> >>
> >> Regards,
> >>
> >> Hans
> >>
> >>> + if (ret)
> >>> + goto err_pipeline_start;
> >>> +
> >>> /* Enable stream on the sub device */
> >>> ret = v4l2_subdev_call(isc->current_subdev->sd, video, s_stream, 1);
> >>> if (ret && ret != -ENOIOCTLCMD) {
> >>> @@ -385,6 +384,9 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)
> >>> v4l2_subdev_call(isc->current_subdev->sd, video, s_stream, 0);
> >>>
> >>> err_start_stream:
> >>> + media_pipeline_stop(&isc->video_dev.entity);
> >>> +
> >>> +err_pipeline_start:
> >>> spin_lock_irqsave(&isc->dma_queue_lock, flags);
> >>> list_for_each_entry(buf, &isc->dma_queue, list)
> >>> vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_QUEUED);
> >>> @@ -423,6 +425,9 @@ static void isc_stop_streaming(struct vb2_queue *vq)
> >>> if (ret && ret != -ENOIOCTLCMD)
> >>> v4l2_err(&isc->v4l2_dev, "stream off failed in subdev\n");
> >>>
> >>> + /* Stop media pipeline */
> >>> + media_pipeline_stop(&isc->video_dev.entity);
> >>> +
> >>> /* Release all active buffers */
> >>> spin_lock_irqsave(&isc->dma_queue_lock, flags);
> >>> if (unlikely(isc->cur_frm)) {
> >>> @@ -453,22 +458,6 @@ static void isc_buffer_queue(struct vb2_buffer *vb)
> >>> spin_unlock_irqrestore(&isc->dma_queue_lock, flags);
> >>> }
> >>>
> >>> -static struct isc_format *find_format_by_fourcc(struct isc_device *isc,
> >>> - unsigned int fourcc)
> >>> -{
> >>> - unsigned int num_formats = isc->num_user_formats;
> >>> - struct isc_format *fmt;
> >>> - unsigned int i;
> >>> -
> >>> - for (i = 0; i < num_formats; i++) {
> >>> - fmt = isc->user_formats[i];
> >>> - if (fmt->fourcc == fourcc)
> >>> - return fmt;
> >>> - }
> >>> -
> >>> - return NULL;
> >>> -}
> >>> -
> >>> static const struct vb2_ops isc_vb2_ops = {
> >>> .queue_setup = isc_queue_setup,
> >>> .wait_prepare = vb2_ops_wait_prepare,
> >>> @@ -497,23 +486,57 @@ static int isc_enum_fmt_vid_cap(struct file *file, void *priv,
> >>> {
> >>> struct isc_device *isc = video_drvdata(file);
> >>> u32 index = f->index;
> >>> - u32 i, supported_index;
> >>> + u32 i, supported_index = 0;
> >>> + struct isc_format *fmt;
> >>> +
> >>> + /*
> >>> + * If we are not asked a specific mbus_code, we have to report all
> >>> + * the formats that we can output.
> >>> + */
> >>> + if (!f->mbus_code) {
> >>> + if (index >= isc->controller_formats_size)
> >>> + return -EINVAL;
> >>>
> >>> - if (index < isc->controller_formats_size) {
> >>> f->pixelformat = isc->controller_formats[index].fourcc;
> >>> +
> >>> + return 0;
> >>> + }
> >>> +
> >>> + /*
> >>> + * If a specific mbus_code is requested, check if we support
> >>> + * this mbus_code as input for the ISC.
> >>> + * If it's supported, then we report the corresponding pixelformat
> >>> + * as first possible option for the ISC.
> >>> + * E.g. mbus MEDIA_BUS_FMT_YUYV8_2X8 and report
> >>> + * 'YUYV' (YUYV 4:2:2)
> >>> + */
> >>> + fmt = isc_find_format_by_code(isc, f->mbus_code, &i);
> >>> + if (!fmt)
> >>> + return -EINVAL;
> >>> +
> >>> + if (!index) {
> >>> + f->pixelformat = fmt->fourcc;
> >>> +
> >>> return 0;
> >>> }
> >>>
> >>> - index -= isc->controller_formats_size;
> >>> + supported_index++;
> >>>
> >>> - supported_index = 0;
> >>> + /* If the index is not raw, we don't have anymore formats to report */
> >>> + if (!ISC_IS_FORMAT_RAW(f->mbus_code))
> >>> + return -EINVAL;
> >>>
> >>> - for (i = 0; i < isc->formats_list_size; i++) {
> >>> - if (!ISC_IS_FORMAT_RAW(isc->formats_list[i].mbus_code) ||
> >>> - !isc->formats_list[i].sd_support)
> >>> + /*
> >>> + * We are asked for a specific mbus code, which is raw.
> >>> + * We have to search through the formats we can convert to.
> >>> + * We have to skip the raw formats, we cannot convert to raw.
> >>> + * E.g. 'AR12' (16-bit ARGB 4-4-4-4), 'AR15' (16-bit ARGB 1-5-5-5), etc.
> >>> + */
> >>> + for (i = 0; i < isc->controller_formats_size; i++) {
> >>> + if (isc->controller_formats[i].raw)
> >>> continue;
> >>> - if (supported_index == index) {
> >>> - f->pixelformat = isc->formats_list[i].fourcc;
> >>> + if (index == supported_index) {
> >>> + f->pixelformat = isc->controller_formats[i].fourcc;
> >>> return 0;
> >>> }
> >>> supported_index++;
> >>> @@ -584,20 +607,30 @@ static int isc_try_validate_formats(struct isc_device *isc)
> >>> break;
> >>> default:
> >>> /* any other different formats are not supported */
> >>> + v4l2_err(&isc->v4l2_dev, "Requested unsupported format.\n");
> >>> ret = -EINVAL;
> >>> }
> >>> v4l2_dbg(1, debug, &isc->v4l2_dev,
> >>> "Format validation, requested rgb=%u, yuv=%u, grey=%u, bayer=%u\n",
> >>> rgb, yuv, grey, bayer);
> >>>
> >>> - /* we cannot output RAW if we do not receive RAW */
> >>> - if ((bayer) && !ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code))
> >>> + if (bayer &&
> >>> + !ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code)) {
> >>> + v4l2_err(&isc->v4l2_dev, "Cannot output RAW if we do not receive RAW.\n");
> >>> return -EINVAL;
> >>> + }
> >>>
> >>> - /* we cannot output GREY if we do not receive RAW/GREY */
> >>> if (grey && !ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code) &&
> >>> - !ISC_IS_FORMAT_GREY(isc->try_config.sd_format->mbus_code))
> >>> + !ISC_IS_FORMAT_GREY(isc->try_config.sd_format->mbus_code)) {
> >>> + v4l2_err(&isc->v4l2_dev, "Cannot output GREY if we do not receive RAW/GREY.\n");
> >>> return -EINVAL;
> >>> + }
> >>> +
> >>> + if ((rgb || bayer || yuv) &&
> >>> + ISC_IS_FORMAT_GREY(isc->try_config.sd_format->mbus_code)) {
> >>> + v4l2_err(&isc->v4l2_dev, "Cannot convert GREY to another format.\n");
> >>> + return -EINVAL;
> >>> + }
> >>>
> >>> return ret;
> >>> }
> >>> @@ -825,7 +858,7 @@ static void isc_try_fse(struct isc_device *isc,
> >>> * If we do not know yet which format the subdev is using, we cannot
> >>> * do anything.
> >>> */
> >>> - if (!isc->try_config.sd_format)
> >>> + if (!isc->config.sd_format)
> >>> return;
> >>>
> >>> fse.code = isc->try_config.sd_format->mbus_code;
> >>> @@ -846,180 +879,140 @@ static void isc_try_fse(struct isc_device *isc,
> >>> }
> >>> }
> >>>
> >>> -static int isc_try_fmt(struct isc_device *isc, struct v4l2_format *f,
> >>> - u32 *code)
> >>> +static int isc_try_fmt(struct isc_device *isc, struct v4l2_format *f)
> >>> {
> >>> - int i;
> >>> - struct isc_format *sd_fmt = NULL, *direct_fmt = NULL;
> >>> struct v4l2_pix_format *pixfmt = &f->fmt.pix;
> >>> - struct v4l2_subdev_pad_config pad_cfg = {};
> >>> - struct v4l2_subdev_state pad_state = {
> >>> - .pads = &pad_cfg
> >>> - };
> >>> - struct v4l2_subdev_format format = {
> >>> - .which = V4L2_SUBDEV_FORMAT_TRY,
> >>> - };
> >>> - u32 mbus_code;
> >>> - int ret;
> >>> - bool rlp_dma_direct_dump = false;
> >>> + unsigned int i;
> >>>
> >>> if (f->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
> >>> return -EINVAL;
> >>>
> >>> - /* Step 1: find a RAW format that is supported */
> >>> - for (i = 0; i < isc->num_user_formats; i++) {
> >>> - if (ISC_IS_FORMAT_RAW(isc->user_formats[i]->mbus_code)) {
> >>> - sd_fmt = isc->user_formats[i];
> >>> + isc->try_config.fourcc = isc->controller_formats[0].fourcc;
> >>> +
> >>> + /* find if the format requested is supported */
> >>> + for (i = 0; i < isc->controller_formats_size; i++)
> >>> + if (isc->controller_formats[i].fourcc == pixfmt->pixelformat) {
> >>> + isc->try_config.fourcc = pixfmt->pixelformat;
> >>> break;
> >>> }
> >>> - }
> >>> - /* Step 2: We can continue with this RAW format, or we can look
> >>> - * for better: maybe sensor supports directly what we need.
> >>> - */
> >>> - direct_fmt = find_format_by_fourcc(isc, pixfmt->pixelformat);
> >>> -
> >>> - /* Step 3: We have both. We decide given the module parameter which
> >>> - * one to use.
> >>> - */
> >>> - if (direct_fmt && sd_fmt && sensor_preferred)
> >>> - sd_fmt = direct_fmt;
> >>> -
> >>> - /* Step 4: we do not have RAW but we have a direct format. Use it. */
> >>> - if (direct_fmt && !sd_fmt)
> >>> - sd_fmt = direct_fmt;
> >>> -
> >>> - /* Step 5: if we are using a direct format, we need to package
> >>> - * everything as 8 bit data and just dump it
> >>> - */
> >>> - if (sd_fmt == direct_fmt)
> >>> - rlp_dma_direct_dump = true;
> >>> -
> >>> - /* Step 6: We have no format. This can happen if the userspace
> >>> - * requests some weird/invalid format.
> >>> - * In this case, default to whatever we have
> >>> - */
> >>> - if (!sd_fmt && !direct_fmt) {
> >>> - sd_fmt = isc->user_formats[isc->num_user_formats - 1];
> >>> - v4l2_dbg(1, debug, &isc->v4l2_dev,
> >>> - "Sensor not supporting %.4s, using %.4s\n",
> >>> - (char *)&pixfmt->pixelformat, (char *)&sd_fmt->fourcc);
> >>> - }
> >>> -
> >>> - if (!sd_fmt) {
> >>> - ret = -EINVAL;
> >>> - goto isc_try_fmt_err;
> >>> - }
> >>> -
> >>> - /* Step 7: Print out what we decided for debugging */
> >>> - v4l2_dbg(1, debug, &isc->v4l2_dev,
> >>> - "Preferring to have sensor using format %.4s\n",
> >>> - (char *)&sd_fmt->fourcc);
> >>>
> >>> - /* Step 8: at this moment we decided which format the subdev will use */
> >>> - isc->try_config.sd_format = sd_fmt;
> >>> + isc_try_configure_rlp_dma(isc, false);
> >>>
> >>> /* Limit to Atmel ISC hardware capabilities */
> >>> - if (pixfmt->width > isc->max_width)
> >>> - pixfmt->width = isc->max_width;
> >>> - if (pixfmt->height > isc->max_height)
> >>> - pixfmt->height = isc->max_height;
> >>> -
> >>> - /*
> >>> - * The mbus format is the one the subdev outputs.
> >>> - * The pixels will be transferred in this format Sensor -> ISC
> >>> - */
> >>> - mbus_code = sd_fmt->mbus_code;
> >>> -
> >>> - /*
> >>> - * Validate formats. If the required format is not OK, default to raw.
> >>> - */
> >>> -
> >>> - isc->try_config.fourcc = pixfmt->pixelformat;
> >>> -
> >>> - if (isc_try_validate_formats(isc)) {
> >>> - pixfmt->pixelformat = isc->try_config.fourcc = sd_fmt->fourcc;
> >>> - /* Re-try to validate the new format */
> >>> - ret = isc_try_validate_formats(isc);
> >>> - if (ret)
> >>> - goto isc_try_fmt_err;
> >>> - }
> >>> -
> >>> - ret = isc_try_configure_rlp_dma(isc, rlp_dma_direct_dump);
> >>> - if (ret)
> >>> - goto isc_try_fmt_err;
> >>> -
> >>> - ret = isc_try_configure_pipeline(isc);
> >>> - if (ret)
> >>> - goto isc_try_fmt_err;
> >>> -
> >>> - /* Obtain frame sizes if possible to have crop requirements ready */
> >>> - isc_try_fse(isc, &pad_state);
> >>> -
> >>> - v4l2_fill_mbus_format(&format.format, pixfmt, mbus_code);
> >>> - ret = v4l2_subdev_call(isc->current_subdev->sd, pad, set_fmt,
> >>> - &pad_state, &format);
> >>> - if (ret < 0)
> >>> - goto isc_try_fmt_subdev_err;
> >>> -
> >>> - v4l2_fill_pix_format(pixfmt, &format.format);
> >>> + v4l_bound_align_image(&pixfmt->width, 16, isc->max_width, 0,
> >>> + &pixfmt->height, 16, isc->max_height, 0, 0);
> >>> + /* If we did not find the requested format, we will fallback here */
> >>> + pixfmt->pixelformat = isc->try_config.fourcc;
> >>> + pixfmt->colorspace = V4L2_COLORSPACE_SRGB;
> >>> + pixfmt->field = V4L2_FIELD_NONE;
> >>>
> >>> - /* Limit to Atmel ISC hardware capabilities */
> >>> - if (pixfmt->width > isc->max_width)
> >>> - pixfmt->width = isc->max_width;
> >>> - if (pixfmt->height > isc->max_height)
> >>> - pixfmt->height = isc->max_height;
> >>>
> >>> - pixfmt->field = V4L2_FIELD_NONE;
> >>> pixfmt->bytesperline = (pixfmt->width * isc->try_config.bpp_v4l2) >> 3;
> >>> pixfmt->sizeimage = ((pixfmt->width * isc->try_config.bpp) >> 3) *
> >>> pixfmt->height;
> >>>
> >>> - if (code)
> >>> - *code = mbus_code;
> >>> + isc->try_fmt = *f;
> >>>
> >>> return 0;
> >>> +}
> >>>
> >>> -isc_try_fmt_err:
> >>> - v4l2_err(&isc->v4l2_dev, "Could not find any possible format for a working pipeline\n");
> >>> -isc_try_fmt_subdev_err:
> >>> - memset(&isc->try_config, 0, sizeof(isc->try_config));
> >>> +static int isc_set_fmt(struct isc_device *isc, struct v4l2_format *f)
> >>> +{
> >>> + isc_try_fmt(isc, f);
> >>>
> >>> - return ret;
> >>> + /* make the try configuration active */
> >>> + isc->config = isc->try_config;
> >>> + isc->fmt = isc->try_fmt;
> >>> +
> >>> + v4l2_dbg(1, debug, &isc->v4l2_dev, "ISC set_fmt to %.4s @%dx%d\n",
> >>> + (char *)&f->fmt.pix.pixelformat,
> >>> + f->fmt.pix.width, f->fmt.pix.height);
> >>> +
> >>> + return 0;
> >>> }
> >>>
> >>> -static int isc_set_fmt(struct isc_device *isc, struct v4l2_format *f)
> >>> +static int isc_validate(struct isc_device *isc)
> >>> {
> >>> + int ret;
> >>> + int i;
> >>> + struct isc_format *sd_fmt = NULL;
> >>> + struct v4l2_pix_format *pixfmt = &isc->fmt.fmt.pix;
> >>> struct v4l2_subdev_format format = {
> >>> .which = V4L2_SUBDEV_FORMAT_ACTIVE,
> >>> + .pad = isc->remote_pad,
> >>> + };
> >>> + struct v4l2_subdev_pad_config pad_cfg = {};
> >>> + struct v4l2_subdev_state pad_state = {
> >>> + .pads = &pad_cfg,
> >>> };
> >>> - u32 mbus_code = 0;
> >>> - int ret;
> >>>
> >>> - ret = isc_try_fmt(isc, f, &mbus_code);
> >>> + /* Get current format from subdev */
> >>> + ret = v4l2_subdev_call(isc->current_subdev->sd, pad, get_fmt, NULL,
> >>> + &format);
> >>> if (ret)
> >>> return ret;
> >>>
> >>> - v4l2_fill_mbus_format(&format.format, &f->fmt.pix, mbus_code);
> >>> - ret = v4l2_subdev_call(isc->current_subdev->sd, pad,
> >>> - set_fmt, NULL, &format);
> >>> - if (ret < 0)
> >>> - return ret;
> >>> + /* Identify the subdev's format configuration */
> >>> + for (i = 0; i < isc->formats_list_size; i++)
> >>> + if (isc->formats_list[i].mbus_code == format.format.code) {
> >>> + sd_fmt = &isc->formats_list[i];
> >>> + break;
> >>> + }
> >>> +
> >>> + /* Check if the format is not supported */
> >>> + if (!sd_fmt) {
> >>> + v4l2_err(&isc->v4l2_dev,
> >>> + "Current subdevice is streaming a media bus code that is not supported 0x%x\n",
> >>> + format.format.code);
> >>> + return -EPIPE;
> >>> + }
> >>> +
> >>> + /* At this moment we know which format the subdev will use */
> >>> + isc->try_config.sd_format = sd_fmt;
> >>> +
> >>> + /* If the sensor is not RAW, we can only do a direct dump */
> >>> + if (!ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code))
> >>> + isc_try_configure_rlp_dma(isc, true);
> >>>
> >>> /* Limit to Atmel ISC hardware capabilities */
> >>> - if (f->fmt.pix.width > isc->max_width)
> >>> - f->fmt.pix.width = isc->max_width;
> >>> - if (f->fmt.pix.height > isc->max_height)
> >>> - f->fmt.pix.height = isc->max_height;
> >>> + v4l_bound_align_image(&format.format.width, 16, isc->max_width, 0,
> >>> + &format.format.height, 16, isc->max_height, 0, 0);
> >>>
> >>> - isc->fmt = *f;
> >>> + /* Check if the frame size is the same. Otherwise we may overflow */
> >>> + if (pixfmt->height != format.format.height ||
> >>> + pixfmt->width != format.format.width) {
> >>> + v4l2_err(&isc->v4l2_dev,
> >>> + "ISC not configured with the proper frame size: %dx%d\n",
> >>> + format.format.width, format.format.height);
> >>> + return -EPIPE;
> >>> + }
> >>>
> >>> + v4l2_dbg(1, debug, &isc->v4l2_dev,
> >>> + "Identified subdev using format %.4s with %dx%d %d bpp\n",
> >>> + (char *)&sd_fmt->fourcc, pixfmt->width, pixfmt->height,
> >>> + isc->try_config.bpp);
> >>> +
> >>> + /* Reset and restart AWB if the subdevice changed the format */
> >>> if (isc->try_config.sd_format && isc->config.sd_format &&
> >>> isc->try_config.sd_format != isc->config.sd_format) {
> >>> isc->ctrls.hist_stat = HIST_INIT;
> >>> isc_reset_awb_ctrls(isc);
> >>> isc_update_v4l2_ctrls(isc);
> >>> }
> >>> - /* make the try configuration active */
> >>> +
> >>> + /* Validate formats */
> >>> + ret = isc_try_validate_formats(isc);
> >>> + if (ret)
> >>> + return ret;
> >>> +
> >>> + /* Obtain frame sizes if possible to have crop requirements ready */
> >>> + isc_try_fse(isc, &pad_state);
> >>> +
> >>> + /* Configure ISC pipeline for the config */
> >>> + ret = isc_try_configure_pipeline(isc);
> >>> + if (ret)
> >>> + return ret;
> >>> +
> >>> isc->config = isc->try_config;
> >>>
> >>> v4l2_dbg(1, debug, &isc->v4l2_dev, "New ISC configuration in place\n");
> >>> @@ -1043,7 +1036,7 @@ static int isc_try_fmt_vid_cap(struct file *file, void *priv,
> >>> {
> >>> struct isc_device *isc = video_drvdata(file);
> >>>
> >>> - return isc_try_fmt(isc, f, NULL);
> >>> + return isc_try_fmt(isc, f);
> >>> }
> >>>
> >>> static int isc_enum_input(struct file *file, void *priv,
> >>> @@ -1098,10 +1091,6 @@ static int isc_enum_framesizes(struct file *file, void *fh,
> >>> if (fsize->index)
> >>> return -EINVAL;
> >>>
> >>> - for (i = 0; i < isc->num_user_formats; i++)
> >>> - if (isc->user_formats[i]->fourcc == fsize->pixel_format)
> >>> - ret = 0;
> >>> -
> >>> for (i = 0; i < isc->controller_formats_size; i++)
> >>> if (isc->controller_formats[i].fourcc == fsize->pixel_format)
> >>> ret = 0;
> >>> @@ -1782,52 +1771,6 @@ struct isc_format *isc_find_format_by_code(struct isc_device *isc,
> >>> }
> >>> EXPORT_SYMBOL_GPL(isc_find_format_by_code);
> >>>
> >>> -static int isc_formats_init(struct isc_device *isc)
> >>> -{
> >>> - struct isc_format *fmt;
> >>> - struct v4l2_subdev *subdev = isc->current_subdev->sd;
> >>> - unsigned int num_fmts, i, j;
> >>> - u32 list_size = isc->formats_list_size;
> >>> - struct v4l2_subdev_mbus_code_enum mbus_code = {
> >>> - .which = V4L2_SUBDEV_FORMAT_ACTIVE,
> >>> - };
> >>> -
> >>> - num_fmts = 0;
> >>> - while (!v4l2_subdev_call(subdev, pad, enum_mbus_code,
> >>> - NULL, &mbus_code)) {
> >>> - mbus_code.index++;
> >>> -
> >>> - fmt = isc_find_format_by_code(isc, mbus_code.code, &i);
> >>> - if (!fmt) {
> >>> - v4l2_warn(&isc->v4l2_dev, "Mbus code %x not supported\n",
> >>> - mbus_code.code);
> >>> - continue;
> >>> - }
> >>> -
> >>> - fmt->sd_support = true;
> >>> - num_fmts++;
> >>> - }
> >>> -
> >>> - if (!num_fmts)
> >>> - return -ENXIO;
> >>> -
> >>> - isc->num_user_formats = num_fmts;
> >>> - isc->user_formats = devm_kcalloc(isc->dev,
> >>> - num_fmts, sizeof(*isc->user_formats),
> >>> - GFP_KERNEL);
> >>> - if (!isc->user_formats)
> >>> - return -ENOMEM;
> >>> -
> >>> - fmt = &isc->formats_list[0];
> >>> - for (i = 0, j = 0; i < list_size; i++) {
> >>> - if (fmt->sd_support)
> >>> - isc->user_formats[j++] = fmt;
> >>> - fmt++;
> >>> - }
> >>> -
> >>> - return 0;
> >>> -}
> >>> -
> >>> static int isc_set_default_fmt(struct isc_device *isc)
> >>> {
> >>> struct v4l2_format f = {
> >>> @@ -1836,12 +1779,12 @@ static int isc_set_default_fmt(struct isc_device *isc)
> >>> .width = VGA_WIDTH,
> >>> .height = VGA_HEIGHT,
> >>> .field = V4L2_FIELD_NONE,
> >>> - .pixelformat = isc->user_formats[0]->fourcc,
> >>> + .pixelformat = isc->controller_formats[0].fourcc,
> >>> },
> >>> };
> >>> int ret;
> >>>
> >>> - ret = isc_try_fmt(isc, &f, NULL);
> >>> + ret = isc_try_fmt(isc, &f);
> >>> if (ret)
> >>> return ret;
> >>>
> >>> @@ -1896,13 +1839,6 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier)
> >>> spin_lock_init(&isc->dma_queue_lock);
> >>> spin_lock_init(&isc->awb_lock);
> >>>
> >>> - ret = isc_formats_init(isc);
> >>> - if (ret < 0) {
> >>> - v4l2_err(&isc->v4l2_dev,
> >>> - "Init format failed: %d\n", ret);
> >>> - goto isc_async_complete_err;
> >>> - }
> >>> -
> >>> ret = isc_set_default_fmt(isc);
> >>> if (ret) {
> >>> v4l2_err(&isc->v4l2_dev, "Could not set default format\n");
> >>> @@ -2016,6 +1952,24 @@ int isc_pipeline_init(struct isc_device *isc)
> >>> }
> >>> EXPORT_SYMBOL_GPL(isc_pipeline_init);
> >>>
> >>> +static int isc_link_validate(struct media_link *link)
> >>> +{
> >>> + struct video_device *vdev =
> >>> + media_entity_to_video_device(link->sink->entity);
> >>> + struct isc_device *isc = video_get_drvdata(vdev);
> >>> + int ret;
> >>> +
> >>> + ret = v4l2_subdev_link_validate(link);
> >>> + if (ret)
> >>> + return ret;
> >>> +
> >>> + return isc_validate(isc);
> >>> +}
> >>> +
> >>> +static const struct media_entity_operations isc_entity_operations = {
> >>> + .link_validate = isc_link_validate,
> >>> +};
> >>> +
> >>> int isc_mc_init(struct isc_device *isc, u32 ver)
> >>> {
> >>> const struct of_device_id *match;
> >>> @@ -2023,6 +1977,8 @@ int isc_mc_init(struct isc_device *isc, u32 ver)
> >>>
> >>> isc->video_dev.entity.function = MEDIA_ENT_F_IO_V4L;
> >>> isc->video_dev.entity.flags = MEDIA_ENT_FL_DEFAULT;
> >>> + isc->video_dev.entity.ops = &isc_entity_operations;
> >>> +
> >>> isc->pads[ISC_PAD_SINK].flags = MEDIA_PAD_FL_SINK;
> >>>
> >>> ret = media_entity_pads_init(&isc->video_dev.entity, ISC_PADS_NUM,
> >>> diff --git a/drivers/media/platform/atmel/atmel-isc-scaler.c b/drivers/media/platform/atmel/atmel-isc-scaler.c
> >>> index a1ca4633787c..2b0621297dfe 100644
> >>> --- a/drivers/media/platform/atmel/atmel-isc-scaler.c
> >>> +++ b/drivers/media/platform/atmel/atmel-isc-scaler.c
> >>> @@ -173,6 +173,10 @@ static const struct v4l2_subdev_pad_ops isc_scaler_pad_ops = {
> >>> .init_cfg = isc_scaler_init_cfg,
> >>> };
> >>>
> >>> +static const struct media_entity_operations isc_scaler_entity_ops = {
> >>> + .link_validate = v4l2_subdev_link_validate,
> >>> +};
> >>> +
> >>> static const struct v4l2_subdev_ops xisc_scaler_subdev_ops = {
> >>> .pad = &isc_scaler_pad_ops,
> >>> };
> >>> @@ -190,6 +194,7 @@ int isc_scaler_init(struct isc_device *isc)
> >>>
> >>> isc->scaler_sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
> >>> isc->scaler_sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
> >>> + isc->scaler_sd.entity.ops = &isc_scaler_entity_ops;
> >>> isc->scaler_pads[ISC_SCALER_PAD_SINK].flags = MEDIA_PAD_FL_SINK;
> >>> isc->scaler_pads[ISC_SCALER_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
> >>>
> >>> diff --git a/drivers/media/platform/atmel/atmel-isc.h b/drivers/media/platform/atmel/atmel-isc.h
> >>> index f98f25a55e73..5b0100c8c245 100644
> >>> --- a/drivers/media/platform/atmel/atmel-isc.h
> >>> +++ b/drivers/media/platform/atmel/atmel-isc.h
> >>> @@ -63,15 +63,16 @@ struct isc_subdev_entity {
> >>> * @cfa_baycfg: If this format is RAW BAYER, indicate the type of bayer.
> >>> this is either BGBG, RGRG, etc.
> >>> * @pfe_cfg0_bps: Number of hardware data lines connected to the ISC
> >>> + * @raw: If the format is raw bayer.
> >>> */
> >>>
> >>> struct isc_format {
> >>> u32 fourcc;
> >>> u32 mbus_code;
> >>> u32 cfa_baycfg;
> >>> -
> >>> - bool sd_support;
> >>> u32 pfe_cfg0_bps;
> >>> +
> >>> + bool raw;
> >>> };
> >>>
> >>> /* Pipeline bitmap */
> >>> @@ -216,8 +217,7 @@ enum isc_scaler_pads {
> >>> * @comp: completion reference that signals frame completion
> >>> *
> >>> * @fmt: current v42l format
> >>> - * @user_formats: list of formats that are supported and agreed with sd
> >>> - * @num_user_formats: how many formats are in user_formats
> >>> + * @try_fmt: current v4l2 try format
> >>> *
> >>> * @config: current ISC format configuration
> >>> * @try_config: the current ISC try format , not yet activated
> >>> @@ -272,6 +272,7 @@ enum isc_scaler_pads {
> >>> * @formats_list_size: size of formats_list array
> >>> * @pads: media controller pads for isc video entity
> >>> * @mdev: media device that is registered by the isc
> >>> + * @mpipe: media device pipeline used by the isc
> >>> * @remote_pad: remote pad on the connected subdevice
> >>> * @scaler_sd: subdevice for the scaler that isc registers
> >>> * @scaler_pads: media controller pads for the scaler subdevice
> >>> @@ -298,8 +299,7 @@ struct isc_device {
> >>> struct completion comp;
> >>>
> >>> struct v4l2_format fmt;
> >>> - struct isc_format **user_formats;
> >>> - unsigned int num_user_formats;
> >>> + struct v4l2_format try_fmt;
> >>>
> >>> struct fmt_config config;
> >>> struct fmt_config try_config;
> >>> @@ -369,6 +369,7 @@ struct isc_device {
> >>> struct {
> >>> struct media_pad pads[ISC_PADS_NUM];
> >>> struct media_device mdev;
> >>> + struct media_pipeline mpipe;
> >>>
> >>> u32 remote_pad;
> >>> };
> >>> diff --git a/drivers/media/platform/atmel/atmel-sama5d2-isc.c b/drivers/media/platform/atmel/atmel-sama5d2-isc.c
> >>> index d96ee3373889..2a651e237193 100644
> >>> --- a/drivers/media/platform/atmel/atmel-sama5d2-isc.c
> >>> +++ b/drivers/media/platform/atmel/atmel-sama5d2-isc.c
> >>> @@ -80,20 +80,40 @@ static const struct isc_format sama5d2_controller_formats[] = {
> >>> .fourcc = V4L2_PIX_FMT_Y10,
> >>> }, {
> >>> .fourcc = V4L2_PIX_FMT_SBGGR8,
> >>> + .raw = true,
> >>> }, {
> >>> .fourcc = V4L2_PIX_FMT_SGBRG8,
> >>> + .raw = true,
> >>> }, {
> >>> .fourcc = V4L2_PIX_FMT_SGRBG8,
> >>> + .raw = true,
> >>> }, {
> >>> .fourcc = V4L2_PIX_FMT_SRGGB8,
> >>> + .raw = true,
> >>> }, {
> >>> .fourcc = V4L2_PIX_FMT_SBGGR10,
> >>> + .raw = true,
> >>> }, {
> >>> .fourcc = V4L2_PIX_FMT_SGBRG10,
> >>> + .raw = true,
> >>> }, {
> >>> .fourcc = V4L2_PIX_FMT_SGRBG10,
> >>> + .raw = true,
> >>> }, {
> >>> .fourcc = V4L2_PIX_FMT_SRGGB10,
> >>> + .raw = true,
> >>> + }, {
> >>> + .fourcc = V4L2_PIX_FMT_SBGGR12,
> >>> + .raw = true,
> >>> + }, {
> >>> + .fourcc = V4L2_PIX_FMT_SGBRG12,
> >>> + .raw = true,
> >>> + }, {
> >>> + .fourcc = V4L2_PIX_FMT_SGRBG12,
> >>> + .raw = true,
> >>> + }, {
> >>> + .fourcc = V4L2_PIX_FMT_SRGGB12,
> >>> + .raw = true,
> >>> },
> >>> };
> >>>
> >>> diff --git a/drivers/media/platform/atmel/atmel-sama7g5-isc.c b/drivers/media/platform/atmel/atmel-sama7g5-isc.c
> >>> index e07ae188c15f..a0d60cfdba7b 100644
> >>> --- a/drivers/media/platform/atmel/atmel-sama7g5-isc.c
> >>> +++ b/drivers/media/platform/atmel/atmel-sama7g5-isc.c
> >>> @@ -89,20 +89,40 @@ static const struct isc_format sama7g5_controller_formats[] = {
> >>> .fourcc = V4L2_PIX_FMT_Y16,
> >>> }, {
> >>> .fourcc = V4L2_PIX_FMT_SBGGR8,
> >>> + .raw = true,
> >>> }, {
> >>> .fourcc = V4L2_PIX_FMT_SGBRG8,
> >>> + .raw = true,
> >>> }, {
> >>> .fourcc = V4L2_PIX_FMT_SGRBG8,
> >>> + .raw = true,
> >>> }, {
> >>> .fourcc = V4L2_PIX_FMT_SRGGB8,
> >>> + .raw = true,
> >>> }, {
> >>> .fourcc = V4L2_PIX_FMT_SBGGR10,
> >>> + .raw = true,
> >>> }, {
> >>> .fourcc = V4L2_PIX_FMT_SGBRG10,
> >>> + .raw = true,
> >>> }, {
> >>> .fourcc = V4L2_PIX_FMT_SGRBG10,
> >>> + .raw = true,
> >>> }, {
> >>> .fourcc = V4L2_PIX_FMT_SRGGB10,
> >>> + .raw = true,
> >>> + }, {
> >>> + .fourcc = V4L2_PIX_FMT_SBGGR12,
> >>> + .raw = true,
> >>> + }, {
> >>> + .fourcc = V4L2_PIX_FMT_SGBRG12,
> >>> + .raw = true,
> >>> + }, {
> >>> + .fourcc = V4L2_PIX_FMT_SGRBG12,
> >>> + .raw = true,
> >>> + }, {
> >>> + .fourcc = V4L2_PIX_FMT_SRGGB12,
> >>> + .raw = true,
> >>> },
> >>> };
> >>>
> >>
> >
>

2022-04-30 18:58:37

[permalink] [raw]

Subject: Re: [PATCH v9 08/13] media: atmel: atmel-isc: change format propagation to subdev into only verification

On 29/04/2022 14:02, Laurent Pinchart wrote:
> On Fri, Apr 29, 2022 at 01:17:27PM +0200, Hans Verkuil wrote:
>> On 29/04/2022 12:20, Laurent Pinchart wrote:
>>> On Fri, Apr 29, 2022 at 12:13:46PM +0200, Hans Verkuil wrote:
>>>> On 29/04/2022 12:07, Laurent Pinchart wrote:
>>>>> On Fri, Apr 29, 2022 at 11:58:48AM +0200, Jacopo Mondi wrote:
>>>>>> On Fri, Apr 29, 2022 at 10:43:09AM +0200, Hans Verkuil wrote:
>>>>>>> On 29/04/2022 10:28, [email protected] wrote:
>>>>>>>> On 4/29/22 11:17 AM, Hans Verkuil wrote:
>>>>>>>>> On 10/03/2022 10:51, Eugen Hristev wrote:
>>>>>>>>>> As a top MC video driver, the atmel-isc should not propagate the format to the
>>>>>>>>>> subdevice, it should rather check at start_streaming() time if the subdev is properly
>>>>>>>>>> configured with a compatible format.
>>>>>>>>>> Removed the whole format finding logic, and reworked the format verification
>>>>>>>>>> at start_streaming time, such that the ISC will return an error if the subdevice
>>>>>>>>>> is not properly configured. To achieve this, media_pipeline_start
>>>>>>>>>> is called and a link_validate callback is created to check the formats.
>>>>>>>>>> With this being done, the module parameter 'sensor_preferred' makes no sense
>>>>>>>>>> anymore. The ISC should not decide which format the sensor is using. The
>>>>>>>>>> ISC should only cope with the situation and inform userspace if the streaming
>>>>>>>>>> is possible in the current configuration.
>>>>>>>>>> The redesign of the format propagation has also risen the question of the
>>>>>>>>>> enumfmt callback. If enumfmt is called with an mbus_code, the enumfmt handler
>>>>>>>>>> should only return the formats that are supported for this mbus_code.
>>>>>>>>>> Otherwise, the enumfmt will report all the formats that the ISC could output.
>>>>>>>>>> With this rework, the dynamic list of user formats is removed. It makes no
>>>>>>>>>> more sense to identify at complete time which formats the sensor could emit,
>>>>>>>>>> and add those into a separate dynamic list.
>>>>>>>>>> The ISC will start with a simple preconfigured default format, and at
>>>>>>>>>> link validate time, decide whether it can use the format that is configured
>>>>>>>>>> on the sink or not.
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Eugen Hristev <[email protected]>
>>>>>>>>>> Reviewed-by: Jacopo Mondi <[email protected]>
>>>>>>>>>> ---
>>>>>>>>>> Changes in v9:
>>>>>>>>>> - isc_link_validate now static
>>>>>>>>>>
>>>>>>>>>> Changes in v7:
>>>>>>>>>> - minor typos as suggested by Jacopo
>>>>>>>>>> - small changes, reduce some indentation, modified an index, as suggested by
>>>>>>>>>> Jacopo
>>>>>>>>>>
>>>>>>>>>> Changes in v6:
>>>>>>>>>> - reworked a bit enum_fmt as suggested by Jacopo
>>>>>>>>>>
>>>>>>>>>> Changes in v5:
>>>>>>>>>> - removed user_formats dynamic list as it is now pointless
>>>>>>>>>> - greatly simplified the enum_fmt function
>>>>>>>>>> - removed some init code that was useless now
>>>>>>>>>>
>>>>>>>>>> Changes in v4:
>>>>>>>>>> - moved validation code into link_validate and used media_pipeline_start
>>>>>>>>>> - merged this patch with the enum_fmt patch which was previously in v3 of
>>>>>>>>>> the series
>>>>>>>>>>
>>>>>>>>>> Changes in v3:
>>>>>>>>>> - clamp to maximum resolution once the frame size from the subdev is found
>>>>>>>>>> drivers/media/platform/atmel/atmel-isc-base.c | 412 ++++++++----------
>>>>>>>>>> .../media/platform/atmel/atmel-isc-scaler.c | 5 +
>>>>>>>>>> drivers/media/platform/atmel/atmel-isc.h | 13 +-
>>>>>>>>>> .../media/platform/atmel/atmel-sama5d2-isc.c | 20 +
>>>>>>>>>> .../media/platform/atmel/atmel-sama7g5-isc.c | 20 +
>>>>>>>>>> 5 files changed, 236 insertions(+), 234 deletions(-)
>>>>>>>>>>
>>>>>>>>>> diff --git a/drivers/media/platform/atmel/atmel-isc-base.c b/drivers/media/platform/atmel/atmel-isc-base.c
>>>>>>>>>> index ee1dda6707a0..fe2c0af58060 100644
>>>>>>>>>> --- a/drivers/media/platform/atmel/atmel-isc-base.c
>>>>>>>>>> +++ b/drivers/media/platform/atmel/atmel-isc-base.c
>>>>>>>>>> @@ -36,11 +36,6 @@ static unsigned int debug;
>>>>>>>>>> module_param(debug, int, 0644);
>>>>>>>>>> MODULE_PARM_DESC(debug, "debug level (0-2)");
>>>>>>>>>>
>>>>>>>>>> -static unsigned int sensor_preferred = 1;
>>>>>>>>>> -module_param(sensor_preferred, uint, 0644);
>>>>>>>>>> -MODULE_PARM_DESC(sensor_preferred,
>>>>>>>>>> - "Sensor is preferred to output the specified format (1-on 0-off), default 1");
>>>>>>>>>> -
>>>>>>>>>> #define ISC_IS_FORMAT_RAW(mbus_code) \
>>>>>>>>>> (((mbus_code) & 0xf000) == 0x3000)
>>>>>>>>>>
>>>>>>>>>> @@ -337,6 +332,10 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)
>>>>>>>>>> unsigned long flags;
>>>>>>>>>> int ret;
>>>>>>>>>>
>>>>>>>>>> + ret = media_pipeline_start(&isc->video_dev.entity, &isc->mpipe);
>>>>>>>>>
>>>>>>>>> The pipeline validation is done in start_streaming, but I don't think that
>>>>>>>>> is the best place: if STREAMON is called before buffers are queued, then
>>>>>>>>> an invalid pipeline isn't discovered until enough buffers are queued to
>>>>>>>>> kick off start_streaming.
>>>>>>>>>
>>>>>>>>> Drivers like vsp1, omap3isp and the samsung drivers all do this in streamon().
>>>>>>>>>
>>>>>>>>> I think that is the correct time to do this.
>>>>>>>>
>>>>>>>> Hello Hans,
>>>>>>>>
>>>>>>>> Initially (v2, v3) I had this in streamon(). The problem that I faced at
>>>>>>>> that time was that streamoff was never called, so I could not call
>>>>>>>> media_pipeline_stop(). Then Jacopo told me to move it to start_streaming
>>>>>>>> (see change log for v4) , and I did not face any more problems.
>>>>>>
>>>>>> Yes indeed, seems I suggested to use media_pipeline_handler in a
>>>>>> comment on your v3
>>>>>>
>>>>>> "at s_stream time your top driver calls media_pipeline_start()"
>>>>>>
>>>>>> sorry about that, I should have looked around a bit more carefully and
>>>>>> notice most drivers do so at vb2 streamon
>>>>>>
>>>>>> However I don't see media_pipeline_start being called at all in v3 of
>>>>>> the patch
>>>>>>
>>>>>>> It's a mess. Looking at some drivers I see that omap3isp calls media_pipeline_stop
>>>>>>> in streamoff (so will have the same problem as you described if VIDIOC_STREAMOFF
>>>>>>> isn't called), exynos4-is does the same, but it also checks the streaming state in
>>>>>>> the release() fop callback, so that would fix this problem. And vimc does this
>>>>>>> in stop_streaming.
>>>>>>>
>>>>>>> I'm in favor of fixing this in vb2, that framework knows exactly when this needs
>>>>>>> to be called.
>>>>>>
>>>>>> Are you suggesting to have vb2 to call media_pipeline_start() or is it
>>>>>> more complex than this ?
>>>>>
>>>>> I think Hans meant adding a .validate() operation to vb2.
>>>>>
>>>>> vb2 is already quite complex, I don't think adding more features is a
>>>>> good idea. I'd rather have vb2 focus on buffer management only
>>>>> (.start_streaming() and .stop_streaming() shouldn't have been in there
>>>>> in my opinion), and handle validation in the .streamon() handler. I'd
>>>>> expect most drivers that deal with media pipelines to do more work in
>>>>> .streamon() anyway.
>>>>
>>>> I disagree with that :-)
>>>>
>>>> It's vb2 that keeps track of the streaming state and when what actions
>>>> need to be taken. Drivers really shouldn't need to care about the ioctls
>>>> themselves, and just implement the relevant vb2 callbacks. Relying on
>>>> drivers to handle any of the streaming ioctls is asking for problems,
>>>> as this shows: most drivers implement this wrong today.
>>>>
>>>> The vb2 framework knows when e.g. the pipeline needs to be started or
>>>> stopped, and can do this at the best time, without drivers needing to
>>>> keep track of when streamon/off/release is called. Keep that logic in
>>>> vb2.
>>>
>>> Pipeline management and buffer management are two different issues.
>>> Don't forget about devices that have multiple video nodes, part of the
>>> same pipeline (possibly a combination of output and capture nodes, or
>>> all of the same type). Forcing drivers to go through vb2 operations to
>>> handle the pipeline will be messy, will result in more bloat in vb2, and
>>> make the result more bug-prone and harder to maintain.
>>>
>>> If pipeline management is too complex, let's simplify it, new helpers
>>> can make sense, but not through vb2.
>>>
>>
>> But it is vb2 that knows when streaming starts and stops.
>
> That's right, but pipeline start (which includes validation and resource
> reservation) needs to be performed synchronously with VIDIOC_STREAMON.
> The streaming state managed by vb2 is not relevant,
> media_pipeline_start() must not be delayed the same way
> .start_streaming() is.

It will be the first thing that vb2_streamon calls. This has nothing to do
with start_streaming: that's called when sufficient number of buffers are
queued up to be able to start the DMA. This proposed prepare_streaming op
will be called when VIDIOC_STREAMON is called.

>
>> The driver just
>> needs to be informed (e.g. prepare_streaming and unprepare_streaming ops).
>>
>> vb2 deals with buffer management and it keeps track of the streaming state
>> and makes the streaming state transitions. That *is* an integral part of
>> vb2. What is missing at the moment are callbacks done at streamon time and
>> when the streaming stops (streamoff, or close() when is_streaming is true).
>>
>> If you want to implement stream validation in a driver, then there are a
>> lot of things you need to do:
>>
>> - override streamon, make sure you call vb2_queue_is_busy(), validate the
>> pipeline, then call vb2_streamon, if that fails, remember to stop the
>> pipeline.
>>
>> - override streamoff, make sure you call vb2_queue_is_busy(), stop the
>> pipeline and call vb2_streamoff.
>>
>> - in the release() function when the fh is closed, you have to check
>> vb2_is_streaming(), check that you are the owner of the queue, and if true,
>> stop the pipeline.
>
> I'm not opposed to helper functions to implement that, they can bundle
> vb2 calls and pipeline management.
>
>> By moving this to vb2 ops all you need to implement are the prepare and
>> unprepare ops.
>>
>> Esp. the release() implementation is tricky. I'm pretty sure that
>> drivers/media/platform/samsung/exynos4-is/fimc-lite.c is wrong, since it
>> should only call media_pipeline_stop() for the owner of the queue. Instead
>> it calls it for the last user of the queue.
>>
>> I see that fimc_lite_streamoff() is wrong too: you can safely call
>> VIDIOC_STREAMOFF twice: the second streamoff just returns 0 without
>> doing anything. Instead media_pipeline_stop is called without testing if
>> the queue is streaming.
>>
>> And yes, this is in part because V4L2 has quite some history and certainly
>> API choice were made in the past that we wouldn't make today. But vb2
>> shields you from that, and behaves much more like a proper state machine.
>>
>> I know you prefer to give a lot more control to driver developers, but
>> in my experience very few developers can do things like this right. And
>> it is really hard as a reviewer to check if all the corner cases are handled
>> correctly in a driver. If vb2 is used, then I know things are called at the
>> right time, and that makes my life as reviewer so much easier.
>
> It's not just about giving more control to drivers, it's about
> organizing the software layers in a way that keeps them maintainable,
> with layered abstractions and not midlayers.
>
> We are extensively reworking the media pipeline management as part of
> the stream series, and there will be more work on top of that that will
> make even more fundamental changes. I would like to at least postpone
> any work on vb2 until then, to be able to evaluate the impact.

I'll make an RFC patch for vb2 so you have a better idea of what it does.

Regards,

Hans

>
>> There may still be a few drivers that really need to do this manually, and
>> that's OK, but a driver like the atmel-isc doesn't need that at all.
>

2022-05-01 07:14:51

by jacopo mondi

[permalink] [raw]

Subject: Re: [PATCH v9 08/13] media: atmel: atmel-isc: change format propagation to subdev into only verification

Hi

On Fri, Apr 29, 2022 at 03:28:11PM +0300, Laurent Pinchart wrote:
> On Fri, Apr 29, 2022 at 02:23:45PM +0200, Hans Verkuil wrote:
> > On 29/04/2022 14:02, Laurent Pinchart wrote:
> > > On Fri, Apr 29, 2022 at 01:17:27PM +0200, Hans Verkuil wrote:
> > >> On 29/04/2022 12:20, Laurent Pinchart wrote:
> > >>> On Fri, Apr 29, 2022 at 12:13:46PM +0200, Hans Verkuil wrote:
> > >>>> On 29/04/2022 12:07, Laurent Pinchart wrote:
> > >>>>> On Fri, Apr 29, 2022 at 11:58:48AM +0200, Jacopo Mondi wrote:
> > >>>>>> On Fri, Apr 29, 2022 at 10:43:09AM +0200, Hans Verkuil wrote:
> > >>>>>>> On 29/04/2022 10:28, [email protected] wrote:
> > >>>>>>>> On 4/29/22 11:17 AM, Hans Verkuil wrote:
> > >>>>>>>>> On 10/03/2022 10:51, Eugen Hristev wrote:
> > >>>>>>>>>> As a top MC video driver, the atmel-isc should not propagate the format to the
> > >>>>>>>>>> subdevice, it should rather check at start_streaming() time if the subdev is properly
> > >>>>>>>>>> configured with a compatible format.
> > >>>>>>>>>> Removed the whole format finding logic, and reworked the format verification
> > >>>>>>>>>> at start_streaming time, such that the ISC will return an error if the subdevice
> > >>>>>>>>>> is not properly configured. To achieve this, media_pipeline_start
> > >>>>>>>>>> is called and a link_validate callback is created to check the formats.
> > >>>>>>>>>> With this being done, the module parameter 'sensor_preferred' makes no sense
> > >>>>>>>>>> anymore. The ISC should not decide which format the sensor is using. The
> > >>>>>>>>>> ISC should only cope with the situation and inform userspace if the streaming
> > >>>>>>>>>> is possible in the current configuration.
> > >>>>>>>>>> The redesign of the format propagation has also risen the question of the
> > >>>>>>>>>> enumfmt callback. If enumfmt is called with an mbus_code, the enumfmt handler
> > >>>>>>>>>> should only return the formats that are supported for this mbus_code.
> > >>>>>>>>>> Otherwise, the enumfmt will report all the formats that the ISC could output.
> > >>>>>>>>>> With this rework, the dynamic list of user formats is removed. It makes no
> > >>>>>>>>>> more sense to identify at complete time which formats the sensor could emit,
> > >>>>>>>>>> and add those into a separate dynamic list.
> > >>>>>>>>>> The ISC will start with a simple preconfigured default format, and at
> > >>>>>>>>>> link validate time, decide whether it can use the format that is configured
> > >>>>>>>>>> on the sink or not.
> > >>>>>>>>>>
> > >>>>>>>>>> Signed-off-by: Eugen Hristev <[email protected]>
> > >>>>>>>>>> Reviewed-by: Jacopo Mondi <[email protected]>
> > >>>>>>>>>> ---
> > >>>>>>>>>> Changes in v9:
> > >>>>>>>>>> - isc_link_validate now static
> > >>>>>>>>>>
> > >>>>>>>>>> Changes in v7:
> > >>>>>>>>>> - minor typos as suggested by Jacopo
> > >>>>>>>>>> - small changes, reduce some indentation, modified an index, as suggested by
> > >>>>>>>>>> Jacopo
> > >>>>>>>>>>
> > >>>>>>>>>> Changes in v6:
> > >>>>>>>>>> - reworked a bit enum_fmt as suggested by Jacopo
> > >>>>>>>>>>
> > >>>>>>>>>> Changes in v5:
> > >>>>>>>>>> - removed user_formats dynamic list as it is now pointless
> > >>>>>>>>>> - greatly simplified the enum_fmt function
> > >>>>>>>>>> - removed some init code that was useless now
> > >>>>>>>>>>
> > >>>>>>>>>> Changes in v4:
> > >>>>>>>>>> - moved validation code into link_validate and used media_pipeline_start
> > >>>>>>>>>> - merged this patch with the enum_fmt patch which was previously in v3 of
> > >>>>>>>>>> the series
> > >>>>>>>>>>
> > >>>>>>>>>> Changes in v3:
> > >>>>>>>>>> - clamp to maximum resolution once the frame size from the subdev is found
> > >>>>>>>>>> drivers/media/platform/atmel/atmel-isc-base.c | 412 ++++++++----------
> > >>>>>>>>>> .../media/platform/atmel/atmel-isc-scaler.c | 5 +
> > >>>>>>>>>> drivers/media/platform/atmel/atmel-isc.h | 13 +-
> > >>>>>>>>>> .../media/platform/atmel/atmel-sama5d2-isc.c | 20 +
> > >>>>>>>>>> .../media/platform/atmel/atmel-sama7g5-isc.c | 20 +
> > >>>>>>>>>> 5 files changed, 236 insertions(+), 234 deletions(-)
> > >>>>>>>>>>
> > >>>>>>>>>> diff --git a/drivers/media/platform/atmel/atmel-isc-base.c b/drivers/media/platform/atmel/atmel-isc-base.c
> > >>>>>>>>>> index ee1dda6707a0..fe2c0af58060 100644
> > >>>>>>>>>> --- a/drivers/media/platform/atmel/atmel-isc-base.c
> > >>>>>>>>>> +++ b/drivers/media/platform/atmel/atmel-isc-base.c
> > >>>>>>>>>> @@ -36,11 +36,6 @@ static unsigned int debug;
> > >>>>>>>>>> module_param(debug, int, 0644);
> > >>>>>>>>>> MODULE_PARM_DESC(debug, "debug level (0-2)");
> > >>>>>>>>>>
> > >>>>>>>>>> -static unsigned int sensor_preferred = 1;
> > >>>>>>>>>> -module_param(sensor_preferred, uint, 0644);
> > >>>>>>>>>> -MODULE_PARM_DESC(sensor_preferred,
> > >>>>>>>>>> - "Sensor is preferred to output the specified format (1-on 0-off), default 1");
> > >>>>>>>>>> -
> > >>>>>>>>>> #define ISC_IS_FORMAT_RAW(mbus_code) \
> > >>>>>>>>>> (((mbus_code) & 0xf000) == 0x3000)
> > >>>>>>>>>>
> > >>>>>>>>>> @@ -337,6 +332,10 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)
> > >>>>>>>>>> unsigned long flags;
> > >>>>>>>>>> int ret;
> > >>>>>>>>>>
> > >>>>>>>>>> + ret = media_pipeline_start(&isc->video_dev.entity, &isc->mpipe);
> > >>>>>>>>>
> > >>>>>>>>> The pipeline validation is done in start_streaming, but I don't think that
> > >>>>>>>>> is the best place: if STREAMON is called before buffers are queued, then
> > >>>>>>>>> an invalid pipeline isn't discovered until enough buffers are queued to
> > >>>>>>>>> kick off start_streaming.
> > >>>>>>>>>
> > >>>>>>>>> Drivers like vsp1, omap3isp and the samsung drivers all do this in streamon().
> > >>>>>>>>>
> > >>>>>>>>> I think that is the correct time to do this.
> > >>>>>>>>
> > >>>>>>>> Hello Hans,
> > >>>>>>>>
> > >>>>>>>> Initially (v2, v3) I had this in streamon(). The problem that I faced at
> > >>>>>>>> that time was that streamoff was never called, so I could not call
> > >>>>>>>> media_pipeline_stop(). Then Jacopo told me to move it to start_streaming
> > >>>>>>>> (see change log for v4) , and I did not face any more problems.
> > >>>>>>
> > >>>>>> Yes indeed, seems I suggested to use media_pipeline_handler in a
> > >>>>>> comment on your v3
> > >>>>>>
> > >>>>>> "at s_stream time your top driver calls media_pipeline_start()"
> > >>>>>>
> > >>>>>> sorry about that, I should have looked around a bit more carefully and
> > >>>>>> notice most drivers do so at vb2 streamon
> > >>>>>>
> > >>>>>> However I don't see media_pipeline_start being called at all in v3 of
> > >>>>>> the patch
> > >>>>>>
> > >>>>>>> It's a mess. Looking at some drivers I see that omap3isp calls media_pipeline_stop
> > >>>>>>> in streamoff (so will have the same problem as you described if VIDIOC_STREAMOFF
> > >>>>>>> isn't called), exynos4-is does the same, but it also checks the streaming state in
> > >>>>>>> the release() fop callback, so that would fix this problem. And vimc does this
> > >>>>>>> in stop_streaming.
> > >>>>>>>
> > >>>>>>> I'm in favor of fixing this in vb2, that framework knows exactly when this needs
> > >>>>>>> to be called.
> > >>>>>>
> > >>>>>> Are you suggesting to have vb2 to call media_pipeline_start() or is it
> > >>>>>> more complex than this ?
> > >>>>>
> > >>>>> I think Hans meant adding a .validate() operation to vb2.
> > >>>>>
> > >>>>> vb2 is already quite complex, I don't think adding more features is a
> > >>>>> good idea. I'd rather have vb2 focus on buffer management only
> > >>>>> (.start_streaming() and .stop_streaming() shouldn't have been in there
> > >>>>> in my opinion), and handle validation in the .streamon() handler. I'd
> > >>>>> expect most drivers that deal with media pipelines to do more work in
> > >>>>> .streamon() anyway.
> > >>>>
> > >>>> I disagree with that :-)
> > >>>>
> > >>>> It's vb2 that keeps track of the streaming state and when what actions
> > >>>> need to be taken. Drivers really shouldn't need to care about the ioctls
> > >>>> themselves, and just implement the relevant vb2 callbacks. Relying on
> > >>>> drivers to handle any of the streaming ioctls is asking for problems,
> > >>>> as this shows: most drivers implement this wrong today.
> > >>>>
> > >>>> The vb2 framework knows when e.g. the pipeline needs to be started or
> > >>>> stopped, and can do this at the best time, without drivers needing to
> > >>>> keep track of when streamon/off/release is called. Keep that logic in
> > >>>> vb2.
> > >>>
> > >>> Pipeline management and buffer management are two different issues.
> > >>> Don't forget about devices that have multiple video nodes, part of the
> > >>> same pipeline (possibly a combination of output and capture nodes, or
> > >>> all of the same type). Forcing drivers to go through vb2 operations to
> > >>> handle the pipeline will be messy, will result in more bloat in vb2, and
> > >>> make the result more bug-prone and harder to maintain.
> > >>>
> > >>> If pipeline management is too complex, let's simplify it, new helpers
> > >>> can make sense, but not through vb2.
> > >>
> > >> But it is vb2 that knows when streaming starts and stops.
> > >
> > > That's right, but pipeline start (which includes validation and resource
> > > reservation) needs to be performed synchronously with VIDIOC_STREAMON.
> > > The streaming state managed by vb2 is not relevant,
> > > media_pipeline_start() must not be delayed the same way
> > > .start_streaming() is.
> >
> > It will be the first thing that vb2_streamon calls. This has nothing to do
> > with start_streaming: that's called when sufficient number of buffers are
> > queued up to be able to start the DMA. This proposed prepare_streaming op
> > will be called when VIDIOC_STREAMON is called.
>
> Then it doesn't need vb2's knowledge of the stream state :-)
>
> > >> The driver just
> > >> needs to be informed (e.g. prepare_streaming and unprepare_streaming ops).
> > >>
> > >> vb2 deals with buffer management and it keeps track of the streaming state
> > >> and makes the streaming state transitions. That *is* an integral part of
> > >> vb2. What is missing at the moment are callbacks done at streamon time and
> > >> when the streaming stops (streamoff, or close() when is_streaming is true).
> > >>
> > >> If you want to implement stream validation in a driver, then there are a
> > >> lot of things you need to do:
> > >>
> > >> - override streamon, make sure you call vb2_queue_is_busy(), validate the
> > >> pipeline, then call vb2_streamon, if that fails, remember to stop the
> > >> pipeline.
> > >>
> > >> - override streamoff, make sure you call vb2_queue_is_busy(), stop the
> > >> pipeline and call vb2_streamoff.
> > >>
> > >> - in the release() function when the fh is closed, you have to check
> > >> vb2_is_streaming(), check that you are the owner of the queue, and if true,
> > >> stop the pipeline.
> > >
> > > I'm not opposed to helper functions to implement that, they can bundle
> > > vb2 calls and pipeline management.
> > >
> > >> By moving this to vb2 ops all you need to implement are the prepare and
> > >> unprepare ops.
> > >>
> > >> Esp. the release() implementation is tricky. I'm pretty sure that
> > >> drivers/media/platform/samsung/exynos4-is/fimc-lite.c is wrong, since it
> > >> should only call media_pipeline_stop() for the owner of the queue. Instead
> > >> it calls it for the last user of the queue.
> > >>
> > >> I see that fimc_lite_streamoff() is wrong too: you can safely call
> > >> VIDIOC_STREAMOFF twice: the second streamoff just returns 0 without
> > >> doing anything. Instead media_pipeline_stop is called without testing if
> > >> the queue is streaming.
> > >>
> > >> And yes, this is in part because V4L2 has quite some history and certainly
> > >> API choice were made in the past that we wouldn't make today. But vb2
> > >> shields you from that, and behaves much more like a proper state machine.
> > >>
> > >> I know you prefer to give a lot more control to driver developers, but
> > >> in my experience very few developers can do things like this right. And
> > >> it is really hard as a reviewer to check if all the corner cases are handled
> > >> correctly in a driver. If vb2 is used, then I know things are called at the
> > >> right time, and that makes my life as reviewer so much easier.
> > >
> > > It's not just about giving more control to drivers, it's about
> > > organizing the software layers in a way that keeps them maintainable,
> > > with layered abstractions and not midlayers.
> > >
> > > We are extensively reworking the media pipeline management as part of
> > > the stream series, and there will be more work on top of that that will
> > > make even more fundamental changes. I would like to at least postpone
> > > any work on vb2 until then, to be able to evaluate the impact.
> >
> > I'll make an RFC patch for vb2 so you have a better idea of what it does.
>
> As long as we don't merge it before I get the chance to send the media
> pipeline management rework, I'm all for RFCs :-)
>

To unblock Eugen is it fine if he moves media_pipeline_start() at
streamon() time for now ? It will require overriding
v4l2_ioctl.vidioc_streamon which might be a bit of additional work
(but probably easier to replace once a proper solution lands)

Otherwise, should the series go in as it is now ?

Thanks
j

> > >> There may still be a few drivers that really need to do this manually, and
> > >> that's OK, but a driver like the atmel-isc doesn't need that at all.
>
> --
> Regards,
>
> Laurent Pinchart

2022-05-01 09:38:26

[permalink] [raw]

Subject: Re: [PATCH v9 08/13] media: atmel: atmel-isc: change format propagation to subdev into only verification

On 29/04/2022 12:20, Laurent Pinchart wrote:
> On Fri, Apr 29, 2022 at 12:13:46PM +0200, Hans Verkuil wrote:
>> On 29/04/2022 12:07, Laurent Pinchart wrote:
>>> On Fri, Apr 29, 2022 at 11:58:48AM +0200, Jacopo Mondi wrote:
>>>> On Fri, Apr 29, 2022 at 10:43:09AM +0200, Hans Verkuil wrote:
>>>>> On 29/04/2022 10:28, [email protected] wrote:
>>>>>> On 4/29/22 11:17 AM, Hans Verkuil wrote:
>>>>>>> On 10/03/2022 10:51, Eugen Hristev wrote:
>>>>>>>> As a top MC video driver, the atmel-isc should not propagate the format to the
>>>>>>>> subdevice, it should rather check at start_streaming() time if the subdev is properly
>>>>>>>> configured with a compatible format.
>>>>>>>> Removed the whole format finding logic, and reworked the format verification
>>>>>>>> at start_streaming time, such that the ISC will return an error if the subdevice
>>>>>>>> is not properly configured. To achieve this, media_pipeline_start
>>>>>>>> is called and a link_validate callback is created to check the formats.
>>>>>>>> With this being done, the module parameter 'sensor_preferred' makes no sense
>>>>>>>> anymore. The ISC should not decide which format the sensor is using. The
>>>>>>>> ISC should only cope with the situation and inform userspace if the streaming
>>>>>>>> is possible in the current configuration.
>>>>>>>> The redesign of the format propagation has also risen the question of the
>>>>>>>> enumfmt callback. If enumfmt is called with an mbus_code, the enumfmt handler
>>>>>>>> should only return the formats that are supported for this mbus_code.
>>>>>>>> Otherwise, the enumfmt will report all the formats that the ISC could output.
>>>>>>>> With this rework, the dynamic list of user formats is removed. It makes no
>>>>>>>> more sense to identify at complete time which formats the sensor could emit,
>>>>>>>> and add those into a separate dynamic list.
>>>>>>>> The ISC will start with a simple preconfigured default format, and at
>>>>>>>> link validate time, decide whether it can use the format that is configured
>>>>>>>> on the sink or not.
>>>>>>>>
>>>>>>>> Signed-off-by: Eugen Hristev <[email protected]>
>>>>>>>> Reviewed-by: Jacopo Mondi <[email protected]>
>>>>>>>> ---
>>>>>>>> Changes in v9:
>>>>>>>> - isc_link_validate now static
>>>>>>>>
>>>>>>>> Changes in v7:
>>>>>>>> - minor typos as suggested by Jacopo
>>>>>>>> - small changes, reduce some indentation, modified an index, as suggested by
>>>>>>>> Jacopo
>>>>>>>>
>>>>>>>> Changes in v6:
>>>>>>>> - reworked a bit enum_fmt as suggested by Jacopo
>>>>>>>>
>>>>>>>> Changes in v5:
>>>>>>>> - removed user_formats dynamic list as it is now pointless
>>>>>>>> - greatly simplified the enum_fmt function
>>>>>>>> - removed some init code that was useless now
>>>>>>>>
>>>>>>>> Changes in v4:
>>>>>>>> - moved validation code into link_validate and used media_pipeline_start
>>>>>>>> - merged this patch with the enum_fmt patch which was previously in v3 of
>>>>>>>> the series
>>>>>>>>
>>>>>>>> Changes in v3:
>>>>>>>> - clamp to maximum resolution once the frame size from the subdev is found
>>>>>>>> drivers/media/platform/atmel/atmel-isc-base.c | 412 ++++++++----------
>>>>>>>> .../media/platform/atmel/atmel-isc-scaler.c | 5 +
>>>>>>>> drivers/media/platform/atmel/atmel-isc.h | 13 +-
>>>>>>>> .../media/platform/atmel/atmel-sama5d2-isc.c | 20 +
>>>>>>>> .../media/platform/atmel/atmel-sama7g5-isc.c | 20 +
>>>>>>>> 5 files changed, 236 insertions(+), 234 deletions(-)
>>>>>>>>
>>>>>>>> diff --git a/drivers/media/platform/atmel/atmel-isc-base.c b/drivers/media/platform/atmel/atmel-isc-base.c
>>>>>>>> index ee1dda6707a0..fe2c0af58060 100644
>>>>>>>> --- a/drivers/media/platform/atmel/atmel-isc-base.c
>>>>>>>> +++ b/drivers/media/platform/atmel/atmel-isc-base.c
>>>>>>>> @@ -36,11 +36,6 @@ static unsigned int debug;
>>>>>>>> module_param(debug, int, 0644);
>>>>>>>> MODULE_PARM_DESC(debug, "debug level (0-2)");
>>>>>>>>
>>>>>>>> -static unsigned int sensor_preferred = 1;
>>>>>>>> -module_param(sensor_preferred, uint, 0644);
>>>>>>>> -MODULE_PARM_DESC(sensor_preferred,
>>>>>>>> - "Sensor is preferred to output the specified format (1-on 0-off), default 1");
>>>>>>>> -
>>>>>>>> #define ISC_IS_FORMAT_RAW(mbus_code) \
>>>>>>>> (((mbus_code) & 0xf000) == 0x3000)
>>>>>>>>
>>>>>>>> @@ -337,6 +332,10 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)
>>>>>>>> unsigned long flags;
>>>>>>>> int ret;
>>>>>>>>
>>>>>>>> + ret = media_pipeline_start(&isc->video_dev.entity, &isc->mpipe);
>>>>>>>
>>>>>>> The pipeline validation is done in start_streaming, but I don't think that
>>>>>>> is the best place: if STREAMON is called before buffers are queued, then
>>>>>>> an invalid pipeline isn't discovered until enough buffers are queued to
>>>>>>> kick off start_streaming.
>>>>>>>
>>>>>>> Drivers like vsp1, omap3isp and the samsung drivers all do this in streamon().
>>>>>>>
>>>>>>> I think that is the correct time to do this.
>>>>>>
>>>>>> Hello Hans,
>>>>>>
>>>>>> Initially (v2, v3) I had this in streamon(). The problem that I faced at
>>>>>> that time was that streamoff was never called, so I could not call
>>>>>> media_pipeline_stop(). Then Jacopo told me to move it to start_streaming
>>>>>> (see change log for v4) , and I did not face any more problems.
>>>>
>>>> Yes indeed, seems I suggested to use media_pipeline_handler in a
>>>> comment on your v3
>>>>
>>>> "at s_stream time your top driver calls media_pipeline_start()"
>>>>
>>>> sorry about that, I should have looked around a bit more carefully and
>>>> notice most drivers do so at vb2 streamon
>>>>
>>>> However I don't see media_pipeline_start being called at all in v3 of
>>>> the patch
>>>>
>>>>> It's a mess. Looking at some drivers I see that omap3isp calls media_pipeline_stop
>>>>> in streamoff (so will have the same problem as you described if VIDIOC_STREAMOFF
>>>>> isn't called), exynos4-is does the same, but it also checks the streaming state in
>>>>> the release() fop callback, so that would fix this problem. And vimc does this
>>>>> in stop_streaming.
>>>>>
>>>>> I'm in favor of fixing this in vb2, that framework knows exactly when this needs
>>>>> to be called.
>>>>
>>>> Are you suggesting to have vb2 to call media_pipeline_start() or is it
>>>> more complex than this ?
>>>
>>> I think Hans meant adding a .validate() operation to vb2.
>>>
>>> vb2 is already quite complex, I don't think adding more features is a
>>> good idea. I'd rather have vb2 focus on buffer management only
>>> (.start_streaming() and .stop_streaming() shouldn't have been in there
>>> in my opinion), and handle validation in the .streamon() handler. I'd
>>> expect most drivers that deal with media pipelines to do more work in
>>> .streamon() anyway.
>>
>> I disagree with that :-)
>>
>> It's vb2 that keeps track of the streaming state and when what actions
>> need to be taken. Drivers really shouldn't need to care about the ioctls
>> themselves, and just implement the relevant vb2 callbacks. Relying on
>> drivers to handle any of the streaming ioctls is asking for problems,
>> as this shows: most drivers implement this wrong today.
>>
>> The vb2 framework knows when e.g. the pipeline needs to be started or
>> stopped, and can do this at the best time, without drivers needing to
>> keep track of when streamon/off/release is called. Keep that logic in
>> vb2.
>
> Pipeline management and buffer management are two different issues.
> Don't forget about devices that have multiple video nodes, part of the
> same pipeline (possibly a combination of output and capture nodes, or
> all of the same type). Forcing drivers to go through vb2 operations to
> handle the pipeline will be messy, will result in more bloat in vb2, and
> make the result more bug-prone and harder to maintain.
>
> If pipeline management is too complex, let's simplify it, new helpers
> can make sense, but not through vb2.
>

But it is vb2 that knows when streaming starts and stops. The driver just
needs to be informed (e.g. prepare_streaming and unprepare_streaming ops).

vb2 deals with buffer management and it keeps track of the streaming state
and makes the streaming state transitions. That *is* an integral part of
vb2. What is missing at the moment are callbacks done at streamon time and
when the streaming stops (streamoff, or close() when is_streaming is true).

If you want to implement stream validation in a driver, then there are a
lot of things you need to do:

- override streamon, make sure you call vb2_queue_is_busy(), validate the
pipeline, then call vb2_streamon, if that fails, remember to stop the
pipeline.

- override streamoff, make sure you call vb2_queue_is_busy(), stop the
pipeline and call vb2_streamoff.

- in the release() function when the fh is closed, you have to check
vb2_is_streaming(), check that you are the owner of the queue, and if true,
stop the pipeline.

By moving this to vb2 ops all you need to implement are the prepare and
unprepare ops.

Esp. the release() implementation is tricky. I'm pretty sure that
drivers/media/platform/samsung/exynos4-is/fimc-lite.c is wrong, since it
should only call media_pipeline_stop() for the owner of the queue. Instead
it calls it for the last user of the queue.

I see that fimc_lite_streamoff() is wrong too: you can safely call
VIDIOC_STREAMOFF twice: the second streamoff just returns 0 without
doing anything. Instead media_pipeline_stop is called without testing if
the queue is streaming.

And yes, this is in part because V4L2 has quite some history and certainly
API choice were made in the past that we wouldn't make today. But vb2
shields you from that, and behaves much more like a proper state machine.

I know you prefer to give a lot more control to driver developers, but
in my experience very few developers can do things like this right. And
it is really hard as a reviewer to check if all the corner cases are handled
correctly in a driver. If vb2 is used, then I know things are called at the
right time, and that makes my life as reviewer so much easier.

There may still be a few drivers that really need to do this manually, and
that's OK, but a driver like the atmel-isc doesn't need that at all.

Regards,

Hans

2022-05-01 15:10:27

[permalink] [raw]

Subject: Re: [PATCH v9 08/13] media: atmel: atmel-isc: change format propagation to subdev into only verification

On Fri, Apr 29, 2022 at 02:23:45PM +0200, Hans Verkuil wrote:
> On 29/04/2022 14:02, Laurent Pinchart wrote:
> > On Fri, Apr 29, 2022 at 01:17:27PM +0200, Hans Verkuil wrote:
> >> On 29/04/2022 12:20, Laurent Pinchart wrote:
> >>> On Fri, Apr 29, 2022 at 12:13:46PM +0200, Hans Verkuil wrote:
> >>>> On 29/04/2022 12:07, Laurent Pinchart wrote:
> >>>>> On Fri, Apr 29, 2022 at 11:58:48AM +0200, Jacopo Mondi wrote:
> >>>>>> On Fri, Apr 29, 2022 at 10:43:09AM +0200, Hans Verkuil wrote:
> >>>>>>> On 29/04/2022 10:28, [email protected] wrote:
> >>>>>>>> On 4/29/22 11:17 AM, Hans Verkuil wrote:
> >>>>>>>>> On 10/03/2022 10:51, Eugen Hristev wrote:
> >>>>>>>>>> As a top MC video driver, the atmel-isc should not propagate the format to the
> >>>>>>>>>> subdevice, it should rather check at start_streaming() time if the subdev is properly
> >>>>>>>>>> configured with a compatible format.
> >>>>>>>>>> Removed the whole format finding logic, and reworked the format verification
> >>>>>>>>>> at start_streaming time, such that the ISC will return an error if the subdevice
> >>>>>>>>>> is not properly configured. To achieve this, media_pipeline_start
> >>>>>>>>>> is called and a link_validate callback is created to check the formats.
> >>>>>>>>>> With this being done, the module parameter 'sensor_preferred' makes no sense
> >>>>>>>>>> anymore. The ISC should not decide which format the sensor is using. The
> >>>>>>>>>> ISC should only cope with the situation and inform userspace if the streaming
> >>>>>>>>>> is possible in the current configuration.
> >>>>>>>>>> The redesign of the format propagation has also risen the question of the
> >>>>>>>>>> enumfmt callback. If enumfmt is called with an mbus_code, the enumfmt handler
> >>>>>>>>>> should only return the formats that are supported for this mbus_code.
> >>>>>>>>>> Otherwise, the enumfmt will report all the formats that the ISC could output.
> >>>>>>>>>> With this rework, the dynamic list of user formats is removed. It makes no
> >>>>>>>>>> more sense to identify at complete time which formats the sensor could emit,
> >>>>>>>>>> and add those into a separate dynamic list.
> >>>>>>>>>> The ISC will start with a simple preconfigured default format, and at
> >>>>>>>>>> link validate time, decide whether it can use the format that is configured
> >>>>>>>>>> on the sink or not.
> >>>>>>>>>>
> >>>>>>>>>> Signed-off-by: Eugen Hristev <[email protected]>
> >>>>>>>>>> Reviewed-by: Jacopo Mondi <[email protected]>
> >>>>>>>>>> ---
> >>>>>>>>>> Changes in v9:
> >>>>>>>>>> - isc_link_validate now static
> >>>>>>>>>>
> >>>>>>>>>> Changes in v7:
> >>>>>>>>>> - minor typos as suggested by Jacopo
> >>>>>>>>>> - small changes, reduce some indentation, modified an index, as suggested by
> >>>>>>>>>> Jacopo
> >>>>>>>>>>
> >>>>>>>>>> Changes in v6:
> >>>>>>>>>> - reworked a bit enum_fmt as suggested by Jacopo
> >>>>>>>>>>
> >>>>>>>>>> Changes in v5:
> >>>>>>>>>> - removed user_formats dynamic list as it is now pointless
> >>>>>>>>>> - greatly simplified the enum_fmt function
> >>>>>>>>>> - removed some init code that was useless now
> >>>>>>>>>>
> >>>>>>>>>> Changes in v4:
> >>>>>>>>>> - moved validation code into link_validate and used media_pipeline_start
> >>>>>>>>>> - merged this patch with the enum_fmt patch which was previously in v3 of
> >>>>>>>>>> the series
> >>>>>>>>>>
> >>>>>>>>>> Changes in v3:
> >>>>>>>>>> - clamp to maximum resolution once the frame size from the subdev is found
> >>>>>>>>>> drivers/media/platform/atmel/atmel-isc-base.c | 412 ++++++++----------
> >>>>>>>>>> .../media/platform/atmel/atmel-isc-scaler.c | 5 +
> >>>>>>>>>> drivers/media/platform/atmel/atmel-isc.h | 13 +-
> >>>>>>>>>> .../media/platform/atmel/atmel-sama5d2-isc.c | 20 +
> >>>>>>>>>> .../media/platform/atmel/atmel-sama7g5-isc.c | 20 +
> >>>>>>>>>> 5 files changed, 236 insertions(+), 234 deletions(-)
> >>>>>>>>>>
> >>>>>>>>>> diff --git a/drivers/media/platform/atmel/atmel-isc-base.c b/drivers/media/platform/atmel/atmel-isc-base.c
> >>>>>>>>>> index ee1dda6707a0..fe2c0af58060 100644
> >>>>>>>>>> --- a/drivers/media/platform/atmel/atmel-isc-base.c
> >>>>>>>>>> +++ b/drivers/media/platform/atmel/atmel-isc-base.c
> >>>>>>>>>> @@ -36,11 +36,6 @@ static unsigned int debug;
> >>>>>>>>>> module_param(debug, int, 0644);
> >>>>>>>>>> MODULE_PARM_DESC(debug, "debug level (0-2)");
> >>>>>>>>>>
> >>>>>>>>>> -static unsigned int sensor_preferred = 1;
> >>>>>>>>>> -module_param(sensor_preferred, uint, 0644);
> >>>>>>>>>> -MODULE_PARM_DESC(sensor_preferred,
> >>>>>>>>>> - "Sensor is preferred to output the specified format (1-on 0-off), default 1");
> >>>>>>>>>> -
> >>>>>>>>>> #define ISC_IS_FORMAT_RAW(mbus_code) \
> >>>>>>>>>> (((mbus_code) & 0xf000) == 0x3000)
> >>>>>>>>>>
> >>>>>>>>>> @@ -337,6 +332,10 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)
> >>>>>>>>>> unsigned long flags;
> >>>>>>>>>> int ret;
> >>>>>>>>>>
> >>>>>>>>>> + ret = media_pipeline_start(&isc->video_dev.entity, &isc->mpipe);
> >>>>>>>>>
> >>>>>>>>> The pipeline validation is done in start_streaming, but I don't think that
> >>>>>>>>> is the best place: if STREAMON is called before buffers are queued, then
> >>>>>>>>> an invalid pipeline isn't discovered until enough buffers are queued to
> >>>>>>>>> kick off start_streaming.
> >>>>>>>>>
> >>>>>>>>> Drivers like vsp1, omap3isp and the samsung drivers all do this in streamon().
> >>>>>>>>>
> >>>>>>>>> I think that is the correct time to do this.
> >>>>>>>>
> >>>>>>>> Hello Hans,
> >>>>>>>>
> >>>>>>>> Initially (v2, v3) I had this in streamon(). The problem that I faced at
> >>>>>>>> that time was that streamoff was never called, so I could not call
> >>>>>>>> media_pipeline_stop(). Then Jacopo told me to move it to start_streaming
> >>>>>>>> (see change log for v4) , and I did not face any more problems.
> >>>>>>
> >>>>>> Yes indeed, seems I suggested to use media_pipeline_handler in a
> >>>>>> comment on your v3
> >>>>>>
> >>>>>> "at s_stream time your top driver calls media_pipeline_start()"
> >>>>>>
> >>>>>> sorry about that, I should have looked around a bit more carefully and
> >>>>>> notice most drivers do so at vb2 streamon
> >>>>>>
> >>>>>> However I don't see media_pipeline_start being called at all in v3 of
> >>>>>> the patch
> >>>>>>
> >>>>>>> It's a mess. Looking at some drivers I see that omap3isp calls media_pipeline_stop
> >>>>>>> in streamoff (so will have the same problem as you described if VIDIOC_STREAMOFF
> >>>>>>> isn't called), exynos4-is does the same, but it also checks the streaming state in
> >>>>>>> the release() fop callback, so that would fix this problem. And vimc does this
> >>>>>>> in stop_streaming.
> >>>>>>>
> >>>>>>> I'm in favor of fixing this in vb2, that framework knows exactly when this needs
> >>>>>>> to be called.
> >>>>>>
> >>>>>> Are you suggesting to have vb2 to call media_pipeline_start() or is it
> >>>>>> more complex than this ?
> >>>>>
> >>>>> I think Hans meant adding a .validate() operation to vb2.
> >>>>>
> >>>>> vb2 is already quite complex, I don't think adding more features is a
> >>>>> good idea. I'd rather have vb2 focus on buffer management only
> >>>>> (.start_streaming() and .stop_streaming() shouldn't have been in there
> >>>>> in my opinion), and handle validation in the .streamon() handler. I'd
> >>>>> expect most drivers that deal with media pipelines to do more work in
> >>>>> .streamon() anyway.
> >>>>
> >>>> I disagree with that :-)
> >>>>
> >>>> It's vb2 that keeps track of the streaming state and when what actions
> >>>> need to be taken. Drivers really shouldn't need to care about the ioctls
> >>>> themselves, and just implement the relevant vb2 callbacks. Relying on
> >>>> drivers to handle any of the streaming ioctls is asking for problems,
> >>>> as this shows: most drivers implement this wrong today.
> >>>>
> >>>> The vb2 framework knows when e.g. the pipeline needs to be started or
> >>>> stopped, and can do this at the best time, without drivers needing to
> >>>> keep track of when streamon/off/release is called. Keep that logic in
> >>>> vb2.
> >>>
> >>> Pipeline management and buffer management are two different issues.
> >>> Don't forget about devices that have multiple video nodes, part of the
> >>> same pipeline (possibly a combination of output and capture nodes, or
> >>> all of the same type). Forcing drivers to go through vb2 operations to
> >>> handle the pipeline will be messy, will result in more bloat in vb2, and
> >>> make the result more bug-prone and harder to maintain.
> >>>
> >>> If pipeline management is too complex, let's simplify it, new helpers
> >>> can make sense, but not through vb2.
> >>
> >> But it is vb2 that knows when streaming starts and stops.
> >
> > That's right, but pipeline start (which includes validation and resource
> > reservation) needs to be performed synchronously with VIDIOC_STREAMON.
> > The streaming state managed by vb2 is not relevant,
> > media_pipeline_start() must not be delayed the same way
> > .start_streaming() is.
>
> It will be the first thing that vb2_streamon calls. This has nothing to do
> with start_streaming: that's called when sufficient number of buffers are
> queued up to be able to start the DMA. This proposed prepare_streaming op
> will be called when VIDIOC_STREAMON is called.

Then it doesn't need vb2's knowledge of the stream state :-)

> >> The driver just
> >> needs to be informed (e.g. prepare_streaming and unprepare_streaming ops).
> >>
> >> vb2 deals with buffer management and it keeps track of the streaming state
> >> and makes the streaming state transitions. That *is* an integral part of
> >> vb2. What is missing at the moment are callbacks done at streamon time and
> >> when the streaming stops (streamoff, or close() when is_streaming is true).
> >>
> >> If you want to implement stream validation in a driver, then there are a
> >> lot of things you need to do:
> >>
> >> - override streamon, make sure you call vb2_queue_is_busy(), validate the
> >> pipeline, then call vb2_streamon, if that fails, remember to stop the
> >> pipeline.
> >>
> >> - override streamoff, make sure you call vb2_queue_is_busy(), stop the
> >> pipeline and call vb2_streamoff.
> >>
> >> - in the release() function when the fh is closed, you have to check
> >> vb2_is_streaming(), check that you are the owner of the queue, and if true,
> >> stop the pipeline.
> >
> > I'm not opposed to helper functions to implement that, they can bundle
> > vb2 calls and pipeline management.
> >
> >> By moving this to vb2 ops all you need to implement are the prepare and
> >> unprepare ops.
> >>
> >> Esp. the release() implementation is tricky. I'm pretty sure that
> >> drivers/media/platform/samsung/exynos4-is/fimc-lite.c is wrong, since it
> >> should only call media_pipeline_stop() for the owner of the queue. Instead
> >> it calls it for the last user of the queue.
> >>
> >> I see that fimc_lite_streamoff() is wrong too: you can safely call
> >> VIDIOC_STREAMOFF twice: the second streamoff just returns 0 without
> >> doing anything. Instead media_pipeline_stop is called without testing if
> >> the queue is streaming.
> >>
> >> And yes, this is in part because V4L2 has quite some history and certainly
> >> API choice were made in the past that we wouldn't make today. But vb2
> >> shields you from that, and behaves much more like a proper state machine.
> >>
> >> I know you prefer to give a lot more control to driver developers, but
> >> in my experience very few developers can do things like this right. And
> >> it is really hard as a reviewer to check if all the corner cases are handled
> >> correctly in a driver. If vb2 is used, then I know things are called at the
> >> right time, and that makes my life as reviewer so much easier.
> >
> > It's not just about giving more control to drivers, it's about
> > organizing the software layers in a way that keeps them maintainable,
> > with layered abstractions and not midlayers.
> >
> > We are extensively reworking the media pipeline management as part of
> > the stream series, and there will be more work on top of that that will
> > make even more fundamental changes. I would like to at least postpone
> > any work on vb2 until then, to be able to evaluate the impact.
>
> I'll make an RFC patch for vb2 so you have a better idea of what it does.

As long as we don't merge it before I get the chance to send the media
pipeline management rework, I'm all for RFCs :-)

> >> There may still be a few drivers that really need to do this manually, and
> >> that's OK, but a driver like the atmel-isc doesn't need that at all.

--
Regards,

Laurent Pinchart

2022-05-01 16:50:35

[permalink] [raw]

Subject: Re: [PATCH v9 08/13] media: atmel: atmel-isc: change format propagation to subdev into only verification

On 4/29/22 3:50 PM, Hans Verkuil wrote:
> On 29/04/2022 14:38, Jacopo Mondi wrote:
>> Hi
>>
>> On Fri, Apr 29, 2022 at 03:28:11PM +0300, Laurent Pinchart wrote:
>>> On Fri, Apr 29, 2022 at 02:23:45PM +0200, Hans Verkuil wrote:
>>>> On 29/04/2022 14:02, Laurent Pinchart wrote:
>>>>> On Fri, Apr 29, 2022 at 01:17:27PM +0200, Hans Verkuil wrote:
>>>>>> On 29/04/2022 12:20, Laurent Pinchart wrote:
>>>>>>> On Fri, Apr 29, 2022 at 12:13:46PM +0200, Hans Verkuil wrote:
>>>>>>>> On 29/04/2022 12:07, Laurent Pinchart wrote:
>>>>>>>>> On Fri, Apr 29, 2022 at 11:58:48AM +0200, Jacopo Mondi wrote:
>>>>>>>>>> On Fri, Apr 29, 2022 at 10:43:09AM +0200, Hans Verkuil wrote:
>>>>>>>>>>> On 29/04/2022 10:28, [email protected] wrote:
>>>>>>>>>>>> On 4/29/22 11:17 AM, Hans Verkuil wrote:
>>>>>>>>>>>>> On 10/03/2022 10:51, Eugen Hristev wrote:
>>>>>>>>>>>>>> As a top MC video driver, the atmel-isc should not propagate the format to the
>>>>>>>>>>>>>> subdevice, it should rather check at start_streaming() time if the subdev is properly
>>>>>>>>>>>>>> configured with a compatible format.
>>>>>>>>>>>>>> Removed the whole format finding logic, and reworked the format verification
>>>>>>>>>>>>>> at start_streaming time, such that the ISC will return an error if the subdevice
>>>>>>>>>>>>>> is not properly configured. To achieve this, media_pipeline_start
>>>>>>>>>>>>>> is called and a link_validate callback is created to check the formats.
>>>>>>>>>>>>>> With this being done, the module parameter 'sensor_preferred' makes no sense
>>>>>>>>>>>>>> anymore. The ISC should not decide which format the sensor is using. The
>>>>>>>>>>>>>> ISC should only cope with the situation and inform userspace if the streaming
>>>>>>>>>>>>>> is possible in the current configuration.
>>>>>>>>>>>>>> The redesign of the format propagation has also risen the question of the
>>>>>>>>>>>>>> enumfmt callback. If enumfmt is called with an mbus_code, the enumfmt handler
>>>>>>>>>>>>>> should only return the formats that are supported for this mbus_code.
>>>>>>>>>>>>>> Otherwise, the enumfmt will report all the formats that the ISC could output.
>>>>>>>>>>>>>> With this rework, the dynamic list of user formats is removed. It makes no
>>>>>>>>>>>>>> more sense to identify at complete time which formats the sensor could emit,
>>>>>>>>>>>>>> and add those into a separate dynamic list.
>>>>>>>>>>>>>> The ISC will start with a simple preconfigured default format, and at
>>>>>>>>>>>>>> link validate time, decide whether it can use the format that is configured
>>>>>>>>>>>>>> on the sink or not.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Signed-off-by: Eugen Hristev <[email protected]>
>>>>>>>>>>>>>> Reviewed-by: Jacopo Mondi <[email protected]>
>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>> Changes in v9:
>>>>>>>>>>>>>> - isc_link_validate now static
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Changes in v7:
>>>>>>>>>>>>>> - minor typos as suggested by Jacopo
>>>>>>>>>>>>>> - small changes, reduce some indentation, modified an index, as suggested by
>>>>>>>>>>>>>> Jacopo
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Changes in v6:
>>>>>>>>>>>>>> - reworked a bit enum_fmt as suggested by Jacopo
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Changes in v5:
>>>>>>>>>>>>>> - removed user_formats dynamic list as it is now pointless
>>>>>>>>>>>>>> - greatly simplified the enum_fmt function
>>>>>>>>>>>>>> - removed some init code that was useless now
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Changes in v4:
>>>>>>>>>>>>>> - moved validation code into link_validate and used media_pipeline_start
>>>>>>>>>>>>>> - merged this patch with the enum_fmt patch which was previously in v3 of
>>>>>>>>>>>>>> the series
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Changes in v3:
>>>>>>>>>>>>>> - clamp to maximum resolution once the frame size from the subdev is found
>>>>>>>>>>>>>> drivers/media/platform/atmel/atmel-isc-base.c | 412 ++++++++----------
>>>>>>>>>>>>>> .../media/platform/atmel/atmel-isc-scaler.c | 5 +
>>>>>>>>>>>>>> drivers/media/platform/atmel/atmel-isc.h | 13 +-
>>>>>>>>>>>>>> .../media/platform/atmel/atmel-sama5d2-isc.c | 20 +
>>>>>>>>>>>>>> .../media/platform/atmel/atmel-sama7g5-isc.c | 20 +
>>>>>>>>>>>>>> 5 files changed, 236 insertions(+), 234 deletions(-)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> diff --git a/drivers/media/platform/atmel/atmel-isc-base.c b/drivers/media/platform/atmel/atmel-isc-base.c
>>>>>>>>>>>>>> index ee1dda6707a0..fe2c0af58060 100644
>>>>>>>>>>>>>> --- a/drivers/media/platform/atmel/atmel-isc-base.c
>>>>>>>>>>>>>> +++ b/drivers/media/platform/atmel/atmel-isc-base.c
>>>>>>>>>>>>>> @@ -36,11 +36,6 @@ static unsigned int debug;
>>>>>>>>>>>>>> module_param(debug, int, 0644);
>>>>>>>>>>>>>> MODULE_PARM_DESC(debug, "debug level (0-2)");
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -static unsigned int sensor_preferred = 1;
>>>>>>>>>>>>>> -module_param(sensor_preferred, uint, 0644);
>>>>>>>>>>>>>> -MODULE_PARM_DESC(sensor_preferred,
>>>>>>>>>>>>>> - "Sensor is preferred to output the specified format (1-on 0-off), default 1");
>>>>>>>>>>>>>> -
>>>>>>>>>>>>>> #define ISC_IS_FORMAT_RAW(mbus_code) \
>>>>>>>>>>>>>> (((mbus_code) & 0xf000) == 0x3000)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> @@ -337,6 +332,10 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)
>>>>>>>>>>>>>> unsigned long flags;
>>>>>>>>>>>>>> int ret;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> + ret = media_pipeline_start(&isc->video_dev.entity, &isc->mpipe);
>>>>>>>>>>>>>
>>>>>>>>>>>>> The pipeline validation is done in start_streaming, but I don't think that
>>>>>>>>>>>>> is the best place: if STREAMON is called before buffers are queued, then
>>>>>>>>>>>>> an invalid pipeline isn't discovered until enough buffers are queued to
>>>>>>>>>>>>> kick off start_streaming.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Drivers like vsp1, omap3isp and the samsung drivers all do this in streamon().
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think that is the correct time to do this.
>>>>>>>>>>>>
>>>>>>>>>>>> Hello Hans,
>>>>>>>>>>>>
>>>>>>>>>>>> Initially (v2, v3) I had this in streamon(). The problem that I faced at
>>>>>>>>>>>> that time was that streamoff was never called, so I could not call
>>>>>>>>>>>> media_pipeline_stop(). Then Jacopo told me to move it to start_streaming
>>>>>>>>>>>> (see change log for v4) , and I did not face any more problems.
>>>>>>>>>>
>>>>>>>>>> Yes indeed, seems I suggested to use media_pipeline_handler in a
>>>>>>>>>> comment on your v3
>>>>>>>>>>
>>>>>>>>>> "at s_stream time your top driver calls media_pipeline_start()"
>>>>>>>>>>
>>>>>>>>>> sorry about that, I should have looked around a bit more carefully and
>>>>>>>>>> notice most drivers do so at vb2 streamon
>>>>>>>>>>
>>>>>>>>>> However I don't see media_pipeline_start being called at all in v3 of
>>>>>>>>>> the patch
>>>>>>>>>>
>>>>>>>>>>> It's a mess. Looking at some drivers I see that omap3isp calls media_pipeline_stop
>>>>>>>>>>> in streamoff (so will have the same problem as you described if VIDIOC_STREAMOFF
>>>>>>>>>>> isn't called), exynos4-is does the same, but it also checks the streaming state in
>>>>>>>>>>> the release() fop callback, so that would fix this problem. And vimc does this
>>>>>>>>>>> in stop_streaming.
>>>>>>>>>>>
>>>>>>>>>>> I'm in favor of fixing this in vb2, that framework knows exactly when this needs
>>>>>>>>>>> to be called.
>>>>>>>>>>
>>>>>>>>>> Are you suggesting to have vb2 to call media_pipeline_start() or is it
>>>>>>>>>> more complex than this ?
>>>>>>>>>
>>>>>>>>> I think Hans meant adding a .validate() operation to vb2.
>>>>>>>>>
>>>>>>>>> vb2 is already quite complex, I don't think adding more features is a
>>>>>>>>> good idea. I'd rather have vb2 focus on buffer management only
>>>>>>>>> (.start_streaming() and .stop_streaming() shouldn't have been in there
>>>>>>>>> in my opinion), and handle validation in the .streamon() handler. I'd
>>>>>>>>> expect most drivers that deal with media pipelines to do more work in
>>>>>>>>> .streamon() anyway.
>>>>>>>>
>>>>>>>> I disagree with that :-)
>>>>>>>>
>>>>>>>> It's vb2 that keeps track of the streaming state and when what actions
>>>>>>>> need to be taken. Drivers really shouldn't need to care about the ioctls
>>>>>>>> themselves, and just implement the relevant vb2 callbacks. Relying on
>>>>>>>> drivers to handle any of the streaming ioctls is asking for problems,
>>>>>>>> as this shows: most drivers implement this wrong today.
>>>>>>>>
>>>>>>>> The vb2 framework knows when e.g. the pipeline needs to be started or
>>>>>>>> stopped, and can do this at the best time, without drivers needing to
>>>>>>>> keep track of when streamon/off/release is called. Keep that logic in
>>>>>>>> vb2.
>>>>>>>
>>>>>>> Pipeline management and buffer management are two different issues.
>>>>>>> Don't forget about devices that have multiple video nodes, part of the
>>>>>>> same pipeline (possibly a combination of output and capture nodes, or
>>>>>>> all of the same type). Forcing drivers to go through vb2 operations to
>>>>>>> handle the pipeline will be messy, will result in more bloat in vb2, and
>>>>>>> make the result more bug-prone and harder to maintain.
>>>>>>>
>>>>>>> If pipeline management is too complex, let's simplify it, new helpers
>>>>>>> can make sense, but not through vb2.
>>>>>>
>>>>>> But it is vb2 that knows when streaming starts and stops.
>>>>>
>>>>> That's right, but pipeline start (which includes validation and resource
>>>>> reservation) needs to be performed synchronously with VIDIOC_STREAMON.
>>>>> The streaming state managed by vb2 is not relevant,
>>>>> media_pipeline_start() must not be delayed the same way
>>>>> .start_streaming() is.
>>>>
>>>> It will be the first thing that vb2_streamon calls. This has nothing to do
>>>> with start_streaming: that's called when sufficient number of buffers are
>>>> queued up to be able to start the DMA. This proposed prepare_streaming op
>>>> will be called when VIDIOC_STREAMON is called.
>>>
>>> Then it doesn't need vb2's knowledge of the stream state :-)
>>>
>>>>>> The driver just
>>>>>> needs to be informed (e.g. prepare_streaming and unprepare_streaming ops).
>>>>>>
>>>>>> vb2 deals with buffer management and it keeps track of the streaming state
>>>>>> and makes the streaming state transitions. That *is* an integral part of
>>>>>> vb2. What is missing at the moment are callbacks done at streamon time and
>>>>>> when the streaming stops (streamoff, or close() when is_streaming is true).
>>>>>>
>>>>>> If you want to implement stream validation in a driver, then there are a
>>>>>> lot of things you need to do:
>>>>>>
>>>>>> - override streamon, make sure you call vb2_queue_is_busy(), validate the
>>>>>> pipeline, then call vb2_streamon, if that fails, remember to stop the
>>>>>> pipeline.
>>>>>>
>>>>>> - override streamoff, make sure you call vb2_queue_is_busy(), stop the
>>>>>> pipeline and call vb2_streamoff.
>>>>>>
>>>>>> - in the release() function when the fh is closed, you have to check
>>>>>> vb2_is_streaming(), check that you are the owner of the queue, and if true,
>>>>>> stop the pipeline.
>>>>>
>>>>> I'm not opposed to helper functions to implement that, they can bundle
>>>>> vb2 calls and pipeline management.
>>>>>
>>>>>> By moving this to vb2 ops all you need to implement are the prepare and
>>>>>> unprepare ops.
>>>>>>
>>>>>> Esp. the release() implementation is tricky. I'm pretty sure that
>>>>>> drivers/media/platform/samsung/exynos4-is/fimc-lite.c is wrong, since it
>>>>>> should only call media_pipeline_stop() for the owner of the queue. Instead
>>>>>> it calls it for the last user of the queue.
>>>>>>
>>>>>> I see that fimc_lite_streamoff() is wrong too: you can safely call
>>>>>> VIDIOC_STREAMOFF twice: the second streamoff just returns 0 without
>>>>>> doing anything. Instead media_pipeline_stop is called without testing if
>>>>>> the queue is streaming.
>>>>>>
>>>>>> And yes, this is in part because V4L2 has quite some history and certainly
>>>>>> API choice were made in the past that we wouldn't make today. But vb2
>>>>>> shields you from that, and behaves much more like a proper state machine.
>>>>>>
>>>>>> I know you prefer to give a lot more control to driver developers, but
>>>>>> in my experience very few developers can do things like this right. And
>>>>>> it is really hard as a reviewer to check if all the corner cases are handled
>>>>>> correctly in a driver. If vb2 is used, then I know things are called at the
>>>>>> right time, and that makes my life as reviewer so much easier.
>>>>>
>>>>> It's not just about giving more control to drivers, it's about
>>>>> organizing the software layers in a way that keeps them maintainable,
>>>>> with layered abstractions and not midlayers.
>>>>>
>>>>> We are extensively reworking the media pipeline management as part of
>>>>> the stream series, and there will be more work on top of that that will
>>>>> make even more fundamental changes. I would like to at least postpone
>>>>> any work on vb2 until then, to be able to evaluate the impact.
>>>>
>>>> I'll make an RFC patch for vb2 so you have a better idea of what it does.
>>>
>>> As long as we don't merge it before I get the chance to send the media
>>> pipeline management rework, I'm all for RFCs :-)
>>>
>>
>> To unblock Eugen is it fine if he moves media_pipeline_start() at
>> streamon() time for now ? It will require overriding
>> v4l2_ioctl.vidioc_streamon which might be a bit of additional work
>> (but probably easier to replace once a proper solution lands)
>>
>> Otherwise, should the series go in as it is now ?
>
> I had a comment about patch 6, so I want a new series anyway. All the
> patches with just fixes (everything but 4 and 8) can go in once I have
> them.
>
> I think - all things considered - it is currently best to start and stop the
> pipeline in start/stop_streaming: that at least avoids all the complications
> with streamon/off/close() that is really hard to get right without (IMHO)
> my proposed vb2 changes.
>
> With that I will accept the series.

Hi,

I will be a few days off, but next week I will address the changes and
come up with v10. I have to test it a little to make sure everything is
fine.

Thanks !
Eugen

>
> Regards,
>
> Hans
>
>>
>> Thanks
>> j
>>
>>>>>> There may still be a few drivers that really need to do this manually, and
>>>>>> that's OK, but a driver like the atmel-isc doesn't need that at all.
>>>
>>> --
>>> Regards,
>>>
>>> Laurent Pinchart
>

2022-05-01 21:46:05

[permalink] [raw]

Subject: Re: [PATCH v9 00/13] media: atmel: atmel-isc: implement media controller

On 4/29/22 11:05 AM, Hans Verkuil wrote:
> Hi Eugen,
>
> On 10/03/2022 10:51, Eugen Hristev wrote:
>> This series is the v9 series that attempts to support media controller in the
>> atmel ISC and XISC drivers.
>> The CSI2DC driver was accepted thus removed from the patch series, together with
>> other patches.
>>
>> Important note: this series applies on top of current media_staging tree, as it
>> relies on previous patches in the series which were accepted.
>>
>> Thanks to everyone who reviewed my work !
>>
>> Eugen
>>
>> Changes in v9:
>> -> kernel robot reported isc_link_validate is not static, changed to static.
>>
>> Changes in v8:
>> -> scaler: modified crop bounds to have the exact source size
>>
>> Changes in v7:
>> -> scaler: modified crop bounds to have maximum isc size
>> -> format propagation: did small changes as per Jacopo review
>>
>>
>> Changes in v6:
>> -> worked a bit on scaler, added try crop and other changes as per Jacopo review
>> -> worked on isc-base enum_fmt , reworked as per Jacopo review
>>
>> Changes in v5:
>> -> removed patch that removed the 'stop' variable as it was still required
>> -> added two new trivial patches
>> -> reworked some parts of the scaler and format propagation after discussions with Jacopo
>>
>>
>> Changes in v4:
>> -> as reviewed by Hans, added new patch to remove the 'stop' variable and reworked
>> one patch that was using it
>> -> as reviewed by Jacopo, reworked some parts of the media controller implementation
>>
>>
>> Changes in v3:
>> - change in bindings, small fixes in csi2dc driver and conversion to mc
>> for the isc-base.
>> - removed some MAINTAINERS patches and used patterns in MAINTAINERS
>>
>> Changes in v2:
>> - integrated many changes suggested by Jacopo in the review of the v1 series.
>> - add a few new patches
>>
>>
>>
>> Eugen Hristev (13):
>> media: atmel: atmel-isc-base: use streaming status when queueing
>> buffers
>> media: atmel: atmel-isc-base: replace is_streaming call in
>> s_fmt_vid_cap
>> media: atmel: atmel-isc: remove redundant comments
>> media: atmel: atmel-isc: implement media controller
>> media: atmel: atmel-sama5d2-isc: fix wrong mask in YUYV format check
>> media: atmel: atmel-isc-base: use mutex to lock awb workqueue from
>> streaming
>> media: atmel: atmel-isc: compact the controller formats list
>> media: atmel: atmel-isc: change format propagation to subdev into only
>> verification
>> media: atmel: atmel-sama7g5-isc: remove stray line
>> dt-bindings: media: microchip,xisc: add bus-width of 14
>
> I'm a bit unhappy with the order of these patches. Mostly these are fixes,
> except for patches 4 and 8, which are the meat of this series and actually
> switching on the MC support.
>
> Can those be moved to the end? That also makes it easier to merge the earlier
> patches if some more work is needed for the MC part.
>
> I'm also not sure whether patches 4 and 8 shouldn't be a single patch,
> since patch 4 leaves the driver in an inconsistent state since it is
> missing the link validation code that patch 8 adds. Unless I missed
> something?

Hello Hans,

The difference that patch 8 is making is that the 'old way' of
configuring the ISC is no longer possible.

Patch 4 makes the ISC 'mc-ready' with all entities, links, but the old
way still works (meaning that the top driver will call s_fmt down to the
subdev ).
After patch 8, the driver no longer has this support at all, and
validates links at start_streaming, and no longer sets anything to the
subdev, just validates the config that the subdev already has.
So one reason that I had things in two patches was that patch 8 makes
this big change that also makes userspace behave differently and has to
configure all the subdevs and media pipeline.

If you feel patch 4 and patch 8 should be squashed, I can do it, definitely.

Let me know how to proceed ?

Thanks,
Eugen

>
> Regards,
>
> Hans
>
>> ARM: dts: at91: sama7g5: add nodes for video capture
>> ARM: configs: at91: sama7: add xisc and csi2dc
>> ARM: multi_v7_defconfig: add atmel video pipeline modules
>>
>> .../bindings/media/microchip,xisc.yaml | 2 +-
>> arch/arm/boot/dts/sama7g5.dtsi | 49 ++
>> arch/arm/configs/multi_v7_defconfig | 3 +
>> arch/arm/configs/sama7_defconfig | 2 +
>> drivers/media/platform/atmel/Makefile | 2 +-
>> drivers/media/platform/atmel/atmel-isc-base.c | 518 ++++++++++--------
>> .../media/platform/atmel/atmel-isc-scaler.c | 267 +++++++++
>> drivers/media/platform/atmel/atmel-isc.h | 58 +-
>> .../media/platform/atmel/atmel-sama5d2-isc.c | 87 +--
>> .../media/platform/atmel/atmel-sama7g5-isc.c | 93 ++--
>> 10 files changed, 754 insertions(+), 327 deletions(-)
>> create mode 100644 drivers/media/platform/atmel/atmel-isc-scaler.c
>>
>

2022-05-02 08:16:45

[permalink] [raw]

Subject: Re: [PATCH v9 08/13] media: atmel: atmel-isc: change format propagation to subdev into only verification

On 29/04/2022 10:28, [email protected] wrote:
> On 4/29/22 11:17 AM, Hans Verkuil wrote:
>> On 10/03/2022 10:51, Eugen Hristev wrote:
>>> As a top MC video driver, the atmel-isc should not propagate the format to the
>>> subdevice, it should rather check at start_streaming() time if the subdev is properly
>>> configured with a compatible format.
>>> Removed the whole format finding logic, and reworked the format verification
>>> at start_streaming time, such that the ISC will return an error if the subdevice
>>> is not properly configured. To achieve this, media_pipeline_start
>>> is called and a link_validate callback is created to check the formats.
>>> With this being done, the module parameter 'sensor_preferred' makes no sense
>>> anymore. The ISC should not decide which format the sensor is using. The
>>> ISC should only cope with the situation and inform userspace if the streaming
>>> is possible in the current configuration.
>>> The redesign of the format propagation has also risen the question of the
>>> enumfmt callback. If enumfmt is called with an mbus_code, the enumfmt handler
>>> should only return the formats that are supported for this mbus_code.
>>> Otherwise, the enumfmt will report all the formats that the ISC could output.
>>> With this rework, the dynamic list of user formats is removed. It makes no
>>> more sense to identify at complete time which formats the sensor could emit,
>>> and add those into a separate dynamic list.
>>> The ISC will start with a simple preconfigured default format, and at
>>> link validate time, decide whether it can use the format that is configured
>>> on the sink or not.
>>>
>>> Signed-off-by: Eugen Hristev <[email protected]>
>>> Reviewed-by: Jacopo Mondi <[email protected]>
>>> ---
>>> Changes in v9:
>>> - isc_link_validate now static
>>>
>>> Changes in v7:
>>> - minor typos as suggested by Jacopo
>>> - small changes, reduce some indentation, modified an index, as suggested by
>>> Jacopo
>>>
>>> Changes in v6:
>>> - reworked a bit enum_fmt as suggested by Jacopo
>>>
>>> Changes in v5:
>>> - removed user_formats dynamic list as it is now pointless
>>> - greatly simplified the enum_fmt function
>>> - removed some init code that was useless now
>>>
>>> Changes in v4:
>>> - moved validation code into link_validate and used media_pipeline_start
>>> - merged this patch with the enum_fmt patch which was previously in v3 of
>>> the series
>>>
>>> Changes in v3:
>>> - clamp to maximum resolution once the frame size from the subdev is found
>>> drivers/media/platform/atmel/atmel-isc-base.c | 412 ++++++++----------
>>> .../media/platform/atmel/atmel-isc-scaler.c | 5 +
>>> drivers/media/platform/atmel/atmel-isc.h | 13 +-
>>> .../media/platform/atmel/atmel-sama5d2-isc.c | 20 +
>>> .../media/platform/atmel/atmel-sama7g5-isc.c | 20 +
>>> 5 files changed, 236 insertions(+), 234 deletions(-)
>>>
>>> diff --git a/drivers/media/platform/atmel/atmel-isc-base.c b/drivers/media/platform/atmel/atmel-isc-base.c
>>> index ee1dda6707a0..fe2c0af58060 100644
>>> --- a/drivers/media/platform/atmel/atmel-isc-base.c
>>> +++ b/drivers/media/platform/atmel/atmel-isc-base.c
>>> @@ -36,11 +36,6 @@ static unsigned int debug;
>>> module_param(debug, int, 0644);
>>> MODULE_PARM_DESC(debug, "debug level (0-2)");
>>>
>>> -static unsigned int sensor_preferred = 1;
>>> -module_param(sensor_preferred, uint, 0644);
>>> -MODULE_PARM_DESC(sensor_preferred,
>>> - "Sensor is preferred to output the specified format (1-on 0-off), default 1");
>>> -
>>> #define ISC_IS_FORMAT_RAW(mbus_code) \
>>> (((mbus_code) & 0xf000) == 0x3000)
>>>
>>> @@ -337,6 +332,10 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)
>>> unsigned long flags;
>>> int ret;
>>>
>>> + ret = media_pipeline_start(&isc->video_dev.entity, &isc->mpipe);
>>
>> The pipeline validation is done in start_streaming, but I don't think that
>> is the best place: if STREAMON is called before buffers are queued, then
>> an invalid pipeline isn't discovered until enough buffers are queued to
>> kick off start_streaming.
>>
>> Drivers like vsp1, omap3isp and the samsung drivers all do this in streamon().
>>
>> I think that is the correct time to do this.
>
> Hello Hans,
>
> Initially (v2, v3) I had this in streamon(). The problem that I faced at
> that time was that streamoff was never called, so I could not call
> media_pipeline_stop(). Then Jacopo told me to move it to start_streaming
> (see change log for v4) , and I did not face any more problems.

It's a mess. Looking at some drivers I see that omap3isp calls media_pipeline_stop
in streamoff (so will have the same problem as you described if VIDIOC_STREAMOFF
isn't called), exynos4-is does the same, but it also checks the streaming state in
the release() fop callback, so that would fix this problem. And vimc does this
in stop_streaming.

I'm in favor of fixing this in vb2, that framework knows exactly when this needs
to be called.

I'd like to know what others think.

Regards,

Hans

>
> Eugen
>
>>
>> Laurent, Jacopo, do you agree with that?
>>
>> I notice that quite a few other drivers do this in start_streaming as well,
>> but I think it just weird that a QBUF can return EPIPE while STREAMON might
>> succeed.
>>
>> Perhaps we need an optional vb2 validate() callback that is called by
>> vb2_core_streamon() to validate the pipeline before streaming. That makes
>> it easier to implement these checks rather than having to implement your
>> own streamon ioctl.
>>
>> Regards,
>>
>> Hans
>>
>>> + if (ret)
>>> + goto err_pipeline_start;
>>> +
>>> /* Enable stream on the sub device */
>>> ret = v4l2_subdev_call(isc->current_subdev->sd, video, s_stream, 1);
>>> if (ret && ret != -ENOIOCTLCMD) {
>>> @@ -385,6 +384,9 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)
>>> v4l2_subdev_call(isc->current_subdev->sd, video, s_stream, 0);
>>>
>>> err_start_stream:
>>> + media_pipeline_stop(&isc->video_dev.entity);
>>> +
>>> +err_pipeline_start:
>>> spin_lock_irqsave(&isc->dma_queue_lock, flags);
>>> list_for_each_entry(buf, &isc->dma_queue, list)
>>> vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_QUEUED);
>>> @@ -423,6 +425,9 @@ static void isc_stop_streaming(struct vb2_queue *vq)
>>> if (ret && ret != -ENOIOCTLCMD)
>>> v4l2_err(&isc->v4l2_dev, "stream off failed in subdev\n");
>>>
>>> + /* Stop media pipeline */
>>> + media_pipeline_stop(&isc->video_dev.entity);
>>> +
>>> /* Release all active buffers */
>>> spin_lock_irqsave(&isc->dma_queue_lock, flags);
>>> if (unlikely(isc->cur_frm)) {
>>> @@ -453,22 +458,6 @@ static void isc_buffer_queue(struct vb2_buffer *vb)
>>> spin_unlock_irqrestore(&isc->dma_queue_lock, flags);
>>> }
>>>
>>> -static struct isc_format *find_format_by_fourcc(struct isc_device *isc,
>>> - unsigned int fourcc)
>>> -{
>>> - unsigned int num_formats = isc->num_user_formats;
>>> - struct isc_format *fmt;
>>> - unsigned int i;
>>> -
>>> - for (i = 0; i < num_formats; i++) {
>>> - fmt = isc->user_formats[i];
>>> - if (fmt->fourcc == fourcc)
>>> - return fmt;
>>> - }
>>> -
>>> - return NULL;
>>> -}
>>> -
>>> static const struct vb2_ops isc_vb2_ops = {
>>> .queue_setup = isc_queue_setup,
>>> .wait_prepare = vb2_ops_wait_prepare,
>>> @@ -497,23 +486,57 @@ static int isc_enum_fmt_vid_cap(struct file *file, void *priv,
>>> {
>>> struct isc_device *isc = video_drvdata(file);
>>> u32 index = f->index;
>>> - u32 i, supported_index;
>>> + u32 i, supported_index = 0;
>>> + struct isc_format *fmt;
>>> +
>>> + /*
>>> + * If we are not asked a specific mbus_code, we have to report all
>>> + * the formats that we can output.
>>> + */
>>> + if (!f->mbus_code) {
>>> + if (index >= isc->controller_formats_size)
>>> + return -EINVAL;
>>>
>>> - if (index < isc->controller_formats_size) {
>>> f->pixelformat = isc->controller_formats[index].fourcc;
>>> +
>>> + return 0;
>>> + }
>>> +
>>> + /*
>>> + * If a specific mbus_code is requested, check if we support
>>> + * this mbus_code as input for the ISC.
>>> + * If it's supported, then we report the corresponding pixelformat
>>> + * as first possible option for the ISC.
>>> + * E.g. mbus MEDIA_BUS_FMT_YUYV8_2X8 and report
>>> + * 'YUYV' (YUYV 4:2:2)
>>> + */
>>> + fmt = isc_find_format_by_code(isc, f->mbus_code, &i);
>>> + if (!fmt)
>>> + return -EINVAL;
>>> +
>>> + if (!index) {
>>> + f->pixelformat = fmt->fourcc;
>>> +
>>> return 0;
>>> }
>>>
>>> - index -= isc->controller_formats_size;
>>> + supported_index++;
>>>
>>> - supported_index = 0;
>>> + /* If the index is not raw, we don't have anymore formats to report */
>>> + if (!ISC_IS_FORMAT_RAW(f->mbus_code))
>>> + return -EINVAL;
>>>
>>> - for (i = 0; i < isc->formats_list_size; i++) {
>>> - if (!ISC_IS_FORMAT_RAW(isc->formats_list[i].mbus_code) ||
>>> - !isc->formats_list[i].sd_support)
>>> + /*
>>> + * We are asked for a specific mbus code, which is raw.
>>> + * We have to search through the formats we can convert to.
>>> + * We have to skip the raw formats, we cannot convert to raw.
>>> + * E.g. 'AR12' (16-bit ARGB 4-4-4-4), 'AR15' (16-bit ARGB 1-5-5-5), etc.
>>> + */
>>> + for (i = 0; i < isc->controller_formats_size; i++) {
>>> + if (isc->controller_formats[i].raw)
>>> continue;
>>> - if (supported_index == index) {
>>> - f->pixelformat = isc->formats_list[i].fourcc;
>>> + if (index == supported_index) {
>>> + f->pixelformat = isc->controller_formats[i].fourcc;
>>> return 0;
>>> }
>>> supported_index++;
>>> @@ -584,20 +607,30 @@ static int isc_try_validate_formats(struct isc_device *isc)
>>> break;
>>> default:
>>> /* any other different formats are not supported */
>>> + v4l2_err(&isc->v4l2_dev, "Requested unsupported format.\n");
>>> ret = -EINVAL;
>>> }
>>> v4l2_dbg(1, debug, &isc->v4l2_dev,
>>> "Format validation, requested rgb=%u, yuv=%u, grey=%u, bayer=%u\n",
>>> rgb, yuv, grey, bayer);
>>>
>>> - /* we cannot output RAW if we do not receive RAW */
>>> - if ((bayer) && !ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code))
>>> + if (bayer &&
>>> + !ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code)) {
>>> + v4l2_err(&isc->v4l2_dev, "Cannot output RAW if we do not receive RAW.\n");
>>> return -EINVAL;
>>> + }
>>>
>>> - /* we cannot output GREY if we do not receive RAW/GREY */
>>> if (grey && !ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code) &&
>>> - !ISC_IS_FORMAT_GREY(isc->try_config.sd_format->mbus_code))
>>> + !ISC_IS_FORMAT_GREY(isc->try_config.sd_format->mbus_code)) {
>>> + v4l2_err(&isc->v4l2_dev, "Cannot output GREY if we do not receive RAW/GREY.\n");
>>> return -EINVAL;
>>> + }
>>> +
>>> + if ((rgb || bayer || yuv) &&
>>> + ISC_IS_FORMAT_GREY(isc->try_config.sd_format->mbus_code)) {
>>> + v4l2_err(&isc->v4l2_dev, "Cannot convert GREY to another format.\n");
>>> + return -EINVAL;
>>> + }
>>>
>>> return ret;
>>> }
>>> @@ -825,7 +858,7 @@ static void isc_try_fse(struct isc_device *isc,
>>> * If we do not know yet which format the subdev is using, we cannot
>>> * do anything.
>>> */
>>> - if (!isc->try_config.sd_format)
>>> + if (!isc->config.sd_format)
>>> return;
>>>
>>> fse.code = isc->try_config.sd_format->mbus_code;
>>> @@ -846,180 +879,140 @@ static void isc_try_fse(struct isc_device *isc,
>>> }
>>> }
>>>
>>> -static int isc_try_fmt(struct isc_device *isc, struct v4l2_format *f,
>>> - u32 *code)
>>> +static int isc_try_fmt(struct isc_device *isc, struct v4l2_format *f)
>>> {
>>> - int i;
>>> - struct isc_format *sd_fmt = NULL, *direct_fmt = NULL;
>>> struct v4l2_pix_format *pixfmt = &f->fmt.pix;
>>> - struct v4l2_subdev_pad_config pad_cfg = {};
>>> - struct v4l2_subdev_state pad_state = {
>>> - .pads = &pad_cfg
>>> - };
>>> - struct v4l2_subdev_format format = {
>>> - .which = V4L2_SUBDEV_FORMAT_TRY,
>>> - };
>>> - u32 mbus_code;
>>> - int ret;
>>> - bool rlp_dma_direct_dump = false;
>>> + unsigned int i;
>>>
>>> if (f->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
>>> return -EINVAL;
>>>
>>> - /* Step 1: find a RAW format that is supported */
>>> - for (i = 0; i < isc->num_user_formats; i++) {
>>> - if (ISC_IS_FORMAT_RAW(isc->user_formats[i]->mbus_code)) {
>>> - sd_fmt = isc->user_formats[i];
>>> + isc->try_config.fourcc = isc->controller_formats[0].fourcc;
>>> +
>>> + /* find if the format requested is supported */
>>> + for (i = 0; i < isc->controller_formats_size; i++)
>>> + if (isc->controller_formats[i].fourcc == pixfmt->pixelformat) {
>>> + isc->try_config.fourcc = pixfmt->pixelformat;
>>> break;
>>> }
>>> - }
>>> - /* Step 2: We can continue with this RAW format, or we can look
>>> - * for better: maybe sensor supports directly what we need.
>>> - */
>>> - direct_fmt = find_format_by_fourcc(isc, pixfmt->pixelformat);
>>> -
>>> - /* Step 3: We have both. We decide given the module parameter which
>>> - * one to use.
>>> - */
>>> - if (direct_fmt && sd_fmt && sensor_preferred)
>>> - sd_fmt = direct_fmt;
>>> -
>>> - /* Step 4: we do not have RAW but we have a direct format. Use it. */
>>> - if (direct_fmt && !sd_fmt)
>>> - sd_fmt = direct_fmt;
>>> -
>>> - /* Step 5: if we are using a direct format, we need to package
>>> - * everything as 8 bit data and just dump it
>>> - */
>>> - if (sd_fmt == direct_fmt)
>>> - rlp_dma_direct_dump = true;
>>> -
>>> - /* Step 6: We have no format. This can happen if the userspace
>>> - * requests some weird/invalid format.
>>> - * In this case, default to whatever we have
>>> - */
>>> - if (!sd_fmt && !direct_fmt) {
>>> - sd_fmt = isc->user_formats[isc->num_user_formats - 1];
>>> - v4l2_dbg(1, debug, &isc->v4l2_dev,
>>> - "Sensor not supporting %.4s, using %.4s\n",
>>> - (char *)&pixfmt->pixelformat, (char *)&sd_fmt->fourcc);
>>> - }
>>> -
>>> - if (!sd_fmt) {
>>> - ret = -EINVAL;
>>> - goto isc_try_fmt_err;
>>> - }
>>> -
>>> - /* Step 7: Print out what we decided for debugging */
>>> - v4l2_dbg(1, debug, &isc->v4l2_dev,
>>> - "Preferring to have sensor using format %.4s\n",
>>> - (char *)&sd_fmt->fourcc);
>>>
>>> - /* Step 8: at this moment we decided which format the subdev will use */
>>> - isc->try_config.sd_format = sd_fmt;
>>> + isc_try_configure_rlp_dma(isc, false);
>>>
>>> /* Limit to Atmel ISC hardware capabilities */
>>> - if (pixfmt->width > isc->max_width)
>>> - pixfmt->width = isc->max_width;
>>> - if (pixfmt->height > isc->max_height)
>>> - pixfmt->height = isc->max_height;
>>> -
>>> - /*
>>> - * The mbus format is the one the subdev outputs.
>>> - * The pixels will be transferred in this format Sensor -> ISC
>>> - */
>>> - mbus_code = sd_fmt->mbus_code;
>>> -
>>> - /*
>>> - * Validate formats. If the required format is not OK, default to raw.
>>> - */
>>> -
>>> - isc->try_config.fourcc = pixfmt->pixelformat;
>>> -
>>> - if (isc_try_validate_formats(isc)) {
>>> - pixfmt->pixelformat = isc->try_config.fourcc = sd_fmt->fourcc;
>>> - /* Re-try to validate the new format */
>>> - ret = isc_try_validate_formats(isc);
>>> - if (ret)
>>> - goto isc_try_fmt_err;
>>> - }
>>> -
>>> - ret = isc_try_configure_rlp_dma(isc, rlp_dma_direct_dump);
>>> - if (ret)
>>> - goto isc_try_fmt_err;
>>> -
>>> - ret = isc_try_configure_pipeline(isc);
>>> - if (ret)
>>> - goto isc_try_fmt_err;
>>> -
>>> - /* Obtain frame sizes if possible to have crop requirements ready */
>>> - isc_try_fse(isc, &pad_state);
>>> -
>>> - v4l2_fill_mbus_format(&format.format, pixfmt, mbus_code);
>>> - ret = v4l2_subdev_call(isc->current_subdev->sd, pad, set_fmt,
>>> - &pad_state, &format);
>>> - if (ret < 0)
>>> - goto isc_try_fmt_subdev_err;
>>> -
>>> - v4l2_fill_pix_format(pixfmt, &format.format);
>>> + v4l_bound_align_image(&pixfmt->width, 16, isc->max_width, 0,
>>> + &pixfmt->height, 16, isc->max_height, 0, 0);
>>> + /* If we did not find the requested format, we will fallback here */
>>> + pixfmt->pixelformat = isc->try_config.fourcc;
>>> + pixfmt->colorspace = V4L2_COLORSPACE_SRGB;
>>> + pixfmt->field = V4L2_FIELD_NONE;
>>>
>>> - /* Limit to Atmel ISC hardware capabilities */
>>> - if (pixfmt->width > isc->max_width)
>>> - pixfmt->width = isc->max_width;
>>> - if (pixfmt->height > isc->max_height)
>>> - pixfmt->height = isc->max_height;
>>>
>>> - pixfmt->field = V4L2_FIELD_NONE;
>>> pixfmt->bytesperline = (pixfmt->width * isc->try_config.bpp_v4l2) >> 3;
>>> pixfmt->sizeimage = ((pixfmt->width * isc->try_config.bpp) >> 3) *
>>> pixfmt->height;
>>>
>>> - if (code)
>>> - *code = mbus_code;
>>> + isc->try_fmt = *f;
>>>
>>> return 0;
>>> +}
>>>
>>> -isc_try_fmt_err:
>>> - v4l2_err(&isc->v4l2_dev, "Could not find any possible format for a working pipeline\n");
>>> -isc_try_fmt_subdev_err:
>>> - memset(&isc->try_config, 0, sizeof(isc->try_config));
>>> +static int isc_set_fmt(struct isc_device *isc, struct v4l2_format *f)
>>> +{
>>> + isc_try_fmt(isc, f);
>>>
>>> - return ret;
>>> + /* make the try configuration active */
>>> + isc->config = isc->try_config;
>>> + isc->fmt = isc->try_fmt;
>>> +
>>> + v4l2_dbg(1, debug, &isc->v4l2_dev, "ISC set_fmt to %.4s @%dx%d\n",
>>> + (char *)&f->fmt.pix.pixelformat,
>>> + f->fmt.pix.width, f->fmt.pix.height);
>>> +
>>> + return 0;
>>> }
>>>
>>> -static int isc_set_fmt(struct isc_device *isc, struct v4l2_format *f)
>>> +static int isc_validate(struct isc_device *isc)
>>> {
>>> + int ret;
>>> + int i;
>>> + struct isc_format *sd_fmt = NULL;
>>> + struct v4l2_pix_format *pixfmt = &isc->fmt.fmt.pix;
>>> struct v4l2_subdev_format format = {
>>> .which = V4L2_SUBDEV_FORMAT_ACTIVE,
>>> + .pad = isc->remote_pad,
>>> + };
>>> + struct v4l2_subdev_pad_config pad_cfg = {};
>>> + struct v4l2_subdev_state pad_state = {
>>> + .pads = &pad_cfg,
>>> };
>>> - u32 mbus_code = 0;
>>> - int ret;
>>>
>>> - ret = isc_try_fmt(isc, f, &mbus_code);
>>> + /* Get current format from subdev */
>>> + ret = v4l2_subdev_call(isc->current_subdev->sd, pad, get_fmt, NULL,
>>> + &format);
>>> if (ret)
>>> return ret;
>>>
>>> - v4l2_fill_mbus_format(&format.format, &f->fmt.pix, mbus_code);
>>> - ret = v4l2_subdev_call(isc->current_subdev->sd, pad,
>>> - set_fmt, NULL, &format);
>>> - if (ret < 0)
>>> - return ret;
>>> + /* Identify the subdev's format configuration */
>>> + for (i = 0; i < isc->formats_list_size; i++)
>>> + if (isc->formats_list[i].mbus_code == format.format.code) {
>>> + sd_fmt = &isc->formats_list[i];
>>> + break;
>>> + }
>>> +
>>> + /* Check if the format is not supported */
>>> + if (!sd_fmt) {
>>> + v4l2_err(&isc->v4l2_dev,
>>> + "Current subdevice is streaming a media bus code that is not supported 0x%x\n",
>>> + format.format.code);
>>> + return -EPIPE;
>>> + }
>>> +
>>> + /* At this moment we know which format the subdev will use */
>>> + isc->try_config.sd_format = sd_fmt;
>>> +
>>> + /* If the sensor is not RAW, we can only do a direct dump */
>>> + if (!ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code))
>>> + isc_try_configure_rlp_dma(isc, true);
>>>
>>> /* Limit to Atmel ISC hardware capabilities */
>>> - if (f->fmt.pix.width > isc->max_width)
>>> - f->fmt.pix.width = isc->max_width;
>>> - if (f->fmt.pix.height > isc->max_height)
>>> - f->fmt.pix.height = isc->max_height;
>>> + v4l_bound_align_image(&format.format.width, 16, isc->max_width, 0,
>>> + &format.format.height, 16, isc->max_height, 0, 0);
>>>
>>> - isc->fmt = *f;
>>> + /* Check if the frame size is the same. Otherwise we may overflow */
>>> + if (pixfmt->height != format.format.height ||
>>> + pixfmt->width != format.format.width) {
>>> + v4l2_err(&isc->v4l2_dev,
>>> + "ISC not configured with the proper frame size: %dx%d\n",
>>> + format.format.width, format.format.height);
>>> + return -EPIPE;
>>> + }
>>>
>>> + v4l2_dbg(1, debug, &isc->v4l2_dev,
>>> + "Identified subdev using format %.4s with %dx%d %d bpp\n",
>>> + (char *)&sd_fmt->fourcc, pixfmt->width, pixfmt->height,
>>> + isc->try_config.bpp);
>>> +
>>> + /* Reset and restart AWB if the subdevice changed the format */
>>> if (isc->try_config.sd_format && isc->config.sd_format &&
>>> isc->try_config.sd_format != isc->config.sd_format) {
>>> isc->ctrls.hist_stat = HIST_INIT;
>>> isc_reset_awb_ctrls(isc);
>>> isc_update_v4l2_ctrls(isc);
>>> }
>>> - /* make the try configuration active */
>>> +
>>> + /* Validate formats */
>>> + ret = isc_try_validate_formats(isc);
>>> + if (ret)
>>> + return ret;
>>> +
>>> + /* Obtain frame sizes if possible to have crop requirements ready */
>>> + isc_try_fse(isc, &pad_state);
>>> +
>>> + /* Configure ISC pipeline for the config */
>>> + ret = isc_try_configure_pipeline(isc);
>>> + if (ret)
>>> + return ret;
>>> +
>>> isc->config = isc->try_config;
>>>
>>> v4l2_dbg(1, debug, &isc->v4l2_dev, "New ISC configuration in place\n");
>>> @@ -1043,7 +1036,7 @@ static int isc_try_fmt_vid_cap(struct file *file, void *priv,
>>> {
>>> struct isc_device *isc = video_drvdata(file);
>>>
>>> - return isc_try_fmt(isc, f, NULL);
>>> + return isc_try_fmt(isc, f);
>>> }
>>>
>>> static int isc_enum_input(struct file *file, void *priv,
>>> @@ -1098,10 +1091,6 @@ static int isc_enum_framesizes(struct file *file, void *fh,
>>> if (fsize->index)
>>> return -EINVAL;
>>>
>>> - for (i = 0; i < isc->num_user_formats; i++)
>>> - if (isc->user_formats[i]->fourcc == fsize->pixel_format)
>>> - ret = 0;
>>> -
>>> for (i = 0; i < isc->controller_formats_size; i++)
>>> if (isc->controller_formats[i].fourcc == fsize->pixel_format)
>>> ret = 0;
>>> @@ -1782,52 +1771,6 @@ struct isc_format *isc_find_format_by_code(struct isc_device *isc,
>>> }
>>> EXPORT_SYMBOL_GPL(isc_find_format_by_code);
>>>
>>> -static int isc_formats_init(struct isc_device *isc)
>>> -{
>>> - struct isc_format *fmt;
>>> - struct v4l2_subdev *subdev = isc->current_subdev->sd;
>>> - unsigned int num_fmts, i, j;
>>> - u32 list_size = isc->formats_list_size;
>>> - struct v4l2_subdev_mbus_code_enum mbus_code = {
>>> - .which = V4L2_SUBDEV_FORMAT_ACTIVE,
>>> - };
>>> -
>>> - num_fmts = 0;
>>> - while (!v4l2_subdev_call(subdev, pad, enum_mbus_code,
>>> - NULL, &mbus_code)) {
>>> - mbus_code.index++;
>>> -
>>> - fmt = isc_find_format_by_code(isc, mbus_code.code, &i);
>>> - if (!fmt) {
>>> - v4l2_warn(&isc->v4l2_dev, "Mbus code %x not supported\n",
>>> - mbus_code.code);
>>> - continue;
>>> - }
>>> -
>>> - fmt->sd_support = true;
>>> - num_fmts++;
>>> - }
>>> -
>>> - if (!num_fmts)
>>> - return -ENXIO;
>>> -
>>> - isc->num_user_formats = num_fmts;
>>> - isc->user_formats = devm_kcalloc(isc->dev,
>>> - num_fmts, sizeof(*isc->user_formats),
>>> - GFP_KERNEL);
>>> - if (!isc->user_formats)
>>> - return -ENOMEM;
>>> -
>>> - fmt = &isc->formats_list[0];
>>> - for (i = 0, j = 0; i < list_size; i++) {
>>> - if (fmt->sd_support)
>>> - isc->user_formats[j++] = fmt;
>>> - fmt++;
>>> - }
>>> -
>>> - return 0;
>>> -}
>>> -
>>> static int isc_set_default_fmt(struct isc_device *isc)
>>> {
>>> struct v4l2_format f = {
>>> @@ -1836,12 +1779,12 @@ static int isc_set_default_fmt(struct isc_device *isc)
>>> .width = VGA_WIDTH,
>>> .height = VGA_HEIGHT,
>>> .field = V4L2_FIELD_NONE,
>>> - .pixelformat = isc->user_formats[0]->fourcc,
>>> + .pixelformat = isc->controller_formats[0].fourcc,
>>> },
>>> };
>>> int ret;
>>>
>>> - ret = isc_try_fmt(isc, &f, NULL);
>>> + ret = isc_try_fmt(isc, &f);
>>> if (ret)
>>> return ret;
>>>
>>> @@ -1896,13 +1839,6 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier)
>>> spin_lock_init(&isc->dma_queue_lock);
>>> spin_lock_init(&isc->awb_lock);
>>>
>>> - ret = isc_formats_init(isc);
>>> - if (ret < 0) {
>>> - v4l2_err(&isc->v4l2_dev,
>>> - "Init format failed: %d\n", ret);
>>> - goto isc_async_complete_err;
>>> - }
>>> -
>>> ret = isc_set_default_fmt(isc);
>>> if (ret) {
>>> v4l2_err(&isc->v4l2_dev, "Could not set default format\n");
>>> @@ -2016,6 +1952,24 @@ int isc_pipeline_init(struct isc_device *isc)
>>> }
>>> EXPORT_SYMBOL_GPL(isc_pipeline_init);
>>>
>>> +static int isc_link_validate(struct media_link *link)
>>> +{
>>> + struct video_device *vdev =
>>> + media_entity_to_video_device(link->sink->entity);
>>> + struct isc_device *isc = video_get_drvdata(vdev);
>>> + int ret;
>>> +
>>> + ret = v4l2_subdev_link_validate(link);
>>> + if (ret)
>>> + return ret;
>>> +
>>> + return isc_validate(isc);
>>> +}
>>> +
>>> +static const struct media_entity_operations isc_entity_operations = {
>>> + .link_validate = isc_link_validate,
>>> +};
>>> +
>>> int isc_mc_init(struct isc_device *isc, u32 ver)
>>> {
>>> const struct of_device_id *match;
>>> @@ -2023,6 +1977,8 @@ int isc_mc_init(struct isc_device *isc, u32 ver)
>>>
>>> isc->video_dev.entity.function = MEDIA_ENT_F_IO_V4L;
>>> isc->video_dev.entity.flags = MEDIA_ENT_FL_DEFAULT;
>>> + isc->video_dev.entity.ops = &isc_entity_operations;
>>> +
>>> isc->pads[ISC_PAD_SINK].flags = MEDIA_PAD_FL_SINK;
>>>
>>> ret = media_entity_pads_init(&isc->video_dev.entity, ISC_PADS_NUM,
>>> diff --git a/drivers/media/platform/atmel/atmel-isc-scaler.c b/drivers/media/platform/atmel/atmel-isc-scaler.c
>>> index a1ca4633787c..2b0621297dfe 100644
>>> --- a/drivers/media/platform/atmel/atmel-isc-scaler.c
>>> +++ b/drivers/media/platform/atmel/atmel-isc-scaler.c
>>> @@ -173,6 +173,10 @@ static const struct v4l2_subdev_pad_ops isc_scaler_pad_ops = {
>>> .init_cfg = isc_scaler_init_cfg,
>>> };
>>>
>>> +static const struct media_entity_operations isc_scaler_entity_ops = {
>>> + .link_validate = v4l2_subdev_link_validate,
>>> +};
>>> +
>>> static const struct v4l2_subdev_ops xisc_scaler_subdev_ops = {
>>> .pad = &isc_scaler_pad_ops,
>>> };
>>> @@ -190,6 +194,7 @@ int isc_scaler_init(struct isc_device *isc)
>>>
>>> isc->scaler_sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
>>> isc->scaler_sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
>>> + isc->scaler_sd.entity.ops = &isc_scaler_entity_ops;
>>> isc->scaler_pads[ISC_SCALER_PAD_SINK].flags = MEDIA_PAD_FL_SINK;
>>> isc->scaler_pads[ISC_SCALER_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
>>>
>>> diff --git a/drivers/media/platform/atmel/atmel-isc.h b/drivers/media/platform/atmel/atmel-isc.h
>>> index f98f25a55e73..5b0100c8c245 100644
>>> --- a/drivers/media/platform/atmel/atmel-isc.h
>>> +++ b/drivers/media/platform/atmel/atmel-isc.h
>>> @@ -63,15 +63,16 @@ struct isc_subdev_entity {
>>> * @cfa_baycfg: If this format is RAW BAYER, indicate the type of bayer.
>>> this is either BGBG, RGRG, etc.
>>> * @pfe_cfg0_bps: Number of hardware data lines connected to the ISC
>>> + * @raw: If the format is raw bayer.
>>> */
>>>
>>> struct isc_format {
>>> u32 fourcc;
>>> u32 mbus_code;
>>> u32 cfa_baycfg;
>>> -
>>> - bool sd_support;
>>> u32 pfe_cfg0_bps;
>>> +
>>> + bool raw;
>>> };
>>>
>>> /* Pipeline bitmap */
>>> @@ -216,8 +217,7 @@ enum isc_scaler_pads {
>>> * @comp: completion reference that signals frame completion
>>> *
>>> * @fmt: current v42l format
>>> - * @user_formats: list of formats that are supported and agreed with sd
>>> - * @num_user_formats: how many formats are in user_formats
>>> + * @try_fmt: current v4l2 try format
>>> *
>>> * @config: current ISC format configuration
>>> * @try_config: the current ISC try format , not yet activated
>>> @@ -272,6 +272,7 @@ enum isc_scaler_pads {
>>> * @formats_list_size: size of formats_list array
>>> * @pads: media controller pads for isc video entity
>>> * @mdev: media device that is registered by the isc
>>> + * @mpipe: media device pipeline used by the isc
>>> * @remote_pad: remote pad on the connected subdevice
>>> * @scaler_sd: subdevice for the scaler that isc registers
>>> * @scaler_pads: media controller pads for the scaler subdevice
>>> @@ -298,8 +299,7 @@ struct isc_device {
>>> struct completion comp;
>>>
>>> struct v4l2_format fmt;
>>> - struct isc_format **user_formats;
>>> - unsigned int num_user_formats;
>>> + struct v4l2_format try_fmt;
>>>
>>> struct fmt_config config;
>>> struct fmt_config try_config;
>>> @@ -369,6 +369,7 @@ struct isc_device {
>>> struct {
>>> struct media_pad pads[ISC_PADS_NUM];
>>> struct media_device mdev;
>>> + struct media_pipeline mpipe;
>>>
>>> u32 remote_pad;
>>> };
>>> diff --git a/drivers/media/platform/atmel/atmel-sama5d2-isc.c b/drivers/media/platform/atmel/atmel-sama5d2-isc.c
>>> index d96ee3373889..2a651e237193 100644
>>> --- a/drivers/media/platform/atmel/atmel-sama5d2-isc.c
>>> +++ b/drivers/media/platform/atmel/atmel-sama5d2-isc.c
>>> @@ -80,20 +80,40 @@ static const struct isc_format sama5d2_controller_formats[] = {
>>> .fourcc = V4L2_PIX_FMT_Y10,
>>> }, {
>>> .fourcc = V4L2_PIX_FMT_SBGGR8,
>>> + .raw = true,
>>> }, {
>>> .fourcc = V4L2_PIX_FMT_SGBRG8,
>>> + .raw = true,
>>> }, {
>>> .fourcc = V4L2_PIX_FMT_SGRBG8,
>>> + .raw = true,
>>> }, {
>>> .fourcc = V4L2_PIX_FMT_SRGGB8,
>>> + .raw = true,
>>> }, {
>>> .fourcc = V4L2_PIX_FMT_SBGGR10,
>>> + .raw = true,
>>> }, {
>>> .fourcc = V4L2_PIX_FMT_SGBRG10,
>>> + .raw = true,
>>> }, {
>>> .fourcc = V4L2_PIX_FMT_SGRBG10,
>>> + .raw = true,
>>> }, {
>>> .fourcc = V4L2_PIX_FMT_SRGGB10,
>>> + .raw = true,
>>> + }, {
>>> + .fourcc = V4L2_PIX_FMT_SBGGR12,
>>> + .raw = true,
>>> + }, {
>>> + .fourcc = V4L2_PIX_FMT_SGBRG12,
>>> + .raw = true,
>>> + }, {
>>> + .fourcc = V4L2_PIX_FMT_SGRBG12,
>>> + .raw = true,
>>> + }, {
>>> + .fourcc = V4L2_PIX_FMT_SRGGB12,
>>> + .raw = true,
>>> },
>>> };
>>>
>>> diff --git a/drivers/media/platform/atmel/atmel-sama7g5-isc.c b/drivers/media/platform/atmel/atmel-sama7g5-isc.c
>>> index e07ae188c15f..a0d60cfdba7b 100644
>>> --- a/drivers/media/platform/atmel/atmel-sama7g5-isc.c
>>> +++ b/drivers/media/platform/atmel/atmel-sama7g5-isc.c
>>> @@ -89,20 +89,40 @@ static const struct isc_format sama7g5_controller_formats[] = {
>>> .fourcc = V4L2_PIX_FMT_Y16,
>>> }, {
>>> .fourcc = V4L2_PIX_FMT_SBGGR8,
>>> + .raw = true,
>>> }, {
>>> .fourcc = V4L2_PIX_FMT_SGBRG8,
>>> + .raw = true,
>>> }, {
>>> .fourcc = V4L2_PIX_FMT_SGRBG8,
>>> + .raw = true,
>>> }, {
>>> .fourcc = V4L2_PIX_FMT_SRGGB8,
>>> + .raw = true,
>>> }, {
>>> .fourcc = V4L2_PIX_FMT_SBGGR10,
>>> + .raw = true,
>>> }, {
>>> .fourcc = V4L2_PIX_FMT_SGBRG10,
>>> + .raw = true,
>>> }, {
>>> .fourcc = V4L2_PIX_FMT_SGRBG10,
>>> + .raw = true,
>>> }, {
>>> .fourcc = V4L2_PIX_FMT_SRGGB10,
>>> + .raw = true,
>>> + }, {
>>> + .fourcc = V4L2_PIX_FMT_SBGGR12,
>>> + .raw = true,
>>> + }, {
>>> + .fourcc = V4L2_PIX_FMT_SGBRG12,
>>> + .raw = true,
>>> + }, {
>>> + .fourcc = V4L2_PIX_FMT_SGRBG12,
>>> + .raw = true,
>>> + }, {
>>> + .fourcc = V4L2_PIX_FMT_SRGGB12,
>>> + .raw = true,
>>> },
>>> };
>>>
>>
>

2022-05-02 11:13:17

[permalink] [raw]

Subject: Re: [PATCH v9 00/13] media: atmel: atmel-isc: implement media controller

Hi Eugen,

On 10/03/2022 10:51, Eugen Hristev wrote:
> This series is the v9 series that attempts to support media controller in the
> atmel ISC and XISC drivers.
> The CSI2DC driver was accepted thus removed from the patch series, together with
> other patches.
>
> Important note: this series applies on top of current media_staging tree, as it
> relies on previous patches in the series which were accepted.
>
> Thanks to everyone who reviewed my work !
>
> Eugen
>
> Changes in v9:
> -> kernel robot reported isc_link_validate is not static, changed to static.
>
> Changes in v8:
> -> scaler: modified crop bounds to have the exact source size
>
> Changes in v7:
> -> scaler: modified crop bounds to have maximum isc size
> -> format propagation: did small changes as per Jacopo review
>
>
> Changes in v6:
> -> worked a bit on scaler, added try crop and other changes as per Jacopo review
> -> worked on isc-base enum_fmt , reworked as per Jacopo review
>
> Changes in v5:
> -> removed patch that removed the 'stop' variable as it was still required
> -> added two new trivial patches
> -> reworked some parts of the scaler and format propagation after discussions with Jacopo
>
>
> Changes in v4:
> -> as reviewed by Hans, added new patch to remove the 'stop' variable and reworked
> one patch that was using it
> -> as reviewed by Jacopo, reworked some parts of the media controller implementation
>
>
> Changes in v3:
> - change in bindings, small fixes in csi2dc driver and conversion to mc
> for the isc-base.
> - removed some MAINTAINERS patches and used patterns in MAINTAINERS
>
> Changes in v2:
> - integrated many changes suggested by Jacopo in the review of the v1 series.
> - add a few new patches
>
>
>
> Eugen Hristev (13):
> media: atmel: atmel-isc-base: use streaming status when queueing
> buffers
> media: atmel: atmel-isc-base: replace is_streaming call in
> s_fmt_vid_cap
> media: atmel: atmel-isc: remove redundant comments
> media: atmel: atmel-isc: implement media controller
> media: atmel: atmel-sama5d2-isc: fix wrong mask in YUYV format check
> media: atmel: atmel-isc-base: use mutex to lock awb workqueue from
> streaming
> media: atmel: atmel-isc: compact the controller formats list
> media: atmel: atmel-isc: change format propagation to subdev into only
> verification
> media: atmel: atmel-sama7g5-isc: remove stray line
> dt-bindings: media: microchip,xisc: add bus-width of 14

I'm a bit unhappy with the order of these patches. Mostly these are fixes,
except for patches 4 and 8, which are the meat of this series and actually
switching on the MC support.

Can those be moved to the end? That also makes it easier to merge the earlier
patches if some more work is needed for the MC part.

I'm also not sure whether patches 4 and 8 shouldn't be a single patch,
since patch 4 leaves the driver in an inconsistent state since it is
missing the link validation code that patch 8 adds. Unless I missed
something?

Regards,

Hans

> ARM: dts: at91: sama7g5: add nodes for video capture
> ARM: configs: at91: sama7: add xisc and csi2dc
> ARM: multi_v7_defconfig: add atmel video pipeline modules
>
> .../bindings/media/microchip,xisc.yaml | 2 +-
> arch/arm/boot/dts/sama7g5.dtsi | 49 ++
> arch/arm/configs/multi_v7_defconfig | 3 +
> arch/arm/configs/sama7_defconfig | 2 +
> drivers/media/platform/atmel/Makefile | 2 +-
> drivers/media/platform/atmel/atmel-isc-base.c | 518 ++++++++++--------
> .../media/platform/atmel/atmel-isc-scaler.c | 267 +++++++++
> drivers/media/platform/atmel/atmel-isc.h | 58 +-
> .../media/platform/atmel/atmel-sama5d2-isc.c | 87 +--
> .../media/platform/atmel/atmel-sama7g5-isc.c | 93 ++--
> 10 files changed, 754 insertions(+), 327 deletions(-)
> create mode 100644 drivers/media/platform/atmel/atmel-isc-scaler.c
>

2022-05-02 22:45:11

[permalink] [raw]

Subject: Re: [PATCH v9 08/13] media: atmel: atmel-isc: change format propagation to subdev into only verification

On 29/04/2022 14:38, Jacopo Mondi wrote:
> Hi
>
> On Fri, Apr 29, 2022 at 03:28:11PM +0300, Laurent Pinchart wrote:
>> On Fri, Apr 29, 2022 at 02:23:45PM +0200, Hans Verkuil wrote:
>>> On 29/04/2022 14:02, Laurent Pinchart wrote:
>>>> On Fri, Apr 29, 2022 at 01:17:27PM +0200, Hans Verkuil wrote:
>>>>> On 29/04/2022 12:20, Laurent Pinchart wrote:
>>>>>> On Fri, Apr 29, 2022 at 12:13:46PM +0200, Hans Verkuil wrote:
>>>>>>> On 29/04/2022 12:07, Laurent Pinchart wrote:
>>>>>>>> On Fri, Apr 29, 2022 at 11:58:48AM +0200, Jacopo Mondi wrote:
>>>>>>>>> On Fri, Apr 29, 2022 at 10:43:09AM +0200, Hans Verkuil wrote:
>>>>>>>>>> On 29/04/2022 10:28, [email protected] wrote:
>>>>>>>>>>> On 4/29/22 11:17 AM, Hans Verkuil wrote:
>>>>>>>>>>>> On 10/03/2022 10:51, Eugen Hristev wrote:
>>>>>>>>>>>>> As a top MC video driver, the atmel-isc should not propagate the format to the
>>>>>>>>>>>>> subdevice, it should rather check at start_streaming() time if the subdev is properly
>>>>>>>>>>>>> configured with a compatible format.
>>>>>>>>>>>>> Removed the whole format finding logic, and reworked the format verification
>>>>>>>>>>>>> at start_streaming time, such that the ISC will return an error if the subdevice
>>>>>>>>>>>>> is not properly configured. To achieve this, media_pipeline_start
>>>>>>>>>>>>> is called and a link_validate callback is created to check the formats.
>>>>>>>>>>>>> With this being done, the module parameter 'sensor_preferred' makes no sense
>>>>>>>>>>>>> anymore. The ISC should not decide which format the sensor is using. The
>>>>>>>>>>>>> ISC should only cope with the situation and inform userspace if the streaming
>>>>>>>>>>>>> is possible in the current configuration.
>>>>>>>>>>>>> The redesign of the format propagation has also risen the question of the
>>>>>>>>>>>>> enumfmt callback. If enumfmt is called with an mbus_code, the enumfmt handler
>>>>>>>>>>>>> should only return the formats that are supported for this mbus_code.
>>>>>>>>>>>>> Otherwise, the enumfmt will report all the formats that the ISC could output.
>>>>>>>>>>>>> With this rework, the dynamic list of user formats is removed. It makes no
>>>>>>>>>>>>> more sense to identify at complete time which formats the sensor could emit,
>>>>>>>>>>>>> and add those into a separate dynamic list.
>>>>>>>>>>>>> The ISC will start with a simple preconfigured default format, and at
>>>>>>>>>>>>> link validate time, decide whether it can use the format that is configured
>>>>>>>>>>>>> on the sink or not.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Signed-off-by: Eugen Hristev <[email protected]>
>>>>>>>>>>>>> Reviewed-by: Jacopo Mondi <[email protected]>
>>>>>>>>>>>>> ---
>>>>>>>>>>>>> Changes in v9:
>>>>>>>>>>>>> - isc_link_validate now static
>>>>>>>>>>>>>
>>>>>>>>>>>>> Changes in v7:
>>>>>>>>>>>>> - minor typos as suggested by Jacopo
>>>>>>>>>>>>> - small changes, reduce some indentation, modified an index, as suggested by
>>>>>>>>>>>>> Jacopo
>>>>>>>>>>>>>
>>>>>>>>>>>>> Changes in v6:
>>>>>>>>>>>>> - reworked a bit enum_fmt as suggested by Jacopo
>>>>>>>>>>>>>
>>>>>>>>>>>>> Changes in v5:
>>>>>>>>>>>>> - removed user_formats dynamic list as it is now pointless
>>>>>>>>>>>>> - greatly simplified the enum_fmt function
>>>>>>>>>>>>> - removed some init code that was useless now
>>>>>>>>>>>>>
>>>>>>>>>>>>> Changes in v4:
>>>>>>>>>>>>> - moved validation code into link_validate and used media_pipeline_start
>>>>>>>>>>>>> - merged this patch with the enum_fmt patch which was previously in v3 of
>>>>>>>>>>>>> the series
>>>>>>>>>>>>>
>>>>>>>>>>>>> Changes in v3:
>>>>>>>>>>>>> - clamp to maximum resolution once the frame size from the subdev is found
>>>>>>>>>>>>> drivers/media/platform/atmel/atmel-isc-base.c | 412 ++++++++----------
>>>>>>>>>>>>> .../media/platform/atmel/atmel-isc-scaler.c | 5 +
>>>>>>>>>>>>> drivers/media/platform/atmel/atmel-isc.h | 13 +-
>>>>>>>>>>>>> .../media/platform/atmel/atmel-sama5d2-isc.c | 20 +
>>>>>>>>>>>>> .../media/platform/atmel/atmel-sama7g5-isc.c | 20 +
>>>>>>>>>>>>> 5 files changed, 236 insertions(+), 234 deletions(-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> diff --git a/drivers/media/platform/atmel/atmel-isc-base.c b/drivers/media/platform/atmel/atmel-isc-base.c
>>>>>>>>>>>>> index ee1dda6707a0..fe2c0af58060 100644
>>>>>>>>>>>>> --- a/drivers/media/platform/atmel/atmel-isc-base.c
>>>>>>>>>>>>> +++ b/drivers/media/platform/atmel/atmel-isc-base.c
>>>>>>>>>>>>> @@ -36,11 +36,6 @@ static unsigned int debug;
>>>>>>>>>>>>> module_param(debug, int, 0644);
>>>>>>>>>>>>> MODULE_PARM_DESC(debug, "debug level (0-2)");
>>>>>>>>>>>>>
>>>>>>>>>>>>> -static unsigned int sensor_preferred = 1;
>>>>>>>>>>>>> -module_param(sensor_preferred, uint, 0644);
>>>>>>>>>>>>> -MODULE_PARM_DESC(sensor_preferred,
>>>>>>>>>>>>> - "Sensor is preferred to output the specified format (1-on 0-off), default 1");
>>>>>>>>>>>>> -
>>>>>>>>>>>>> #define ISC_IS_FORMAT_RAW(mbus_code) \
>>>>>>>>>>>>> (((mbus_code) & 0xf000) == 0x3000)
>>>>>>>>>>>>>
>>>>>>>>>>>>> @@ -337,6 +332,10 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)
>>>>>>>>>>>>> unsigned long flags;
>>>>>>>>>>>>> int ret;
>>>>>>>>>>>>>
>>>>>>>>>>>>> + ret = media_pipeline_start(&isc->video_dev.entity, &isc->mpipe);
>>>>>>>>>>>>
>>>>>>>>>>>> The pipeline validation is done in start_streaming, but I don't think that
>>>>>>>>>>>> is the best place: if STREAMON is called before buffers are queued, then
>>>>>>>>>>>> an invalid pipeline isn't discovered until enough buffers are queued to
>>>>>>>>>>>> kick off start_streaming.
>>>>>>>>>>>>
>>>>>>>>>>>> Drivers like vsp1, omap3isp and the samsung drivers all do this in streamon().
>>>>>>>>>>>>
>>>>>>>>>>>> I think that is the correct time to do this.
>>>>>>>>>>>
>>>>>>>>>>> Hello Hans,
>>>>>>>>>>>
>>>>>>>>>>> Initially (v2, v3) I had this in streamon(). The problem that I faced at
>>>>>>>>>>> that time was that streamoff was never called, so I could not call
>>>>>>>>>>> media_pipeline_stop(). Then Jacopo told me to move it to start_streaming
>>>>>>>>>>> (see change log for v4) , and I did not face any more problems.
>>>>>>>>>
>>>>>>>>> Yes indeed, seems I suggested to use media_pipeline_handler in a
>>>>>>>>> comment on your v3
>>>>>>>>>
>>>>>>>>> "at s_stream time your top driver calls media_pipeline_start()"
>>>>>>>>>
>>>>>>>>> sorry about that, I should have looked around a bit more carefully and
>>>>>>>>> notice most drivers do so at vb2 streamon
>>>>>>>>>
>>>>>>>>> However I don't see media_pipeline_start being called at all in v3 of
>>>>>>>>> the patch
>>>>>>>>>
>>>>>>>>>> It's a mess. Looking at some drivers I see that omap3isp calls media_pipeline_stop
>>>>>>>>>> in streamoff (so will have the same problem as you described if VIDIOC_STREAMOFF
>>>>>>>>>> isn't called), exynos4-is does the same, but it also checks the streaming state in
>>>>>>>>>> the release() fop callback, so that would fix this problem. And vimc does this
>>>>>>>>>> in stop_streaming.
>>>>>>>>>>
>>>>>>>>>> I'm in favor of fixing this in vb2, that framework knows exactly when this needs
>>>>>>>>>> to be called.
>>>>>>>>>
>>>>>>>>> Are you suggesting to have vb2 to call media_pipeline_start() or is it
>>>>>>>>> more complex than this ?
>>>>>>>>
>>>>>>>> I think Hans meant adding a .validate() operation to vb2.
>>>>>>>>
>>>>>>>> vb2 is already quite complex, I don't think adding more features is a
>>>>>>>> good idea. I'd rather have vb2 focus on buffer management only
>>>>>>>> (.start_streaming() and .stop_streaming() shouldn't have been in there
>>>>>>>> in my opinion), and handle validation in the .streamon() handler. I'd
>>>>>>>> expect most drivers that deal with media pipelines to do more work in
>>>>>>>> .streamon() anyway.
>>>>>>>
>>>>>>> I disagree with that :-)
>>>>>>>
>>>>>>> It's vb2 that keeps track of the streaming state and when what actions
>>>>>>> need to be taken. Drivers really shouldn't need to care about the ioctls
>>>>>>> themselves, and just implement the relevant vb2 callbacks. Relying on
>>>>>>> drivers to handle any of the streaming ioctls is asking for problems,
>>>>>>> as this shows: most drivers implement this wrong today.
>>>>>>>
>>>>>>> The vb2 framework knows when e.g. the pipeline needs to be started or
>>>>>>> stopped, and can do this at the best time, without drivers needing to
>>>>>>> keep track of when streamon/off/release is called. Keep that logic in
>>>>>>> vb2.
>>>>>>
>>>>>> Pipeline management and buffer management are two different issues.
>>>>>> Don't forget about devices that have multiple video nodes, part of the
>>>>>> same pipeline (possibly a combination of output and capture nodes, or
>>>>>> all of the same type). Forcing drivers to go through vb2 operations to
>>>>>> handle the pipeline will be messy, will result in more bloat in vb2, and
>>>>>> make the result more bug-prone and harder to maintain.
>>>>>>
>>>>>> If pipeline management is too complex, let's simplify it, new helpers
>>>>>> can make sense, but not through vb2.
>>>>>
>>>>> But it is vb2 that knows when streaming starts and stops.
>>>>
>>>> That's right, but pipeline start (which includes validation and resource
>>>> reservation) needs to be performed synchronously with VIDIOC_STREAMON.
>>>> The streaming state managed by vb2 is not relevant,
>>>> media_pipeline_start() must not be delayed the same way
>>>> .start_streaming() is.
>>>
>>> It will be the first thing that vb2_streamon calls. This has nothing to do
>>> with start_streaming: that's called when sufficient number of buffers are
>>> queued up to be able to start the DMA. This proposed prepare_streaming op
>>> will be called when VIDIOC_STREAMON is called.
>>
>> Then it doesn't need vb2's knowledge of the stream state :-)
>>
>>>>> The driver just
>>>>> needs to be informed (e.g. prepare_streaming and unprepare_streaming ops).
>>>>>
>>>>> vb2 deals with buffer management and it keeps track of the streaming state
>>>>> and makes the streaming state transitions. That *is* an integral part of
>>>>> vb2. What is missing at the moment are callbacks done at streamon time and
>>>>> when the streaming stops (streamoff, or close() when is_streaming is true).
>>>>>
>>>>> If you want to implement stream validation in a driver, then there are a
>>>>> lot of things you need to do:
>>>>>
>>>>> - override streamon, make sure you call vb2_queue_is_busy(), validate the
>>>>> pipeline, then call vb2_streamon, if that fails, remember to stop the
>>>>> pipeline.
>>>>>
>>>>> - override streamoff, make sure you call vb2_queue_is_busy(), stop the
>>>>> pipeline and call vb2_streamoff.
>>>>>
>>>>> - in the release() function when the fh is closed, you have to check
>>>>> vb2_is_streaming(), check that you are the owner of the queue, and if true,
>>>>> stop the pipeline.
>>>>
>>>> I'm not opposed to helper functions to implement that, they can bundle
>>>> vb2 calls and pipeline management.
>>>>
>>>>> By moving this to vb2 ops all you need to implement are the prepare and
>>>>> unprepare ops.
>>>>>
>>>>> Esp. the release() implementation is tricky. I'm pretty sure that
>>>>> drivers/media/platform/samsung/exynos4-is/fimc-lite.c is wrong, since it
>>>>> should only call media_pipeline_stop() for the owner of the queue. Instead
>>>>> it calls it for the last user of the queue.
>>>>>
>>>>> I see that fimc_lite_streamoff() is wrong too: you can safely call
>>>>> VIDIOC_STREAMOFF twice: the second streamoff just returns 0 without
>>>>> doing anything. Instead media_pipeline_stop is called without testing if
>>>>> the queue is streaming.
>>>>>
>>>>> And yes, this is in part because V4L2 has quite some history and certainly
>>>>> API choice were made in the past that we wouldn't make today. But vb2
>>>>> shields you from that, and behaves much more like a proper state machine.
>>>>>
>>>>> I know you prefer to give a lot more control to driver developers, but
>>>>> in my experience very few developers can do things like this right. And
>>>>> it is really hard as a reviewer to check if all the corner cases are handled
>>>>> correctly in a driver. If vb2 is used, then I know things are called at the
>>>>> right time, and that makes my life as reviewer so much easier.
>>>>
>>>> It's not just about giving more control to drivers, it's about
>>>> organizing the software layers in a way that keeps them maintainable,
>>>> with layered abstractions and not midlayers.
>>>>
>>>> We are extensively reworking the media pipeline management as part of
>>>> the stream series, and there will be more work on top of that that will
>>>> make even more fundamental changes. I would like to at least postpone
>>>> any work on vb2 until then, to be able to evaluate the impact.
>>>
>>> I'll make an RFC patch for vb2 so you have a better idea of what it does.
>>
>> As long as we don't merge it before I get the chance to send the media
>> pipeline management rework, I'm all for RFCs :-)
>>
>
> To unblock Eugen is it fine if he moves media_pipeline_start() at
> streamon() time for now ? It will require overriding
> v4l2_ioctl.vidioc_streamon which might be a bit of additional work
> (but probably easier to replace once a proper solution lands)
>
> Otherwise, should the series go in as it is now ?

I had a comment about patch 6, so I want a new series anyway. All the
patches with just fixes (everything but 4 and 8) can go in once I have
them.

I think - all things considered - it is currently best to start and stop the
pipeline in start/stop_streaming: that at least avoids all the complications
with streamon/off/close() that is really hard to get right without (IMHO)
my proposed vb2 changes.

With that I will accept the series.

Regards,

Hans

>
> Thanks
> j
>
>>>>> There may still be a few drivers that really need to do this manually, and
>>>>> that's OK, but a driver like the atmel-isc doesn't need that at all.
>>
>> --
>> Regards,
>>
>> Laurent Pinchart

2022-05-02 23:13:57

[permalink] [raw]

Subject: Re: [PATCH v9 08/13] media: atmel: atmel-isc: change format propagation to subdev into only verification

On 29/04/2022 12:07, Laurent Pinchart wrote:
> On Fri, Apr 29, 2022 at 11:58:48AM +0200, Jacopo Mondi wrote:
>> On Fri, Apr 29, 2022 at 10:43:09AM +0200, Hans Verkuil wrote:
>>> On 29/04/2022 10:28, [email protected] wrote:
>>>> On 4/29/22 11:17 AM, Hans Verkuil wrote:
>>>>> On 10/03/2022 10:51, Eugen Hristev wrote:
>>>>>> As a top MC video driver, the atmel-isc should not propagate the format to the
>>>>>> subdevice, it should rather check at start_streaming() time if the subdev is properly
>>>>>> configured with a compatible format.
>>>>>> Removed the whole format finding logic, and reworked the format verification
>>>>>> at start_streaming time, such that the ISC will return an error if the subdevice
>>>>>> is not properly configured. To achieve this, media_pipeline_start
>>>>>> is called and a link_validate callback is created to check the formats.
>>>>>> With this being done, the module parameter 'sensor_preferred' makes no sense
>>>>>> anymore. The ISC should not decide which format the sensor is using. The
>>>>>> ISC should only cope with the situation and inform userspace if the streaming
>>>>>> is possible in the current configuration.
>>>>>> The redesign of the format propagation has also risen the question of the
>>>>>> enumfmt callback. If enumfmt is called with an mbus_code, the enumfmt handler
>>>>>> should only return the formats that are supported for this mbus_code.
>>>>>> Otherwise, the enumfmt will report all the formats that the ISC could output.
>>>>>> With this rework, the dynamic list of user formats is removed. It makes no
>>>>>> more sense to identify at complete time which formats the sensor could emit,
>>>>>> and add those into a separate dynamic list.
>>>>>> The ISC will start with a simple preconfigured default format, and at
>>>>>> link validate time, decide whether it can use the format that is configured
>>>>>> on the sink or not.
>>>>>>
>>>>>> Signed-off-by: Eugen Hristev <[email protected]>
>>>>>> Reviewed-by: Jacopo Mondi <[email protected]>
>>>>>> ---
>>>>>> Changes in v9:
>>>>>> - isc_link_validate now static
>>>>>>
>>>>>> Changes in v7:
>>>>>> - minor typos as suggested by Jacopo
>>>>>> - small changes, reduce some indentation, modified an index, as suggested by
>>>>>> Jacopo
>>>>>>
>>>>>> Changes in v6:
>>>>>> - reworked a bit enum_fmt as suggested by Jacopo
>>>>>>
>>>>>> Changes in v5:
>>>>>> - removed user_formats dynamic list as it is now pointless
>>>>>> - greatly simplified the enum_fmt function
>>>>>> - removed some init code that was useless now
>>>>>>
>>>>>> Changes in v4:
>>>>>> - moved validation code into link_validate and used media_pipeline_start
>>>>>> - merged this patch with the enum_fmt patch which was previously in v3 of
>>>>>> the series
>>>>>>
>>>>>> Changes in v3:
>>>>>> - clamp to maximum resolution once the frame size from the subdev is found
>>>>>> drivers/media/platform/atmel/atmel-isc-base.c | 412 ++++++++----------
>>>>>> .../media/platform/atmel/atmel-isc-scaler.c | 5 +
>>>>>> drivers/media/platform/atmel/atmel-isc.h | 13 +-
>>>>>> .../media/platform/atmel/atmel-sama5d2-isc.c | 20 +
>>>>>> .../media/platform/atmel/atmel-sama7g5-isc.c | 20 +
>>>>>> 5 files changed, 236 insertions(+), 234 deletions(-)
>>>>>>
>>>>>> diff --git a/drivers/media/platform/atmel/atmel-isc-base.c b/drivers/media/platform/atmel/atmel-isc-base.c
>>>>>> index ee1dda6707a0..fe2c0af58060 100644
>>>>>> --- a/drivers/media/platform/atmel/atmel-isc-base.c
>>>>>> +++ b/drivers/media/platform/atmel/atmel-isc-base.c
>>>>>> @@ -36,11 +36,6 @@ static unsigned int debug;
>>>>>> module_param(debug, int, 0644);
>>>>>> MODULE_PARM_DESC(debug, "debug level (0-2)");
>>>>>>
>>>>>> -static unsigned int sensor_preferred = 1;
>>>>>> -module_param(sensor_preferred, uint, 0644);
>>>>>> -MODULE_PARM_DESC(sensor_preferred,
>>>>>> - "Sensor is preferred to output the specified format (1-on 0-off), default 1");
>>>>>> -
>>>>>> #define ISC_IS_FORMAT_RAW(mbus_code) \
>>>>>> (((mbus_code) & 0xf000) == 0x3000)
>>>>>>
>>>>>> @@ -337,6 +332,10 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)
>>>>>> unsigned long flags;
>>>>>> int ret;
>>>>>>
>>>>>> + ret = media_pipeline_start(&isc->video_dev.entity, &isc->mpipe);
>>>>>
>>>>> The pipeline validation is done in start_streaming, but I don't think that
>>>>> is the best place: if STREAMON is called before buffers are queued, then
>>>>> an invalid pipeline isn't discovered until enough buffers are queued to
>>>>> kick off start_streaming.
>>>>>
>>>>> Drivers like vsp1, omap3isp and the samsung drivers all do this in streamon().
>>>>>
>>>>> I think that is the correct time to do this.
>>>>
>>>> Hello Hans,
>>>>
>>>> Initially (v2, v3) I had this in streamon(). The problem that I faced at
>>>> that time was that streamoff was never called, so I could not call
>>>> media_pipeline_stop(). Then Jacopo told me to move it to start_streaming
>>>> (see change log for v4) , and I did not face any more problems.
>>
>> Yes indeed, seems I suggested to use media_pipeline_handler in a
>> comment on your v3
>>
>> "at s_stream time your top driver calls media_pipeline_start()"
>>
>> sorry about that, I should have looked around a bit more carefully and
>> notice most drivers do so at vb2 streamon
>>
>> However I don't see media_pipeline_start being called at all in v3 of
>> the patch
>>
>>> It's a mess. Looking at some drivers I see that omap3isp calls media_pipeline_stop
>>> in streamoff (so will have the same problem as you described if VIDIOC_STREAMOFF
>>> isn't called), exynos4-is does the same, but it also checks the streaming state in
>>> the release() fop callback, so that would fix this problem. And vimc does this
>>> in stop_streaming.
>>>
>>> I'm in favor of fixing this in vb2, that framework knows exactly when this needs
>>> to be called.
>>
>> Are you suggesting to have vb2 to call media_pipeline_start() or is it
>> more complex than this ?
>
> I think Hans meant adding a .validate() operation to vb2.
>
> vb2 is already quite complex, I don't think adding more features is a
> good idea. I'd rather have vb2 focus on buffer management only
> (.start_streaming() and .stop_streaming() shouldn't have been in there
> in my opinion), and handle validation in the .streamon() handler. I'd
> expect most drivers that deal with media pipelines to do more work in
> .streamon() anyway.

I disagree with that :-)

It's vb2 that keeps track of the streaming state and when what actions
need to be taken. Drivers really shouldn't need to care about the ioctls
themselves, and just implement the relevant vb2 callbacks. Relying on
drivers to handle any of the streaming ioctls is asking for problems,
as this shows: most drivers implement this wrong today.

The vb2 framework knows when e.g. the pipeline needs to be started or
stopped, and can do this at the best time, without drivers needing to
keep track of when streamon/off/release is called. Keep that logic in
vb2.

Regards,

Hans

2022-05-02 23:24:12

[permalink] [raw]

Subject: Re: [PATCH v9 08/13] media: atmel: atmel-isc: change format propagation to subdev into only verification

On Fri, Apr 29, 2022 at 02:38:59PM +0200, Jacopo Mondi wrote:
> On Fri, Apr 29, 2022 at 03:28:11PM +0300, Laurent Pinchart wrote:
> > On Fri, Apr 29, 2022 at 02:23:45PM +0200, Hans Verkuil wrote:
> > > On 29/04/2022 14:02, Laurent Pinchart wrote:
> > > > On Fri, Apr 29, 2022 at 01:17:27PM +0200, Hans Verkuil wrote:
> > > >> On 29/04/2022 12:20, Laurent Pinchart wrote:
> > > >>> On Fri, Apr 29, 2022 at 12:13:46PM +0200, Hans Verkuil wrote:
> > > >>>> On 29/04/2022 12:07, Laurent Pinchart wrote:
> > > >>>>> On Fri, Apr 29, 2022 at 11:58:48AM +0200, Jacopo Mondi wrote:
> > > >>>>>> On Fri, Apr 29, 2022 at 10:43:09AM +0200, Hans Verkuil wrote:
> > > >>>>>>> On 29/04/2022 10:28, [email protected] wrote:
> > > >>>>>>>> On 4/29/22 11:17 AM, Hans Verkuil wrote:
> > > >>>>>>>>> On 10/03/2022 10:51, Eugen Hristev wrote:
> > > >>>>>>>>>> As a top MC video driver, the atmel-isc should not propagate the format to the
> > > >>>>>>>>>> subdevice, it should rather check at start_streaming() time if the subdev is properly
> > > >>>>>>>>>> configured with a compatible format.
> > > >>>>>>>>>> Removed the whole format finding logic, and reworked the format verification
> > > >>>>>>>>>> at start_streaming time, such that the ISC will return an error if the subdevice
> > > >>>>>>>>>> is not properly configured. To achieve this, media_pipeline_start
> > > >>>>>>>>>> is called and a link_validate callback is created to check the formats.
> > > >>>>>>>>>> With this being done, the module parameter 'sensor_preferred' makes no sense
> > > >>>>>>>>>> anymore. The ISC should not decide which format the sensor is using. The
> > > >>>>>>>>>> ISC should only cope with the situation and inform userspace if the streaming
> > > >>>>>>>>>> is possible in the current configuration.
> > > >>>>>>>>>> The redesign of the format propagation has also risen the question of the
> > > >>>>>>>>>> enumfmt callback. If enumfmt is called with an mbus_code, the enumfmt handler
> > > >>>>>>>>>> should only return the formats that are supported for this mbus_code.
> > > >>>>>>>>>> Otherwise, the enumfmt will report all the formats that the ISC could output.
> > > >>>>>>>>>> With this rework, the dynamic list of user formats is removed. It makes no
> > > >>>>>>>>>> more sense to identify at complete time which formats the sensor could emit,
> > > >>>>>>>>>> and add those into a separate dynamic list.
> > > >>>>>>>>>> The ISC will start with a simple preconfigured default format, and at
> > > >>>>>>>>>> link validate time, decide whether it can use the format that is configured
> > > >>>>>>>>>> on the sink or not.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Signed-off-by: Eugen Hristev <[email protected]>
> > > >>>>>>>>>> Reviewed-by: Jacopo Mondi <[email protected]>
> > > >>>>>>>>>> ---
> > > >>>>>>>>>> Changes in v9:
> > > >>>>>>>>>> - isc_link_validate now static
> > > >>>>>>>>>>
> > > >>>>>>>>>> Changes in v7:
> > > >>>>>>>>>> - minor typos as suggested by Jacopo
> > > >>>>>>>>>> - small changes, reduce some indentation, modified an index, as suggested by
> > > >>>>>>>>>> Jacopo
> > > >>>>>>>>>>
> > > >>>>>>>>>> Changes in v6:
> > > >>>>>>>>>> - reworked a bit enum_fmt as suggested by Jacopo
> > > >>>>>>>>>>
> > > >>>>>>>>>> Changes in v5:
> > > >>>>>>>>>> - removed user_formats dynamic list as it is now pointless
> > > >>>>>>>>>> - greatly simplified the enum_fmt function
> > > >>>>>>>>>> - removed some init code that was useless now
> > > >>>>>>>>>>
> > > >>>>>>>>>> Changes in v4:
> > > >>>>>>>>>> - moved validation code into link_validate and used media_pipeline_start
> > > >>>>>>>>>> - merged this patch with the enum_fmt patch which was previously in v3 of
> > > >>>>>>>>>> the series
> > > >>>>>>>>>>
> > > >>>>>>>>>> Changes in v3:
> > > >>>>>>>>>> - clamp to maximum resolution once the frame size from the subdev is found
> > > >>>>>>>>>> drivers/media/platform/atmel/atmel-isc-base.c | 412 ++++++++----------
> > > >>>>>>>>>> .../media/platform/atmel/atmel-isc-scaler.c | 5 +
> > > >>>>>>>>>> drivers/media/platform/atmel/atmel-isc.h | 13 +-
> > > >>>>>>>>>> .../media/platform/atmel/atmel-sama5d2-isc.c | 20 +
> > > >>>>>>>>>> .../media/platform/atmel/atmel-sama7g5-isc.c | 20 +
> > > >>>>>>>>>> 5 files changed, 236 insertions(+), 234 deletions(-)
> > > >>>>>>>>>>
> > > >>>>>>>>>> diff --git a/drivers/media/platform/atmel/atmel-isc-base.c b/drivers/media/platform/atmel/atmel-isc-base.c
> > > >>>>>>>>>> index ee1dda6707a0..fe2c0af58060 100644
> > > >>>>>>>>>> --- a/drivers/media/platform/atmel/atmel-isc-base.c
> > > >>>>>>>>>> +++ b/drivers/media/platform/atmel/atmel-isc-base.c
> > > >>>>>>>>>> @@ -36,11 +36,6 @@ static unsigned int debug;
> > > >>>>>>>>>> module_param(debug, int, 0644);
> > > >>>>>>>>>> MODULE_PARM_DESC(debug, "debug level (0-2)");
> > > >>>>>>>>>>
> > > >>>>>>>>>> -static unsigned int sensor_preferred = 1;
> > > >>>>>>>>>> -module_param(sensor_preferred, uint, 0644);
> > > >>>>>>>>>> -MODULE_PARM_DESC(sensor_preferred,
> > > >>>>>>>>>> - "Sensor is preferred to output the specified format (1-on 0-off), default 1");
> > > >>>>>>>>>> -
> > > >>>>>>>>>> #define ISC_IS_FORMAT_RAW(mbus_code) \
> > > >>>>>>>>>> (((mbus_code) & 0xf000) == 0x3000)
> > > >>>>>>>>>>
> > > >>>>>>>>>> @@ -337,6 +332,10 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)
> > > >>>>>>>>>> unsigned long flags;
> > > >>>>>>>>>> int ret;
> > > >>>>>>>>>>
> > > >>>>>>>>>> + ret = media_pipeline_start(&isc->video_dev.entity, &isc->mpipe);
> > > >>>>>>>>>
> > > >>>>>>>>> The pipeline validation is done in start_streaming, but I don't think that
> > > >>>>>>>>> is the best place: if STREAMON is called before buffers are queued, then
> > > >>>>>>>>> an invalid pipeline isn't discovered until enough buffers are queued to
> > > >>>>>>>>> kick off start_streaming.
> > > >>>>>>>>>
> > > >>>>>>>>> Drivers like vsp1, omap3isp and the samsung drivers all do this in streamon().
> > > >>>>>>>>>
> > > >>>>>>>>> I think that is the correct time to do this.
> > > >>>>>>>>
> > > >>>>>>>> Hello Hans,
> > > >>>>>>>>
> > > >>>>>>>> Initially (v2, v3) I had this in streamon(). The problem that I faced at
> > > >>>>>>>> that time was that streamoff was never called, so I could not call
> > > >>>>>>>> media_pipeline_stop(). Then Jacopo told me to move it to start_streaming
> > > >>>>>>>> (see change log for v4) , and I did not face any more problems.
> > > >>>>>>
> > > >>>>>> Yes indeed, seems I suggested to use media_pipeline_handler in a
> > > >>>>>> comment on your v3
> > > >>>>>>
> > > >>>>>> "at s_stream time your top driver calls media_pipeline_start()"
> > > >>>>>>
> > > >>>>>> sorry about that, I should have looked around a bit more carefully and
> > > >>>>>> notice most drivers do so at vb2 streamon
> > > >>>>>>
> > > >>>>>> However I don't see media_pipeline_start being called at all in v3 of
> > > >>>>>> the patch
> > > >>>>>>
> > > >>>>>>> It's a mess. Looking at some drivers I see that omap3isp calls media_pipeline_stop
> > > >>>>>>> in streamoff (so will have the same problem as you described if VIDIOC_STREAMOFF
> > > >>>>>>> isn't called), exynos4-is does the same, but it also checks the streaming state in
> > > >>>>>>> the release() fop callback, so that would fix this problem. And vimc does this
> > > >>>>>>> in stop_streaming.
> > > >>>>>>>
> > > >>>>>>> I'm in favor of fixing this in vb2, that framework knows exactly when this needs
> > > >>>>>>> to be called.
> > > >>>>>>
> > > >>>>>> Are you suggesting to have vb2 to call media_pipeline_start() or is it
> > > >>>>>> more complex than this ?
> > > >>>>>
> > > >>>>> I think Hans meant adding a .validate() operation to vb2.
> > > >>>>>
> > > >>>>> vb2 is already quite complex, I don't think adding more features is a
> > > >>>>> good idea. I'd rather have vb2 focus on buffer management only
> > > >>>>> (.start_streaming() and .stop_streaming() shouldn't have been in there
> > > >>>>> in my opinion), and handle validation in the .streamon() handler. I'd
> > > >>>>> expect most drivers that deal with media pipelines to do more work in
> > > >>>>> .streamon() anyway.
> > > >>>>
> > > >>>> I disagree with that :-)
> > > >>>>
> > > >>>> It's vb2 that keeps track of the streaming state and when what actions
> > > >>>> need to be taken. Drivers really shouldn't need to care about the ioctls
> > > >>>> themselves, and just implement the relevant vb2 callbacks. Relying on
> > > >>>> drivers to handle any of the streaming ioctls is asking for problems,
> > > >>>> as this shows: most drivers implement this wrong today.
> > > >>>>
> > > >>>> The vb2 framework knows when e.g. the pipeline needs to be started or
> > > >>>> stopped, and can do this at the best time, without drivers needing to
> > > >>>> keep track of when streamon/off/release is called. Keep that logic in
> > > >>>> vb2.
> > > >>>
> > > >>> Pipeline management and buffer management are two different issues.
> > > >>> Don't forget about devices that have multiple video nodes, part of the
> > > >>> same pipeline (possibly a combination of output and capture nodes, or
> > > >>> all of the same type). Forcing drivers to go through vb2 operations to
> > > >>> handle the pipeline will be messy, will result in more bloat in vb2, and
> > > >>> make the result more bug-prone and harder to maintain.
> > > >>>
> > > >>> If pipeline management is too complex, let's simplify it, new helpers
> > > >>> can make sense, but not through vb2.
> > > >>
> > > >> But it is vb2 that knows when streaming starts and stops.
> > > >
> > > > That's right, but pipeline start (which includes validation and resource
> > > > reservation) needs to be performed synchronously with VIDIOC_STREAMON.
> > > > The streaming state managed by vb2 is not relevant,
> > > > media_pipeline_start() must not be delayed the same way
> > > > .start_streaming() is.
> > >
> > > It will be the first thing that vb2_streamon calls. This has nothing to do
> > > with start_streaming: that's called when sufficient number of buffers are
> > > queued up to be able to start the DMA. This proposed prepare_streaming op
> > > will be called when VIDIOC_STREAMON is called.
> >
> > Then it doesn't need vb2's knowledge of the stream state :-)
> >
> > > >> The driver just
> > > >> needs to be informed (e.g. prepare_streaming and unprepare_streaming ops).
> > > >>
> > > >> vb2 deals with buffer management and it keeps track of the streaming state
> > > >> and makes the streaming state transitions. That *is* an integral part of
> > > >> vb2. What is missing at the moment are callbacks done at streamon time and
> > > >> when the streaming stops (streamoff, or close() when is_streaming is true).
> > > >>
> > > >> If you want to implement stream validation in a driver, then there are a
> > > >> lot of things you need to do:
> > > >>
> > > >> - override streamon, make sure you call vb2_queue_is_busy(), validate the
> > > >> pipeline, then call vb2_streamon, if that fails, remember to stop the
> > > >> pipeline.
> > > >>
> > > >> - override streamoff, make sure you call vb2_queue_is_busy(), stop the
> > > >> pipeline and call vb2_streamoff.
> > > >>
> > > >> - in the release() function when the fh is closed, you have to check
> > > >> vb2_is_streaming(), check that you are the owner of the queue, and if true,
> > > >> stop the pipeline.
> > > >
> > > > I'm not opposed to helper functions to implement that, they can bundle
> > > > vb2 calls and pipeline management.
> > > >
> > > >> By moving this to vb2 ops all you need to implement are the prepare and
> > > >> unprepare ops.
> > > >>
> > > >> Esp. the release() implementation is tricky. I'm pretty sure that
> > > >> drivers/media/platform/samsung/exynos4-is/fimc-lite.c is wrong, since it
> > > >> should only call media_pipeline_stop() for the owner of the queue. Instead
> > > >> it calls it for the last user of the queue.
> > > >>
> > > >> I see that fimc_lite_streamoff() is wrong too: you can safely call
> > > >> VIDIOC_STREAMOFF twice: the second streamoff just returns 0 without
> > > >> doing anything. Instead media_pipeline_stop is called without testing if
> > > >> the queue is streaming.
> > > >>
> > > >> And yes, this is in part because V4L2 has quite some history and certainly
> > > >> API choice were made in the past that we wouldn't make today. But vb2
> > > >> shields you from that, and behaves much more like a proper state machine.
> > > >>
> > > >> I know you prefer to give a lot more control to driver developers, but
> > > >> in my experience very few developers can do things like this right. And
> > > >> it is really hard as a reviewer to check if all the corner cases are handled
> > > >> correctly in a driver. If vb2 is used, then I know things are called at the
> > > >> right time, and that makes my life as reviewer so much easier.
> > > >
> > > > It's not just about giving more control to drivers, it's about
> > > > organizing the software layers in a way that keeps them maintainable,
> > > > with layered abstractions and not midlayers.
> > > >
> > > > We are extensively reworking the media pipeline management as part of
> > > > the stream series, and there will be more work on top of that that will
> > > > make even more fundamental changes. I would like to at least postpone
> > > > any work on vb2 until then, to be able to evaluate the impact.
> > >
> > > I'll make an RFC patch for vb2 so you have a better idea of what it does.
> >
> > As long as we don't merge it before I get the chance to send the media
> > pipeline management rework, I'm all for RFCs :-)
> >
>
> To unblock Eugen is it fine if he moves media_pipeline_start() at
> streamon() time for now ? It will require overriding
> v4l2_ioctl.vidioc_streamon which might be a bit of additional work
> (but probably easier to replace once a proper solution lands)

Fine with me.

> Otherwise, should the series go in as it is now ?
>
> > > >> There may still be a few drivers that really need to do this manually, and
> > > >> that's OK, but a driver like the atmel-isc doesn't need that at all.

--
Regards,

Laurent Pinchart

2022-05-03 00:29:49

[permalink] [raw]

Subject: Re: [PATCH v9 00/13] media: atmel: atmel-isc: implement media controller

On 29/04/2022 10:20, [email protected] wrote:
> On 4/29/22 11:05 AM, Hans Verkuil wrote:
>> Hi Eugen,
>>
>> On 10/03/2022 10:51, Eugen Hristev wrote:
>>> This series is the v9 series that attempts to support media controller in the
>>> atmel ISC and XISC drivers.
>>> The CSI2DC driver was accepted thus removed from the patch series, together with
>>> other patches.
>>>
>>> Important note: this series applies on top of current media_staging tree, as it
>>> relies on previous patches in the series which were accepted.
>>>
>>> Thanks to everyone who reviewed my work !
>>>
>>> Eugen
>>>
>>> Changes in v9:
>>> -> kernel robot reported isc_link_validate is not static, changed to static.
>>>
>>> Changes in v8:
>>> -> scaler: modified crop bounds to have the exact source size
>>>
>>> Changes in v7:
>>> -> scaler: modified crop bounds to have maximum isc size
>>> -> format propagation: did small changes as per Jacopo review
>>>
>>>
>>> Changes in v6:
>>> -> worked a bit on scaler, added try crop and other changes as per Jacopo review
>>> -> worked on isc-base enum_fmt , reworked as per Jacopo review
>>>
>>> Changes in v5:
>>> -> removed patch that removed the 'stop' variable as it was still required
>>> -> added two new trivial patches
>>> -> reworked some parts of the scaler and format propagation after discussions with Jacopo
>>>
>>>
>>> Changes in v4:
>>> -> as reviewed by Hans, added new patch to remove the 'stop' variable and reworked
>>> one patch that was using it
>>> -> as reviewed by Jacopo, reworked some parts of the media controller implementation
>>>
>>>
>>> Changes in v3:
>>> - change in bindings, small fixes in csi2dc driver and conversion to mc
>>> for the isc-base.
>>> - removed some MAINTAINERS patches and used patterns in MAINTAINERS
>>>
>>> Changes in v2:
>>> - integrated many changes suggested by Jacopo in the review of the v1 series.
>>> - add a few new patches
>>>
>>>
>>>
>>> Eugen Hristev (13):
>>> media: atmel: atmel-isc-base: use streaming status when queueing
>>> buffers
>>> media: atmel: atmel-isc-base: replace is_streaming call in
>>> s_fmt_vid_cap
>>> media: atmel: atmel-isc: remove redundant comments
>>> media: atmel: atmel-isc: implement media controller
>>> media: atmel: atmel-sama5d2-isc: fix wrong mask in YUYV format check
>>> media: atmel: atmel-isc-base: use mutex to lock awb workqueue from
>>> streaming
>>> media: atmel: atmel-isc: compact the controller formats list
>>> media: atmel: atmel-isc: change format propagation to subdev into only
>>> verification
>>> media: atmel: atmel-sama7g5-isc: remove stray line
>>> dt-bindings: media: microchip,xisc: add bus-width of 14
>>
>> I'm a bit unhappy with the order of these patches. Mostly these are fixes,
>> except for patches 4 and 8, which are the meat of this series and actually
>> switching on the MC support.
>>
>> Can those be moved to the end? That also makes it easier to merge the earlier
>> patches if some more work is needed for the MC part.
>>
>> I'm also not sure whether patches 4 and 8 shouldn't be a single patch,
>> since patch 4 leaves the driver in an inconsistent state since it is
>> missing the link validation code that patch 8 adds. Unless I missed
>> something?
>
> Hello Hans,
>
> The difference that patch 8 is making is that the 'old way' of
> configuring the ISC is no longer possible.
>
> Patch 4 makes the ISC 'mc-ready' with all entities, links, but the old
> way still works (meaning that the top driver will call s_fmt down to the
> subdev ).
> After patch 8, the driver no longer has this support at all, and
> validates links at start_streaming, and no longer sets anything to the
> subdev, just validates the config that the subdev already has.
> So one reason that I had things in two patches was that patch 8 makes
> this big change that also makes userspace behave differently and has to
> configure all the subdevs and media pipeline.
>
> If you feel patch 4 and patch 8 should be squashed, I can do it, definitely.
>
> Let me know how to proceed ?

Ah, that was your intention. The problem in patch 4 is that you set
V4L2_CAP_IO_MC, which indicates to applications that they have to use
the MC and configure the pipeline correctly. Moving that to patch 8
should resolve that, I think.

Regards,

Hans

>
> Thanks,
> Eugen
>
>>
>> Regards,
>>
>> Hans
>>
>>> ARM: dts: at91: sama7g5: add nodes for video capture
>>> ARM: configs: at91: sama7: add xisc and csi2dc
>>> ARM: multi_v7_defconfig: add atmel video pipeline modules
>>>
>>> .../bindings/media/microchip,xisc.yaml | 2 +-
>>> arch/arm/boot/dts/sama7g5.dtsi | 49 ++
>>> arch/arm/configs/multi_v7_defconfig | 3 +
>>> arch/arm/configs/sama7_defconfig | 2 +
>>> drivers/media/platform/atmel/Makefile | 2 +-
>>> drivers/media/platform/atmel/atmel-isc-base.c | 518 ++++++++++--------
>>> .../media/platform/atmel/atmel-isc-scaler.c | 267 +++++++++
>>> drivers/media/platform/atmel/atmel-isc.h | 58 +-
>>> .../media/platform/atmel/atmel-sama5d2-isc.c | 87 +--
>>> .../media/platform/atmel/atmel-sama7g5-isc.c | 93 ++--
>>> 10 files changed, 754 insertions(+), 327 deletions(-)
>>> create mode 100644 drivers/media/platform/atmel/atmel-isc-scaler.c
>>>
>>
>

2022-05-03 01:10:28