Audio signal processing also has the requirement for memory to
memory similar as Video.
This asrc memory to memory (memory ->asrc->memory) case is a non
real time use case.
User fills the input buffer to the asrc module, after conversion, then asrc
sends back the output buffer to user. So it is not a traditional ALSA playback
and capture case.
It is a specific use case, there is no reference in current kernel.
v4l2 memory to memory is the closed implementation, v4l2 current
support video, image, radio, tuner, touch devices, so it is not
complicated to add support for this specific audio case.
Because we had implemented the "memory -> asrc ->i2s device-> codec"
use case in ALSA. Now the "memory->asrc->memory" needs
to reuse the code in asrc driver, so the first 3 patches is for refining
the code to make it can be shared by the "memory->asrc->memory"
driver.
The main change is in the v4l2 side, A /dev/vl4-audioX will be created,
user applications only use the ioctl of v4l2 framework.
Other change is to add memory to memory support for two kinds of i.MX ASRC
module.
changes in v14:
- document the reservation of 'AUXX' fourcc format.
- add v4l2_audfmt_to_fourcc() definition.
changes in v13
- change 'pixelformat' to 'audioformat' in dev-audio-mem2mem.rst
- add more description for clock drift in ext-ctrls-audio-m2m.rst
- Add "media: v4l2-ctrls: add support for fraction_bits" from Hans
to avoid build issue for kernel test robot
changes in v12
- minor changes according to comments
- drop min_buffers_needed = 1 and V4L2_CTRL_FLAG_UPDATE flag
- drop bus_info
changes in v11
- add add-fixed-point-test-controls in vivid.
- add v4l2_ctrl_fp_compose() helper function for min and max
changes in v10
- remove FIXED_POINT type
- change code base on media: v4l2-ctrls: add support for fraction_bits
- fix issue reported by kernel test robot
- remove module_alias
changes in v9:
- add MEDIA_ENT_F_PROC_AUDIO_RESAMPLER.
- add MEDIA_INTF_T_V4L_AUDIO
- add media controller support
- refine the vim2m-audio to support 8k<->16k conversion.
changes in v8:
- refine V4L2_CAP_AUDIO_M2M to be 0x00000008
- update doc for FIXED_POINT
- address comments for imx-asrc
changes in v7:
- add acked-by from Mark
- separate commit for fixed point, m2m audio class, audio rate controls
- use INTEGER_MENU for rate, FIXED_POINT for rate offset
- remove used fmts
- address other comments for Hans
changes in v6:
- use m2m_prepare/m2m_unprepare/m2m_start/m2m_stop to replace
m2m_start_part_one/m2m_stop_part_one, m2m_start_part_two/m2m_stop_part_two.
- change V4L2_CTRL_TYPE_ASRC_RATE to V4L2_CTRL_TYPE_FIXED_POINT
- fix warning by kernel test rebot
- remove some unused format V4L2_AUDIO_FMT_XX
- Get SNDRV_PCM_FORMAT from V4L2_AUDIO_FMT in driver.
- rename audm2m to viaudm2m.
changes in v5:
- remove V4L2_AUDIO_FMT_LPCM
- define audio pixel format like V4L2_AUDIO_FMT_S8...
- remove rate and format in struct v4l2_audio_format.
- Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE controls
- updata document accordingly.
changes in v4:
- update document style
- separate V4L2_AUDIO_FMT_LPCM and V4L2_CAP_AUDIO_M2M in separate commit
changes in v3:
- Modify documents for adding audio m2m support
- Add audio virtual m2m driver
- Defined V4L2_AUDIO_FMT_LPCM format type for audio.
- Defined V4L2_CAP_AUDIO_M2M capability type for audio m2m case.
- with modification in v4l-utils, pass v4l2-compliance test.
changes in v2:
- decouple the implementation in v4l2 and ALSA
- implement the memory to memory driver as a platfrom driver
and move it to driver/media
- move fsl_asrc_common.h to include/sound folder
Hans Verkuil (1):
media: v4l2-ctrls: add support for fraction_bits
Shengjiu Wang (15):
ASoC: fsl_asrc: define functions for memory to memory usage
ASoC: fsl_easrc: define functions for memory to memory usage
ASoC: fsl_asrc: move fsl_asrc_common.h to include/sound
ASoC: fsl_asrc: register m2m platform device
ASoC: fsl_easrc: register m2m platform device
media: uapi: Add V4L2_CAP_AUDIO_M2M capability flag
media: v4l2: Add audio capture and output support
media: uapi: Define audio sample format fourcc type
media: uapi: Add V4L2_CTRL_CLASS_M2M_AUDIO
media: uapi: Add audio rate controls support
media: uapi: Declare interface types for Audio
media: uapi: Add an entity type for audio resampler
media: vivid: add fixed point test controls
media: imx-asrc: Add memory to memory driver
media: vim2m-audio: add virtual driver for audio memory to memory
.../media/mediactl/media-types.rst | 11 +
.../userspace-api/media/v4l/buffer.rst | 6 +
.../userspace-api/media/v4l/common.rst | 1 +
.../media/v4l/dev-audio-mem2mem.rst | 71 +
.../userspace-api/media/v4l/devices.rst | 1 +
.../media/v4l/ext-ctrls-audio-m2m.rst | 59 +
.../userspace-api/media/v4l/pixfmt-audio.rst | 100 ++
.../userspace-api/media/v4l/pixfmt.rst | 1 +
.../media/v4l/vidioc-enum-fmt.rst | 2 +
.../media/v4l/vidioc-g-ext-ctrls.rst | 4 +
.../userspace-api/media/v4l/vidioc-g-fmt.rst | 4 +
.../media/v4l/vidioc-querycap.rst | 3 +
.../media/v4l/vidioc-queryctrl.rst | 11 +-
.../media/videodev2.h.rst.exceptions | 3 +
.../media/common/videobuf2/videobuf2-v4l2.c | 4 +
drivers/media/platform/nxp/Kconfig | 13 +
drivers/media/platform/nxp/Makefile | 1 +
drivers/media/platform/nxp/imx-asrc.c | 1256 +++++++++++++++++
drivers/media/test-drivers/Kconfig | 10 +
drivers/media/test-drivers/Makefile | 1 +
drivers/media/test-drivers/vim2m-audio.c | 793 +++++++++++
drivers/media/test-drivers/vivid/vivid-core.h | 2 +
.../media/test-drivers/vivid/vivid-ctrls.c | 26 +
drivers/media/v4l2-core/v4l2-compat-ioctl32.c | 9 +
drivers/media/v4l2-core/v4l2-ctrls-api.c | 1 +
drivers/media/v4l2-core/v4l2-ctrls-core.c | 93 +-
drivers/media/v4l2-core/v4l2-ctrls-defs.c | 10 +
drivers/media/v4l2-core/v4l2-dev.c | 21 +
drivers/media/v4l2-core/v4l2-ioctl.c | 66 +
drivers/media/v4l2-core/v4l2-mem2mem.c | 13 +-
include/media/v4l2-ctrls.h | 13 +-
include/media/v4l2-dev.h | 2 +
include/media/v4l2-ioctl.h | 34 +
.../fsl => include/sound}/fsl_asrc_common.h | 60 +
include/uapi/linux/media.h | 2 +
include/uapi/linux/v4l2-controls.h | 9 +
include/uapi/linux/videodev2.h | 50 +-
sound/soc/fsl/fsl_asrc.c | 144 ++
sound/soc/fsl/fsl_asrc.h | 4 +-
sound/soc/fsl/fsl_asrc_dma.c | 2 +-
sound/soc/fsl/fsl_easrc.c | 233 +++
sound/soc/fsl/fsl_easrc.h | 6 +-
42 files changed, 3128 insertions(+), 27 deletions(-)
create mode 100644 Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst
create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-audio-m2m.rst
create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-audio.rst
create mode 100644 drivers/media/platform/nxp/imx-asrc.c
create mode 100644 drivers/media/test-drivers/vim2m-audio.c
rename {sound/soc/fsl => include/sound}/fsl_asrc_common.h (60%)
--
2.34.1
From: Hans Verkuil <[email protected]>
This adds support for the fraction_bits field, used with integer controls.
This allows fixed point formats to be described.
The fraction_bits field is only exposed through VIDIOC_QUERY_EXT_CTRL.
For a given signed two's complement Qf fixed point value 'f' equals
fraction_bits.
Signed-off-by: Hans Verkuil <[email protected]>
---
.../media/v4l/vidioc-queryctrl.rst | 11 ++-
drivers/media/v4l2-core/v4l2-ctrls-api.c | 1 +
drivers/media/v4l2-core/v4l2-ctrls-core.c | 93 +++++++++++++++----
include/media/v4l2-ctrls.h | 7 +-
include/uapi/linux/videodev2.h | 3 +-
5 files changed, 95 insertions(+), 20 deletions(-)
diff --git a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst
index 4d38acafe8e1..e65c7e5d78ec 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-queryctrl.rst
@@ -267,7 +267,16 @@ See also the examples in :ref:`control`.
- The size of each dimension. The first ``nr_of_dims`` elements of
this array must be non-zero, all remaining elements must be zero.
* - __u32
- - ``reserved``\ [32]
+ - ``fraction_bits``
+ - The number of least significant bits of the control value that
+ form the fraction of the fixed point value. This is 0 if the value
+ is a regular integer. This can be used with all integer control types
+ (``INTEGER``, ``INTEGER64``, ``U8``, ``U16`` and ``U32``).
+ For the signed types the signed two's complement representation is used.
+ This field applies to the control value as well as the ``minimum``,
+ ``maximum``, ``step`` and ``default_value`` fields.
+ * - __u32
+ - ``reserved``\ [31]
- Reserved for future extensions. Applications and drivers must set
the array to zero.
diff --git a/drivers/media/v4l2-core/v4l2-ctrls-api.c b/drivers/media/v4l2-core/v4l2-ctrls-api.c
index d9a422017bd9..ef16b00421ec 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls-api.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls-api.c
@@ -1101,6 +1101,7 @@ int v4l2_query_ext_ctrl(struct v4l2_ctrl_handler *hdl, struct v4l2_query_ext_ctr
qc->elems = ctrl->elems;
qc->nr_of_dims = ctrl->nr_of_dims;
memcpy(qc->dims, ctrl->dims, qc->nr_of_dims * sizeof(qc->dims[0]));
+ qc->fraction_bits = ctrl->fraction_bits;
qc->minimum = ctrl->minimum;
qc->maximum = ctrl->maximum;
qc->default_value = ctrl->default_value;
diff --git a/drivers/media/v4l2-core/v4l2-ctrls-core.c b/drivers/media/v4l2-core/v4l2-ctrls-core.c
index c4d995f32191..d83a37198bb5 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls-core.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls-core.c
@@ -252,12 +252,61 @@ void v4l2_ctrl_type_op_init(const struct v4l2_ctrl *ctrl, u32 from_idx,
}
EXPORT_SYMBOL(v4l2_ctrl_type_op_init);
+static void v4l2_ctrl_log_fp(s64 v, unsigned int fraction_bits)
+{
+ s64 i, f, mask;
+
+ if (!fraction_bits) {
+ pr_cont("%lld", v);
+ return;
+ }
+
+ mask = (1ULL << fraction_bits) - 1;
+
+ /*
+ * Note: this function does not support fixed point u64 with
+ * fraction_bits set to 64. At the moment there is no U64
+ * control type, but if that is added, then this code will have
+ * to add support for it.
+ */
+ if (fraction_bits >= 63)
+ i = v < 0 ? -1 : 0;
+ else
+ i = div64_s64(v, 1LL << fraction_bits);
+
+ f = v < 0 ? -((-v) & mask) : (v & mask);
+
+ if (!f) {
+ pr_cont("%lld", i);
+ } else if (fraction_bits < 20) {
+ u64 div = 1ULL << fraction_bits;
+
+ if (!i && f < 0)
+ pr_cont("-%lld/%llu", -f, div);
+ else if (!i)
+ pr_cont("%lld/%llu", f, div);
+ else if (i < 0 || f < 0)
+ pr_cont("-%lld-%llu/%llu", -i, -f, div);
+ else
+ pr_cont("%lld+%llu/%llu", i, f, div);
+ } else {
+ if (!i && f < 0)
+ pr_cont("-%lld/(2^%u)", -f, fraction_bits);
+ else if (!i)
+ pr_cont("%lld/(2^%u)", f, fraction_bits);
+ else if (i < 0 || f < 0)
+ pr_cont("-%lld-%llu/(2^%u)", -i, -f, fraction_bits);
+ else
+ pr_cont("%lld+%llu/(2^%u)", i, f, fraction_bits);
+ }
+}
+
void v4l2_ctrl_type_op_log(const struct v4l2_ctrl *ctrl)
{
union v4l2_ctrl_ptr ptr = ctrl->p_cur;
if (ctrl->is_array) {
- unsigned i;
+ unsigned int i;
for (i = 0; i < ctrl->nr_of_dims; i++)
pr_cont("[%u]", ctrl->dims[i]);
@@ -266,7 +315,7 @@ void v4l2_ctrl_type_op_log(const struct v4l2_ctrl *ctrl)
switch (ctrl->type) {
case V4L2_CTRL_TYPE_INTEGER:
- pr_cont("%d", *ptr.p_s32);
+ v4l2_ctrl_log_fp(*ptr.p_s32, ctrl->fraction_bits);
break;
case V4L2_CTRL_TYPE_BOOLEAN:
pr_cont("%s", *ptr.p_s32 ? "true" : "false");
@@ -281,19 +330,19 @@ void v4l2_ctrl_type_op_log(const struct v4l2_ctrl *ctrl)
pr_cont("0x%08x", *ptr.p_s32);
break;
case V4L2_CTRL_TYPE_INTEGER64:
- pr_cont("%lld", *ptr.p_s64);
+ v4l2_ctrl_log_fp(*ptr.p_s64, ctrl->fraction_bits);
break;
case V4L2_CTRL_TYPE_STRING:
pr_cont("%s", ptr.p_char);
break;
case V4L2_CTRL_TYPE_U8:
- pr_cont("%u", (unsigned)*ptr.p_u8);
+ v4l2_ctrl_log_fp(*ptr.p_u8, ctrl->fraction_bits);
break;
case V4L2_CTRL_TYPE_U16:
- pr_cont("%u", (unsigned)*ptr.p_u16);
+ v4l2_ctrl_log_fp(*ptr.p_u16, ctrl->fraction_bits);
break;
case V4L2_CTRL_TYPE_U32:
- pr_cont("%u", (unsigned)*ptr.p_u32);
+ v4l2_ctrl_log_fp(*ptr.p_u32, ctrl->fraction_bits);
break;
case V4L2_CTRL_TYPE_H264_SPS:
pr_cont("H264_SPS");
@@ -1753,11 +1802,12 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct v4l2_ctrl_handler *hdl,
u32 id, const char *name, enum v4l2_ctrl_type type,
s64 min, s64 max, u64 step, s64 def,
const u32 dims[V4L2_CTRL_MAX_DIMS], u32 elem_size,
- u32 flags, const char * const *qmenu,
+ u32 fraction_bits, u32 flags, const char * const *qmenu,
const s64 *qmenu_int, const union v4l2_ctrl_ptr p_def,
void *priv)
{
struct v4l2_ctrl *ctrl;
+ unsigned int max_fraction_bits = 0;
unsigned sz_extra;
unsigned nr_of_dims = 0;
unsigned elems = 1;
@@ -1779,20 +1829,28 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct v4l2_ctrl_handler *hdl,
/* Prefill elem_size for all types handled by std_type_ops */
switch ((u32)type) {
+ case V4L2_CTRL_TYPE_INTEGER:
+ elem_size = sizeof(s32);
+ max_fraction_bits = 31;
+ break;
case V4L2_CTRL_TYPE_INTEGER64:
elem_size = sizeof(s64);
+ max_fraction_bits = 63;
break;
case V4L2_CTRL_TYPE_STRING:
elem_size = max + 1;
break;
case V4L2_CTRL_TYPE_U8:
elem_size = sizeof(u8);
+ max_fraction_bits = 8;
break;
case V4L2_CTRL_TYPE_U16:
elem_size = sizeof(u16);
+ max_fraction_bits = 16;
break;
case V4L2_CTRL_TYPE_U32:
elem_size = sizeof(u32);
+ max_fraction_bits = 32;
break;
case V4L2_CTRL_TYPE_MPEG2_SEQUENCE:
elem_size = sizeof(struct v4l2_ctrl_mpeg2_sequence);
@@ -1876,10 +1934,10 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct v4l2_ctrl_handler *hdl,
}
/* Sanity checks */
- if (id == 0 || name == NULL || !elem_size ||
- id >= V4L2_CID_PRIVATE_BASE ||
- (type == V4L2_CTRL_TYPE_MENU && qmenu == NULL) ||
- (type == V4L2_CTRL_TYPE_INTEGER_MENU && qmenu_int == NULL)) {
+ if (id == 0 || !name || !elem_size ||
+ fraction_bits > max_fraction_bits || id >= V4L2_CID_PRIVATE_BASE ||
+ (type == V4L2_CTRL_TYPE_MENU && !qmenu) ||
+ (type == V4L2_CTRL_TYPE_INTEGER_MENU && !qmenu_int)) {
handler_set_err(hdl, -ERANGE);
return NULL;
}
@@ -1940,6 +1998,7 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct v4l2_ctrl_handler *hdl,
ctrl->name = name;
ctrl->type = type;
ctrl->flags = flags;
+ ctrl->fraction_bits = fraction_bits;
ctrl->minimum = min;
ctrl->maximum = max;
ctrl->step = step;
@@ -2038,7 +2097,7 @@ struct v4l2_ctrl *v4l2_ctrl_new_custom(struct v4l2_ctrl_handler *hdl,
ctrl = v4l2_ctrl_new(hdl, cfg->ops, cfg->type_ops, cfg->id, name,
type, min, max,
is_menu ? cfg->menu_skip_mask : step, def,
- cfg->dims, cfg->elem_size,
+ cfg->dims, cfg->elem_size, cfg->fraction_bits,
flags, qmenu, qmenu_int, cfg->p_def, priv);
if (ctrl)
ctrl->is_private = cfg->is_private;
@@ -2063,7 +2122,7 @@ struct v4l2_ctrl *v4l2_ctrl_new_std(struct v4l2_ctrl_handler *hdl,
return NULL;
}
return v4l2_ctrl_new(hdl, ops, NULL, id, name, type,
- min, max, step, def, NULL, 0,
+ min, max, step, def, NULL, 0, 0,
flags, NULL, NULL, ptr_null, NULL);
}
EXPORT_SYMBOL(v4l2_ctrl_new_std);
@@ -2096,7 +2155,7 @@ struct v4l2_ctrl *v4l2_ctrl_new_std_menu(struct v4l2_ctrl_handler *hdl,
return NULL;
}
return v4l2_ctrl_new(hdl, ops, NULL, id, name, type,
- 0, max, mask, def, NULL, 0,
+ 0, max, mask, def, NULL, 0, 0,
flags, qmenu, qmenu_int, ptr_null, NULL);
}
EXPORT_SYMBOL(v4l2_ctrl_new_std_menu);
@@ -2128,7 +2187,7 @@ struct v4l2_ctrl *v4l2_ctrl_new_std_menu_items(struct v4l2_ctrl_handler *hdl,
return NULL;
}
return v4l2_ctrl_new(hdl, ops, NULL, id, name, type,
- 0, max, mask, def, NULL, 0,
+ 0, max, mask, def, NULL, 0, 0,
flags, qmenu, NULL, ptr_null, NULL);
}
@@ -2150,7 +2209,7 @@ struct v4l2_ctrl *v4l2_ctrl_new_std_compound(struct v4l2_ctrl_handler *hdl,
return NULL;
}
return v4l2_ctrl_new(hdl, ops, NULL, id, name, type,
- min, max, step, def, NULL, 0,
+ min, max, step, def, NULL, 0, 0,
flags, NULL, NULL, p_def, NULL);
}
EXPORT_SYMBOL(v4l2_ctrl_new_std_compound);
@@ -2174,7 +2233,7 @@ struct v4l2_ctrl *v4l2_ctrl_new_int_menu(struct v4l2_ctrl_handler *hdl,
return NULL;
}
return v4l2_ctrl_new(hdl, ops, NULL, id, name, type,
- 0, max, 0, def, NULL, 0,
+ 0, max, 0, def, NULL, 0, 0,
flags, NULL, qmenu_int, ptr_null, NULL);
}
EXPORT_SYMBOL(v4l2_ctrl_new_int_menu);
diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
index 59679a42b3e7..c35514c5bf88 100644
--- a/include/media/v4l2-ctrls.h
+++ b/include/media/v4l2-ctrls.h
@@ -211,7 +211,8 @@ typedef void (*v4l2_ctrl_notify_fnc)(struct v4l2_ctrl *ctrl, void *priv);
* except for dynamic arrays. In that case it is in the range of
* 1 to @p_array_alloc_elems.
* @dims: The size of each dimension.
- * @nr_of_dims:The number of dimensions in @dims.
+ * @nr_of_dims: The number of dimensions in @dims.
+ * @fraction_bits: The number of fraction bits for fixed point values.
* @menu_skip_mask: The control's skip mask for menu controls. This makes it
* easy to skip menu items that are not valid. If bit X is set,
* then menu item X is skipped. Of course, this only works for
@@ -228,6 +229,7 @@ typedef void (*v4l2_ctrl_notify_fnc)(struct v4l2_ctrl *ctrl, void *priv);
* :math:`ceil(\frac{maximum - minimum}{step}) + 1`.
* Used only if the @type is %V4L2_CTRL_TYPE_INTEGER_MENU.
* @flags: The control's flags.
+ * @fraction_bits: The number of fraction bits for fixed point values.
* @priv: The control's private pointer. For use by the driver. It is
* untouched by the control framework. Note that this pointer is
* not freed when the control is deleted. Should this be needed
@@ -286,6 +288,7 @@ struct v4l2_ctrl {
u32 new_elems;
u32 dims[V4L2_CTRL_MAX_DIMS];
u32 nr_of_dims;
+ u32 fraction_bits;
union {
u64 step;
u64 menu_skip_mask;
@@ -426,6 +429,7 @@ struct v4l2_ctrl_handler {
* @dims: The size of each dimension.
* @elem_size: The size in bytes of the control.
* @flags: The control's flags.
+ * @fraction_bits: The number of fraction bits for fixed point values.
* @menu_skip_mask: The control's skip mask for menu controls. This makes it
* easy to skip menu items that are not valid. If bit X is set,
* then menu item X is skipped. Of course, this only works for
@@ -455,6 +459,7 @@ struct v4l2_ctrl_config {
u32 dims[V4L2_CTRL_MAX_DIMS];
u32 elem_size;
u32 flags;
+ u32 fraction_bits;
u64 menu_skip_mask;
const char * const *qmenu;
const s64 *qmenu_int;
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index a8015e5e7fa4..b8573e9ccde6 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -1947,7 +1947,8 @@ struct v4l2_query_ext_ctrl {
__u32 elems;
__u32 nr_of_dims;
__u32 dims[V4L2_CTRL_MAX_DIMS];
- __u32 reserved[32];
+ __u32 fraction_bits;
+ __u32 reserved[31];
};
/* Used in the VIDIOC_QUERYMENU ioctl for querying menu items */
--
2.34.1
Hi Shengjiu,
I was about to prepare a pull request for this when I realized that
entries for the new drivers need to be added to the MAINTAINERS file.
Can you post patches 'v14 17/16' and 'v14/18/16' adding entries for each
new driver?
Thank you,
Hans
On 11/03/2024 10:53 am, Shengjiu Wang wrote:
> Audio signal processing also has the requirement for memory to
> memory similar as Video.
>
> This asrc memory to memory (memory ->asrc->memory) case is a non
> real time use case.
>
> User fills the input buffer to the asrc module, after conversion, then asrc
> sends back the output buffer to user. So it is not a traditional ALSA playback
> and capture case.
>
> It is a specific use case, there is no reference in current kernel.
> v4l2 memory to memory is the closed implementation, v4l2 current
> support video, image, radio, tuner, touch devices, so it is not
> complicated to add support for this specific audio case.
>
> Because we had implemented the "memory -> asrc ->i2s device-> codec"
> use case in ALSA. Now the "memory->asrc->memory" needs
> to reuse the code in asrc driver, so the first 3 patches is for refining
> the code to make it can be shared by the "memory->asrc->memory"
> driver.
>
> The main change is in the v4l2 side, A /dev/vl4-audioX will be created,
> user applications only use the ioctl of v4l2 framework.
>
> Other change is to add memory to memory support for two kinds of i.MX ASRC
> module.
>
> changes in v14:
> - document the reservation of 'AUXX' fourcc format.
> - add v4l2_audfmt_to_fourcc() definition.
>
> changes in v13
> - change 'pixelformat' to 'audioformat' in dev-audio-mem2mem.rst
> - add more description for clock drift in ext-ctrls-audio-m2m.rst
> - Add "media: v4l2-ctrls: add support for fraction_bits" from Hans
> to avoid build issue for kernel test robot
>
> changes in v12
> - minor changes according to comments
> - drop min_buffers_needed = 1 and V4L2_CTRL_FLAG_UPDATE flag
> - drop bus_info
>
> changes in v11
> - add add-fixed-point-test-controls in vivid.
> - add v4l2_ctrl_fp_compose() helper function for min and max
>
> changes in v10
> - remove FIXED_POINT type
> - change code base on media: v4l2-ctrls: add support for fraction_bits
> - fix issue reported by kernel test robot
> - remove module_alias
>
> changes in v9:
> - add MEDIA_ENT_F_PROC_AUDIO_RESAMPLER.
> - add MEDIA_INTF_T_V4L_AUDIO
> - add media controller support
> - refine the vim2m-audio to support 8k<->16k conversion.
>
> changes in v8:
> - refine V4L2_CAP_AUDIO_M2M to be 0x00000008
> - update doc for FIXED_POINT
> - address comments for imx-asrc
>
> changes in v7:
> - add acked-by from Mark
> - separate commit for fixed point, m2m audio class, audio rate controls
> - use INTEGER_MENU for rate, FIXED_POINT for rate offset
> - remove used fmts
> - address other comments for Hans
>
> changes in v6:
> - use m2m_prepare/m2m_unprepare/m2m_start/m2m_stop to replace
> m2m_start_part_one/m2m_stop_part_one, m2m_start_part_two/m2m_stop_part_two.
> - change V4L2_CTRL_TYPE_ASRC_RATE to V4L2_CTRL_TYPE_FIXED_POINT
> - fix warning by kernel test rebot
> - remove some unused format V4L2_AUDIO_FMT_XX
> - Get SNDRV_PCM_FORMAT from V4L2_AUDIO_FMT in driver.
> - rename audm2m to viaudm2m.
>
> changes in v5:
> - remove V4L2_AUDIO_FMT_LPCM
> - define audio pixel format like V4L2_AUDIO_FMT_S8...
> - remove rate and format in struct v4l2_audio_format.
> - Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE controls
> - updata document accordingly.
>
> changes in v4:
> - update document style
> - separate V4L2_AUDIO_FMT_LPCM and V4L2_CAP_AUDIO_M2M in separate commit
>
> changes in v3:
> - Modify documents for adding audio m2m support
> - Add audio virtual m2m driver
> - Defined V4L2_AUDIO_FMT_LPCM format type for audio.
> - Defined V4L2_CAP_AUDIO_M2M capability type for audio m2m case.
> - with modification in v4l-utils, pass v4l2-compliance test.
>
> changes in v2:
> - decouple the implementation in v4l2 and ALSA
> - implement the memory to memory driver as a platfrom driver
> and move it to driver/media
> - move fsl_asrc_common.h to include/sound folder
>
> Hans Verkuil (1):
> media: v4l2-ctrls: add support for fraction_bits
>
> Shengjiu Wang (15):
> ASoC: fsl_asrc: define functions for memory to memory usage
> ASoC: fsl_easrc: define functions for memory to memory usage
> ASoC: fsl_asrc: move fsl_asrc_common.h to include/sound
> ASoC: fsl_asrc: register m2m platform device
> ASoC: fsl_easrc: register m2m platform device
> media: uapi: Add V4L2_CAP_AUDIO_M2M capability flag
> media: v4l2: Add audio capture and output support
> media: uapi: Define audio sample format fourcc type
> media: uapi: Add V4L2_CTRL_CLASS_M2M_AUDIO
> media: uapi: Add audio rate controls support
> media: uapi: Declare interface types for Audio
> media: uapi: Add an entity type for audio resampler
> media: vivid: add fixed point test controls
> media: imx-asrc: Add memory to memory driver
> media: vim2m-audio: add virtual driver for audio memory to memory
>
> .../media/mediactl/media-types.rst | 11 +
> .../userspace-api/media/v4l/buffer.rst | 6 +
> .../userspace-api/media/v4l/common.rst | 1 +
> .../media/v4l/dev-audio-mem2mem.rst | 71 +
> .../userspace-api/media/v4l/devices.rst | 1 +
> .../media/v4l/ext-ctrls-audio-m2m.rst | 59 +
> .../userspace-api/media/v4l/pixfmt-audio.rst | 100 ++
> .../userspace-api/media/v4l/pixfmt.rst | 1 +
> .../media/v4l/vidioc-enum-fmt.rst | 2 +
> .../media/v4l/vidioc-g-ext-ctrls.rst | 4 +
> .../userspace-api/media/v4l/vidioc-g-fmt.rst | 4 +
> .../media/v4l/vidioc-querycap.rst | 3 +
> .../media/v4l/vidioc-queryctrl.rst | 11 +-
> .../media/videodev2.h.rst.exceptions | 3 +
> .../media/common/videobuf2/videobuf2-v4l2.c | 4 +
> drivers/media/platform/nxp/Kconfig | 13 +
> drivers/media/platform/nxp/Makefile | 1 +
> drivers/media/platform/nxp/imx-asrc.c | 1256 +++++++++++++++++
> drivers/media/test-drivers/Kconfig | 10 +
> drivers/media/test-drivers/Makefile | 1 +
> drivers/media/test-drivers/vim2m-audio.c | 793 +++++++++++
> drivers/media/test-drivers/vivid/vivid-core.h | 2 +
> .../media/test-drivers/vivid/vivid-ctrls.c | 26 +
> drivers/media/v4l2-core/v4l2-compat-ioctl32.c | 9 +
> drivers/media/v4l2-core/v4l2-ctrls-api.c | 1 +
> drivers/media/v4l2-core/v4l2-ctrls-core.c | 93 +-
> drivers/media/v4l2-core/v4l2-ctrls-defs.c | 10 +
> drivers/media/v4l2-core/v4l2-dev.c | 21 +
> drivers/media/v4l2-core/v4l2-ioctl.c | 66 +
> drivers/media/v4l2-core/v4l2-mem2mem.c | 13 +-
> include/media/v4l2-ctrls.h | 13 +-
> include/media/v4l2-dev.h | 2 +
> include/media/v4l2-ioctl.h | 34 +
> .../fsl => include/sound}/fsl_asrc_common.h | 60 +
> include/uapi/linux/media.h | 2 +
> include/uapi/linux/v4l2-controls.h | 9 +
> include/uapi/linux/videodev2.h | 50 +-
> sound/soc/fsl/fsl_asrc.c | 144 ++
> sound/soc/fsl/fsl_asrc.h | 4 +-
> sound/soc/fsl/fsl_asrc_dma.c | 2 +-
> sound/soc/fsl/fsl_easrc.c | 233 +++
> sound/soc/fsl/fsl_easrc.h | 6 +-
> 42 files changed, 3128 insertions(+), 27 deletions(-)
> create mode 100644 Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst
> create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-audio-m2m.rst
> create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-audio.rst
> create mode 100644 drivers/media/platform/nxp/imx-asrc.c
> create mode 100644 drivers/media/test-drivers/vim2m-audio.c
> rename {sound/soc/fsl => include/sound}/fsl_asrc_common.h (60%)
>
On Mon, Mar 18, 2024 at 11:56 PM Hans Verkuil <[email protected]> wrote:
>
> Hi Shengjiu,
>
> I was about to prepare a pull request for this when I realized that
> entries for the new drivers need to be added to the MAINTAINERS file.
>
> Can you post patches 'v14 17/16' and 'v14/18/16' adding entries for each
> new driver?
Ok, How about the change below?
VIRTUAL MEM2MEM DRIVER FOR AUDIO
M: Hans Verkuil <[email protected]>
M: Shengjiu Wang <[email protected]>
L: [email protected]
S: Maintained
W: https://linuxtv.org
T: git git://linuxtv.org/media_tree.git
F: drivers/media/test-drivers/vim2m-audio.c
NXP ASRC V4L2 MEM2MEM DRIVERS
M: Shengjiu Wang <[email protected]>
L: [email protected]
S: Maintained
W: https://linuxtv.org
T: git git://linuxtv.org/media_tree.git
F: drivers/media/platform/nxp/imx-asrc.c
best regards
wang shengjiu
>
> Thank you,
>
> Hans
>
> On 11/03/2024 10:53 am, Shengjiu Wang wrote:
> > Audio signal processing also has the requirement for memory to
> > memory similar as Video.
> >
> > This asrc memory to memory (memory ->asrc->memory) case is a non
> > real time use case.
> >
> > User fills the input buffer to the asrc module, after conversion, then asrc
> > sends back the output buffer to user. So it is not a traditional ALSA playback
> > and capture case.
> >
> > It is a specific use case, there is no reference in current kernel.
> > v4l2 memory to memory is the closed implementation, v4l2 current
> > support video, image, radio, tuner, touch devices, so it is not
> > complicated to add support for this specific audio case.
> >
> > Because we had implemented the "memory -> asrc ->i2s device-> codec"
> > use case in ALSA. Now the "memory->asrc->memory" needs
> > to reuse the code in asrc driver, so the first 3 patches is for refining
> > the code to make it can be shared by the "memory->asrc->memory"
> > driver.
> >
> > The main change is in the v4l2 side, A /dev/vl4-audioX will be created,
> > user applications only use the ioctl of v4l2 framework.
> >
> > Other change is to add memory to memory support for two kinds of i.MX ASRC
> > module.
> >
> > changes in v14:
> > - document the reservation of 'AUXX' fourcc format.
> > - add v4l2_audfmt_to_fourcc() definition.
> >
> > changes in v13
> > - change 'pixelformat' to 'audioformat' in dev-audio-mem2mem.rst
> > - add more description for clock drift in ext-ctrls-audio-m2m.rst
> > - Add "media: v4l2-ctrls: add support for fraction_bits" from Hans
> > to avoid build issue for kernel test robot
> >
> > changes in v12
> > - minor changes according to comments
> > - drop min_buffers_needed = 1 and V4L2_CTRL_FLAG_UPDATE flag
> > - drop bus_info
> >
> > changes in v11
> > - add add-fixed-point-test-controls in vivid.
> > - add v4l2_ctrl_fp_compose() helper function for min and max
> >
> > changes in v10
> > - remove FIXED_POINT type
> > - change code base on media: v4l2-ctrls: add support for fraction_bits
> > - fix issue reported by kernel test robot
> > - remove module_alias
> >
> > changes in v9:
> > - add MEDIA_ENT_F_PROC_AUDIO_RESAMPLER.
> > - add MEDIA_INTF_T_V4L_AUDIO
> > - add media controller support
> > - refine the vim2m-audio to support 8k<->16k conversion.
> >
> > changes in v8:
> > - refine V4L2_CAP_AUDIO_M2M to be 0x00000008
> > - update doc for FIXED_POINT
> > - address comments for imx-asrc
> >
> > changes in v7:
> > - add acked-by from Mark
> > - separate commit for fixed point, m2m audio class, audio rate controls
> > - use INTEGER_MENU for rate, FIXED_POINT for rate offset
> > - remove used fmts
> > - address other comments for Hans
> >
> > changes in v6:
> > - use m2m_prepare/m2m_unprepare/m2m_start/m2m_stop to replace
> > m2m_start_part_one/m2m_stop_part_one, m2m_start_part_two/m2m_stop_part_two.
> > - change V4L2_CTRL_TYPE_ASRC_RATE to V4L2_CTRL_TYPE_FIXED_POINT
> > - fix warning by kernel test rebot
> > - remove some unused format V4L2_AUDIO_FMT_XX
> > - Get SNDRV_PCM_FORMAT from V4L2_AUDIO_FMT in driver.
> > - rename audm2m to viaudm2m.
> >
> > changes in v5:
> > - remove V4L2_AUDIO_FMT_LPCM
> > - define audio pixel format like V4L2_AUDIO_FMT_S8...
> > - remove rate and format in struct v4l2_audio_format.
> > - Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE controls
> > - updata document accordingly.
> >
> > changes in v4:
> > - update document style
> > - separate V4L2_AUDIO_FMT_LPCM and V4L2_CAP_AUDIO_M2M in separate commit
> >
> > changes in v3:
> > - Modify documents for adding audio m2m support
> > - Add audio virtual m2m driver
> > - Defined V4L2_AUDIO_FMT_LPCM format type for audio.
> > - Defined V4L2_CAP_AUDIO_M2M capability type for audio m2m case.
> > - with modification in v4l-utils, pass v4l2-compliance test.
> >
> > changes in v2:
> > - decouple the implementation in v4l2 and ALSA
> > - implement the memory to memory driver as a platfrom driver
> > and move it to driver/media
> > - move fsl_asrc_common.h to include/sound folder
> >
> > Hans Verkuil (1):
> > media: v4l2-ctrls: add support for fraction_bits
> >
> > Shengjiu Wang (15):
> > ASoC: fsl_asrc: define functions for memory to memory usage
> > ASoC: fsl_easrc: define functions for memory to memory usage
> > ASoC: fsl_asrc: move fsl_asrc_common.h to include/sound
> > ASoC: fsl_asrc: register m2m platform device
> > ASoC: fsl_easrc: register m2m platform device
> > media: uapi: Add V4L2_CAP_AUDIO_M2M capability flag
> > media: v4l2: Add audio capture and output support
> > media: uapi: Define audio sample format fourcc type
> > media: uapi: Add V4L2_CTRL_CLASS_M2M_AUDIO
> > media: uapi: Add audio rate controls support
> > media: uapi: Declare interface types for Audio
> > media: uapi: Add an entity type for audio resampler
> > media: vivid: add fixed point test controls
> > media: imx-asrc: Add memory to memory driver
> > media: vim2m-audio: add virtual driver for audio memory to memory
> >
> > .../media/mediactl/media-types.rst | 11 +
> > .../userspace-api/media/v4l/buffer.rst | 6 +
> > .../userspace-api/media/v4l/common.rst | 1 +
> > .../media/v4l/dev-audio-mem2mem.rst | 71 +
> > .../userspace-api/media/v4l/devices.rst | 1 +
> > .../media/v4l/ext-ctrls-audio-m2m.rst | 59 +
> > .../userspace-api/media/v4l/pixfmt-audio.rst | 100 ++
> > .../userspace-api/media/v4l/pixfmt.rst | 1 +
> > .../media/v4l/vidioc-enum-fmt.rst | 2 +
> > .../media/v4l/vidioc-g-ext-ctrls.rst | 4 +
> > .../userspace-api/media/v4l/vidioc-g-fmt.rst | 4 +
> > .../media/v4l/vidioc-querycap.rst | 3 +
> > .../media/v4l/vidioc-queryctrl.rst | 11 +-
> > .../media/videodev2.h.rst.exceptions | 3 +
> > .../media/common/videobuf2/videobuf2-v4l2.c | 4 +
> > drivers/media/platform/nxp/Kconfig | 13 +
> > drivers/media/platform/nxp/Makefile | 1 +
> > drivers/media/platform/nxp/imx-asrc.c | 1256 +++++++++++++++++
> > drivers/media/test-drivers/Kconfig | 10 +
> > drivers/media/test-drivers/Makefile | 1 +
> > drivers/media/test-drivers/vim2m-audio.c | 793 +++++++++++
> > drivers/media/test-drivers/vivid/vivid-core.h | 2 +
> > .../media/test-drivers/vivid/vivid-ctrls.c | 26 +
> > drivers/media/v4l2-core/v4l2-compat-ioctl32.c | 9 +
> > drivers/media/v4l2-core/v4l2-ctrls-api.c | 1 +
> > drivers/media/v4l2-core/v4l2-ctrls-core.c | 93 +-
> > drivers/media/v4l2-core/v4l2-ctrls-defs.c | 10 +
> > drivers/media/v4l2-core/v4l2-dev.c | 21 +
> > drivers/media/v4l2-core/v4l2-ioctl.c | 66 +
> > drivers/media/v4l2-core/v4l2-mem2mem.c | 13 +-
> > include/media/v4l2-ctrls.h | 13 +-
> > include/media/v4l2-dev.h | 2 +
> > include/media/v4l2-ioctl.h | 34 +
> > .../fsl => include/sound}/fsl_asrc_common.h | 60 +
> > include/uapi/linux/media.h | 2 +
> > include/uapi/linux/v4l2-controls.h | 9 +
> > include/uapi/linux/videodev2.h | 50 +-
> > sound/soc/fsl/fsl_asrc.c | 144 ++
> > sound/soc/fsl/fsl_asrc.h | 4 +-
> > sound/soc/fsl/fsl_asrc_dma.c | 2 +-
> > sound/soc/fsl/fsl_easrc.c | 233 +++
> > sound/soc/fsl/fsl_easrc.h | 6 +-
> > 42 files changed, 3128 insertions(+), 27 deletions(-)
> > create mode 100644 Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst
> > create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-audio-m2m.rst
> > create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-audio.rst
> > create mode 100644 drivers/media/platform/nxp/imx-asrc.c
> > create mode 100644 drivers/media/test-drivers/vim2m-audio.c
> > rename {sound/soc/fsl => include/sound}/fsl_asrc_common.h (60%)
> >
>
On Tue, Mar 19, 2024 at 10:24 AM Shengjiu Wang <shengjiu.wang@gmailcom> wrote:
>
> On Mon, Mar 18, 2024 at 11:56 PM Hans Verkuil <[email protected]> wrote:
> >
> > Hi Shengjiu,
> >
> > I was about to prepare a pull request for this when I realized that
> > entries for the new drivers need to be added to the MAINTAINERS file.
> >
> > Can you post patches 'v14 17/16' and 'v14/18/16' adding entries for each
> > new driver?
>
> Ok, How about the change below?
>
> VIRTUAL MEM2MEM DRIVER FOR AUDIO
> M: Hans Verkuil <[email protected]>
> M: Shengjiu Wang <[email protected]>
> L: [email protected]
> S: Maintained
> W: https://linuxtv.org
> T: git git://linuxtv.org/media_tree.git
> F: drivers/media/test-drivers/vim2m-audio.c
>
> NXP ASRC V4L2 MEM2MEM DRIVERS
> M: Shengjiu Wang <[email protected]>
> L: [email protected]
> S: Maintained
> W: https://linuxtv.org
> T: git git://linuxtv.org/media_tree.git
> F: drivers/media/platform/nxp/imx-asrc.c
>
BTW, should I merge this change to PATCH 15/16 and PATCH 16/16?
That I will resend the whole patch series. Seems this is a better way.
Best regrads
Shengjiu Wang
> best regards
> wang shengjiu
>
> >
> > Thank you,
> >
> > Hans
> >
> > On 11/03/2024 10:53 am, Shengjiu Wang wrote:
> > > Audio signal processing also has the requirement for memory to
> > > memory similar as Video.
> > >
> > > This asrc memory to memory (memory ->asrc->memory) case is a non
> > > real time use case.
> > >
> > > User fills the input buffer to the asrc module, after conversion, then asrc
> > > sends back the output buffer to user. So it is not a traditional ALSA playback
> > > and capture case.
> > >
> > > It is a specific use case, there is no reference in current kernel.
> > > v4l2 memory to memory is the closed implementation, v4l2 current
> > > support video, image, radio, tuner, touch devices, so it is not
> > > complicated to add support for this specific audio case.
> > >
> > > Because we had implemented the "memory -> asrc ->i2s device-> codec"
> > > use case in ALSA. Now the "memory->asrc->memory" needs
> > > to reuse the code in asrc driver, so the first 3 patches is for refining
> > > the code to make it can be shared by the "memory->asrc->memory"
> > > driver.
> > >
> > > The main change is in the v4l2 side, A /dev/vl4-audioX will be created,
> > > user applications only use the ioctl of v4l2 framework.
> > >
> > > Other change is to add memory to memory support for two kinds of i.MX ASRC
> > > module.
> > >
> > > changes in v14:
> > > - document the reservation of 'AUXX' fourcc format.
> > > - add v4l2_audfmt_to_fourcc() definition.
> > >
> > > changes in v13
> > > - change 'pixelformat' to 'audioformat' in dev-audio-mem2mem.rst
> > > - add more description for clock drift in ext-ctrls-audio-m2m.rst
> > > - Add "media: v4l2-ctrls: add support for fraction_bits" from Hans
> > > to avoid build issue for kernel test robot
> > >
> > > changes in v12
> > > - minor changes according to comments
> > > - drop min_buffers_needed = 1 and V4L2_CTRL_FLAG_UPDATE flag
> > > - drop bus_info
> > >
> > > changes in v11
> > > - add add-fixed-point-test-controls in vivid.
> > > - add v4l2_ctrl_fp_compose() helper function for min and max
> > >
> > > changes in v10
> > > - remove FIXED_POINT type
> > > - change code base on media: v4l2-ctrls: add support for fraction_bits
> > > - fix issue reported by kernel test robot
> > > - remove module_alias
> > >
> > > changes in v9:
> > > - add MEDIA_ENT_F_PROC_AUDIO_RESAMPLER.
> > > - add MEDIA_INTF_T_V4L_AUDIO
> > > - add media controller support
> > > - refine the vim2m-audio to support 8k<->16k conversion.
> > >
> > > changes in v8:
> > > - refine V4L2_CAP_AUDIO_M2M to be 0x00000008
> > > - update doc for FIXED_POINT
> > > - address comments for imx-asrc
> > >
> > > changes in v7:
> > > - add acked-by from Mark
> > > - separate commit for fixed point, m2m audio class, audio rate controls
> > > - use INTEGER_MENU for rate, FIXED_POINT for rate offset
> > > - remove used fmts
> > > - address other comments for Hans
> > >
> > > changes in v6:
> > > - use m2m_prepare/m2m_unprepare/m2m_start/m2m_stop to replace
> > > m2m_start_part_one/m2m_stop_part_one, m2m_start_part_two/m2m_stop_part_two.
> > > - change V4L2_CTRL_TYPE_ASRC_RATE to V4L2_CTRL_TYPE_FIXED_POINT
> > > - fix warning by kernel test rebot
> > > - remove some unused format V4L2_AUDIO_FMT_XX
> > > - Get SNDRV_PCM_FORMAT from V4L2_AUDIO_FMT in driver.
> > > - rename audm2m to viaudm2m.
> > >
> > > changes in v5:
> > > - remove V4L2_AUDIO_FMT_LPCM
> > > - define audio pixel format like V4L2_AUDIO_FMT_S8...
> > > - remove rate and format in struct v4l2_audio_format.
> > > - Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE controls
> > > - updata document accordingly.
> > >
> > > changes in v4:
> > > - update document style
> > > - separate V4L2_AUDIO_FMT_LPCM and V4L2_CAP_AUDIO_M2M in separate commit
> > >
> > > changes in v3:
> > > - Modify documents for adding audio m2m support
> > > - Add audio virtual m2m driver
> > > - Defined V4L2_AUDIO_FMT_LPCM format type for audio.
> > > - Defined V4L2_CAP_AUDIO_M2M capability type for audio m2m case.
> > > - with modification in v4l-utils, pass v4l2-compliance test.
> > >
> > > changes in v2:
> > > - decouple the implementation in v4l2 and ALSA
> > > - implement the memory to memory driver as a platfrom driver
> > > and move it to driver/media
> > > - move fsl_asrc_common.h to include/sound folder
> > >
> > > Hans Verkuil (1):
> > > media: v4l2-ctrls: add support for fraction_bits
> > >
> > > Shengjiu Wang (15):
> > > ASoC: fsl_asrc: define functions for memory to memory usage
> > > ASoC: fsl_easrc: define functions for memory to memory usage
> > > ASoC: fsl_asrc: move fsl_asrc_common.h to include/sound
> > > ASoC: fsl_asrc: register m2m platform device
> > > ASoC: fsl_easrc: register m2m platform device
> > > media: uapi: Add V4L2_CAP_AUDIO_M2M capability flag
> > > media: v4l2: Add audio capture and output support
> > > media: uapi: Define audio sample format fourcc type
> > > media: uapi: Add V4L2_CTRL_CLASS_M2M_AUDIO
> > > media: uapi: Add audio rate controls support
> > > media: uapi: Declare interface types for Audio
> > > media: uapi: Add an entity type for audio resampler
> > > media: vivid: add fixed point test controls
> > > media: imx-asrc: Add memory to memory driver
> > > media: vim2m-audio: add virtual driver for audio memory to memory
> > >
> > > .../media/mediactl/media-types.rst | 11 +
> > > .../userspace-api/media/v4l/buffer.rst | 6 +
> > > .../userspace-api/media/v4l/common.rst | 1 +
> > > .../media/v4l/dev-audio-mem2mem.rst | 71 +
> > > .../userspace-api/media/v4l/devices.rst | 1 +
> > > .../media/v4l/ext-ctrls-audio-m2m.rst | 59 +
> > > .../userspace-api/media/v4l/pixfmt-audio.rst | 100 ++
> > > .../userspace-api/media/v4l/pixfmt.rst | 1 +
> > > .../media/v4l/vidioc-enum-fmt.rst | 2 +
> > > .../media/v4l/vidioc-g-ext-ctrls.rst | 4 +
> > > .../userspace-api/media/v4l/vidioc-g-fmt.rst | 4 +
> > > .../media/v4l/vidioc-querycap.rst | 3 +
> > > .../media/v4l/vidioc-queryctrl.rst | 11 +-
> > > .../media/videodev2.h.rst.exceptions | 3 +
> > > .../media/common/videobuf2/videobuf2-v4l2.c | 4 +
> > > drivers/media/platform/nxp/Kconfig | 13 +
> > > drivers/media/platform/nxp/Makefile | 1 +
> > > drivers/media/platform/nxp/imx-asrc.c | 1256 +++++++++++++++++
> > > drivers/media/test-drivers/Kconfig | 10 +
> > > drivers/media/test-drivers/Makefile | 1 +
> > > drivers/media/test-drivers/vim2m-audio.c | 793 +++++++++++
> > > drivers/media/test-drivers/vivid/vivid-core.h | 2 +
> > > .../media/test-drivers/vivid/vivid-ctrls.c | 26 +
> > > drivers/media/v4l2-core/v4l2-compat-ioctl32.c | 9 +
> > > drivers/media/v4l2-core/v4l2-ctrls-api.c | 1 +
> > > drivers/media/v4l2-core/v4l2-ctrls-core.c | 93 +-
> > > drivers/media/v4l2-core/v4l2-ctrls-defs.c | 10 +
> > > drivers/media/v4l2-core/v4l2-dev.c | 21 +
> > > drivers/media/v4l2-core/v4l2-ioctl.c | 66 +
> > > drivers/media/v4l2-core/v4l2-mem2mem.c | 13 +-
> > > include/media/v4l2-ctrls.h | 13 +-
> > > include/media/v4l2-dev.h | 2 +
> > > include/media/v4l2-ioctl.h | 34 +
> > > .../fsl => include/sound}/fsl_asrc_common.h | 60 +
> > > include/uapi/linux/media.h | 2 +
> > > include/uapi/linux/v4l2-controls.h | 9 +
> > > include/uapi/linux/videodev2.h | 50 +-
> > > sound/soc/fsl/fsl_asrc.c | 144 ++
> > > sound/soc/fsl/fsl_asrc.h | 4 +-
> > > sound/soc/fsl/fsl_asrc_dma.c | 2 +-
> > > sound/soc/fsl/fsl_easrc.c | 233 +++
> > > sound/soc/fsl/fsl_easrc.h | 6 +-
> > > 42 files changed, 3128 insertions(+), 27 deletions(-)
> > > create mode 100644 Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst
> > > create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-audio-m2m.rst
> > > create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-audio.rst
> > > create mode 100644 drivers/media/platform/nxp/imx-asrc.c
> > > create mode 100644 drivers/media/test-drivers/vim2m-audio.c
> > > rename {sound/soc/fsl => include/sound}/fsl_asrc_common.h (60%)
> > >
> >
On 19/03/2024 4:14 am, Shengjiu Wang wrote:
> On Tue, Mar 19, 2024 at 10:24 AM Shengjiu Wang <[email protected]> wrote:
>>
>> On Mon, Mar 18, 2024 at 11:56 PM Hans Verkuil <[email protected]> wrote:
>>>
>>> Hi Shengjiu,
>>>
>>> I was about to prepare a pull request for this when I realized that
>>> entries for the new drivers need to be added to the MAINTAINERS file.
>>>
>>> Can you post patches 'v14 17/16' and 'v14/18/16' adding entries for each
>>> new driver?
>>
>> Ok, How about the change below?
>>
>> VIRTUAL MEM2MEM DRIVER FOR AUDIO
>> M: Hans Verkuil <[email protected]>
>> M: Shengjiu Wang <[email protected]>
>> L: [email protected]
>> S: Maintained
>> W: https://linuxtv.org
>> T: git git://linuxtv.org/media_tree.git
>> F: drivers/media/test-drivers/vim2m-audio.c
>>
>> NXP ASRC V4L2 MEM2MEM DRIVERS
>> M: Shengjiu Wang <[email protected]>
>> L: [email protected]
>> S: Maintained
>> W: https://linuxtv.org
>> T: git git://linuxtv.org/media_tree.git
>> F: drivers/media/platform/nxp/imx-asrc.c
>>
>
> BTW, should I merge this change to PATCH 15/16 and PATCH 16/16?
> That I will resend the whole patch series. Seems this is a better way.
That's fine!
Regards,
Hans
>
> Best regrads
> Shengjiu Wang
>
>> best regards
>> wang shengjiu
>>
>>>
>>> Thank you,
>>>
>>> Hans
>>>
>>> On 11/03/2024 10:53 am, Shengjiu Wang wrote:
>>>> Audio signal processing also has the requirement for memory to
>>>> memory similar as Video.
>>>>
>>>> This asrc memory to memory (memory ->asrc->memory) case is a non
>>>> real time use case.
>>>>
>>>> User fills the input buffer to the asrc module, after conversion, then asrc
>>>> sends back the output buffer to user. So it is not a traditional ALSA playback
>>>> and capture case.
>>>>
>>>> It is a specific use case, there is no reference in current kernel.
>>>> v4l2 memory to memory is the closed implementation, v4l2 current
>>>> support video, image, radio, tuner, touch devices, so it is not
>>>> complicated to add support for this specific audio case.
>>>>
>>>> Because we had implemented the "memory -> asrc ->i2s device-> codec"
>>>> use case in ALSA. Now the "memory->asrc->memory" needs
>>>> to reuse the code in asrc driver, so the first 3 patches is for refining
>>>> the code to make it can be shared by the "memory->asrc->memory"
>>>> driver.
>>>>
>>>> The main change is in the v4l2 side, A /dev/vl4-audioX will be created,
>>>> user applications only use the ioctl of v4l2 framework.
>>>>
>>>> Other change is to add memory to memory support for two kinds of i.MX ASRC
>>>> module.
>>>>
>>>> changes in v14:
>>>> - document the reservation of 'AUXX' fourcc format.
>>>> - add v4l2_audfmt_to_fourcc() definition.
>>>>
>>>> changes in v13
>>>> - change 'pixelformat' to 'audioformat' in dev-audio-mem2mem.rst
>>>> - add more description for clock drift in ext-ctrls-audio-m2m.rst
>>>> - Add "media: v4l2-ctrls: add support for fraction_bits" from Hans
>>>> to avoid build issue for kernel test robot
>>>>
>>>> changes in v12
>>>> - minor changes according to comments
>>>> - drop min_buffers_needed = 1 and V4L2_CTRL_FLAG_UPDATE flag
>>>> - drop bus_info
>>>>
>>>> changes in v11
>>>> - add add-fixed-point-test-controls in vivid.
>>>> - add v4l2_ctrl_fp_compose() helper function for min and max
>>>>
>>>> changes in v10
>>>> - remove FIXED_POINT type
>>>> - change code base on media: v4l2-ctrls: add support for fraction_bits
>>>> - fix issue reported by kernel test robot
>>>> - remove module_alias
>>>>
>>>> changes in v9:
>>>> - add MEDIA_ENT_F_PROC_AUDIO_RESAMPLER.
>>>> - add MEDIA_INTF_T_V4L_AUDIO
>>>> - add media controller support
>>>> - refine the vim2m-audio to support 8k<->16k conversion.
>>>>
>>>> changes in v8:
>>>> - refine V4L2_CAP_AUDIO_M2M to be 0x00000008
>>>> - update doc for FIXED_POINT
>>>> - address comments for imx-asrc
>>>>
>>>> changes in v7:
>>>> - add acked-by from Mark
>>>> - separate commit for fixed point, m2m audio class, audio rate controls
>>>> - use INTEGER_MENU for rate, FIXED_POINT for rate offset
>>>> - remove used fmts
>>>> - address other comments for Hans
>>>>
>>>> changes in v6:
>>>> - use m2m_prepare/m2m_unprepare/m2m_start/m2m_stop to replace
>>>> m2m_start_part_one/m2m_stop_part_one, m2m_start_part_two/m2m_stop_part_two.
>>>> - change V4L2_CTRL_TYPE_ASRC_RATE to V4L2_CTRL_TYPE_FIXED_POINT
>>>> - fix warning by kernel test rebot
>>>> - remove some unused format V4L2_AUDIO_FMT_XX
>>>> - Get SNDRV_PCM_FORMAT from V4L2_AUDIO_FMT in driver.
>>>> - rename audm2m to viaudm2m.
>>>>
>>>> changes in v5:
>>>> - remove V4L2_AUDIO_FMT_LPCM
>>>> - define audio pixel format like V4L2_AUDIO_FMT_S8...
>>>> - remove rate and format in struct v4l2_audio_format.
>>>> - Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE controls
>>>> - updata document accordingly.
>>>>
>>>> changes in v4:
>>>> - update document style
>>>> - separate V4L2_AUDIO_FMT_LPCM and V4L2_CAP_AUDIO_M2M in separate commit
>>>>
>>>> changes in v3:
>>>> - Modify documents for adding audio m2m support
>>>> - Add audio virtual m2m driver
>>>> - Defined V4L2_AUDIO_FMT_LPCM format type for audio.
>>>> - Defined V4L2_CAP_AUDIO_M2M capability type for audio m2m case.
>>>> - with modification in v4l-utils, pass v4l2-compliance test.
>>>>
>>>> changes in v2:
>>>> - decouple the implementation in v4l2 and ALSA
>>>> - implement the memory to memory driver as a platfrom driver
>>>> and move it to driver/media
>>>> - move fsl_asrc_common.h to include/sound folder
>>>>
>>>> Hans Verkuil (1):
>>>> media: v4l2-ctrls: add support for fraction_bits
>>>>
>>>> Shengjiu Wang (15):
>>>> ASoC: fsl_asrc: define functions for memory to memory usage
>>>> ASoC: fsl_easrc: define functions for memory to memory usage
>>>> ASoC: fsl_asrc: move fsl_asrc_common.h to include/sound
>>>> ASoC: fsl_asrc: register m2m platform device
>>>> ASoC: fsl_easrc: register m2m platform device
>>>> media: uapi: Add V4L2_CAP_AUDIO_M2M capability flag
>>>> media: v4l2: Add audio capture and output support
>>>> media: uapi: Define audio sample format fourcc type
>>>> media: uapi: Add V4L2_CTRL_CLASS_M2M_AUDIO
>>>> media: uapi: Add audio rate controls support
>>>> media: uapi: Declare interface types for Audio
>>>> media: uapi: Add an entity type for audio resampler
>>>> media: vivid: add fixed point test controls
>>>> media: imx-asrc: Add memory to memory driver
>>>> media: vim2m-audio: add virtual driver for audio memory to memory
>>>>
>>>> .../media/mediactl/media-types.rst | 11 +
>>>> .../userspace-api/media/v4l/buffer.rst | 6 +
>>>> .../userspace-api/media/v4l/common.rst | 1 +
>>>> .../media/v4l/dev-audio-mem2mem.rst | 71 +
>>>> .../userspace-api/media/v4l/devices.rst | 1 +
>>>> .../media/v4l/ext-ctrls-audio-m2m.rst | 59 +
>>>> .../userspace-api/media/v4l/pixfmt-audio.rst | 100 ++
>>>> .../userspace-api/media/v4l/pixfmt.rst | 1 +
>>>> .../media/v4l/vidioc-enum-fmt.rst | 2 +
>>>> .../media/v4l/vidioc-g-ext-ctrls.rst | 4 +
>>>> .../userspace-api/media/v4l/vidioc-g-fmt.rst | 4 +
>>>> .../media/v4l/vidioc-querycap.rst | 3 +
>>>> .../media/v4l/vidioc-queryctrl.rst | 11 +-
>>>> .../media/videodev2.h.rst.exceptions | 3 +
>>>> .../media/common/videobuf2/videobuf2-v4l2.c | 4 +
>>>> drivers/media/platform/nxp/Kconfig | 13 +
>>>> drivers/media/platform/nxp/Makefile | 1 +
>>>> drivers/media/platform/nxp/imx-asrc.c | 1256 +++++++++++++++++
>>>> drivers/media/test-drivers/Kconfig | 10 +
>>>> drivers/media/test-drivers/Makefile | 1 +
>>>> drivers/media/test-drivers/vim2m-audio.c | 793 +++++++++++
>>>> drivers/media/test-drivers/vivid/vivid-core.h | 2 +
>>>> .../media/test-drivers/vivid/vivid-ctrls.c | 26 +
>>>> drivers/media/v4l2-core/v4l2-compat-ioctl32.c | 9 +
>>>> drivers/media/v4l2-core/v4l2-ctrls-api.c | 1 +
>>>> drivers/media/v4l2-core/v4l2-ctrls-core.c | 93 +-
>>>> drivers/media/v4l2-core/v4l2-ctrls-defs.c | 10 +
>>>> drivers/media/v4l2-core/v4l2-dev.c | 21 +
>>>> drivers/media/v4l2-core/v4l2-ioctl.c | 66 +
>>>> drivers/media/v4l2-core/v4l2-mem2mem.c | 13 +-
>>>> include/media/v4l2-ctrls.h | 13 +-
>>>> include/media/v4l2-dev.h | 2 +
>>>> include/media/v4l2-ioctl.h | 34 +
>>>> .../fsl => include/sound}/fsl_asrc_common.h | 60 +
>>>> include/uapi/linux/media.h | 2 +
>>>> include/uapi/linux/v4l2-controls.h | 9 +
>>>> include/uapi/linux/videodev2.h | 50 +-
>>>> sound/soc/fsl/fsl_asrc.c | 144 ++
>>>> sound/soc/fsl/fsl_asrc.h | 4 +-
>>>> sound/soc/fsl/fsl_asrc_dma.c | 2 +-
>>>> sound/soc/fsl/fsl_easrc.c | 233 +++
>>>> sound/soc/fsl/fsl_easrc.h | 6 +-
>>>> 42 files changed, 3128 insertions(+), 27 deletions(-)
>>>> create mode 100644 Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst
>>>> create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-audio-m2m.rst
>>>> create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-audio.rst
>>>> create mode 100644 drivers/media/platform/nxp/imx-asrc.c
>>>> create mode 100644 drivers/media/test-drivers/vim2m-audio.c
>>>> rename {sound/soc/fsl => include/sound}/fsl_asrc_common.h (60%)
>>>>
>>>
>