By setting the V4L2_BUF_FLAG_CODECCONFIG flag,
user-space should be able to hint decoder
the vb2 only contains codec config header,
but does not contain any frame data.
It's only used for parsing header, and can't be decoded.
Current, it's usually used by android.
Signed-off-by: Ming Qian <[email protected]>
---
Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++
include/uapi/linux/videodev2.h | 2 ++
2 files changed, 11 insertions(+)
diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst
index 4638ec64db00..acdc4556f4f4 100644
--- a/Documentation/userspace-api/media/v4l/buffer.rst
+++ b/Documentation/userspace-api/media/v4l/buffer.rst
@@ -607,6 +607,15 @@ Buffer Flags
the format. Any subsequent call to the
:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
but return an ``EPIPE`` error code.
+ * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
+
+ - ``V4L2_BUF_FLAG_CODECCONFIG``
+ - 0x00200000
+ - This flag may be set when the buffer only contains codec config
+ header, but does not contain any frame data. Usually the codec config
+ header is merged to the next idr frame, with the flag
+ ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that will
+ split the header and queue it separately.
* .. _`V4L2-BUF-FLAG-REQUEST-FD`:
- ``V4L2_BUF_FLAG_REQUEST_FD``
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 5311ac4fde35..8708ef257710 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const struct timeval *tv)
#define V4L2_BUF_FLAG_TSTAMP_SRC_SOE 0x00010000
/* mem2mem encoder/decoder */
#define V4L2_BUF_FLAG_LAST 0x00100000
+/* Buffer only contains codec header */
+#define V4L2_BUF_FLAG_CODECCONFIG 0x00200000
/* request_fd is valid */
#define V4L2_BUF_FLAG_REQUEST_FD 0x00800000
--
2.36.1
Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit :
> By setting the V4L2_BUF_FLAG_CODECCONFIG flag,
> user-space should be able to hint decoder
> the vb2 only contains codec config header,
> but does not contain any frame data.
> It's only used for parsing header, and can't be decoded.
This is copied from OMX specification. I think we we import this, we should at
least refer to the original.
>
> Current, it's usually used by android.
>
> Signed-off-by: Ming Qian <[email protected]>
> ---
> Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++
> include/uapi/linux/videodev2.h | 2 ++
> 2 files changed, 11 insertions(+)
>
> diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst
> index 4638ec64db00..acdc4556f4f4 100644
> --- a/Documentation/userspace-api/media/v4l/buffer.rst
> +++ b/Documentation/userspace-api/media/v4l/buffer.rst
> @@ -607,6 +607,15 @@ Buffer Flags
> the format. Any subsequent call to the
> :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
> but return an ``EPIPE`` error code.
> + * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
> +
> + - ``V4L2_BUF_FLAG_CODECCONFIG``
> + - 0x00200000
> + - This flag may be set when the buffer only contains codec config
> + header, but does not contain any frame data. Usually the codec config
> + header is merged to the next idr frame, with the flag
> + ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that will
> + split the header and queue it separately.
I think the documentation is clear. Now, if a driver uses this, will existing
userland (perhaps good to check GStreamer, FFMPEG and Chromium ?) will break ?
So we need existing driver to do this when flagged to, and just copy/append when
the userland didn't opt-in that feature ?
> * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
>
> - ``V4L2_BUF_FLAG_REQUEST_FD``
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index 5311ac4fde35..8708ef257710 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const struct timeval *tv)
> #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE 0x00010000
> /* mem2mem encoder/decoder */
> #define V4L2_BUF_FLAG_LAST 0x00100000
> +/* Buffer only contains codec header */
> +#define V4L2_BUF_FLAG_CODECCONFIG 0x00200000
> /* request_fd is valid */
> #define V4L2_BUF_FLAG_REQUEST_FD 0x00800000
>
> From: Nicolas Dufresne <[email protected]>
> Sent: 2022年7月4日 23:53
> To: Ming Qian <[email protected]>; [email protected];
> [email protected]
> Cc: [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]; dl-linux-imx
> <[email protected]>; [email protected];
> [email protected]; [email protected]
> Subject: [EXT] Re: [PATCH] media: videobuf2: add
> V4L2_BUF_FLAG_CODECCONFIG flag
>
> Caution: EXT Email
>
> Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit :
> > By setting the V4L2_BUF_FLAG_CODECCONFIG flag, user-space should be
> > able to hint decoder the vb2 only contains codec config header, but
> > does not contain any frame data.
> > It's only used for parsing header, and can't be decoded.
>
> This is copied from OMX specification. I think we we import this, we should at
> least refer to the original.
>
Hi Nicolas,
Do you mean OMX_BUFFERFLAG_CODECCONFIG?
I'm sorry that I didn't notice it before.
Currently we only encounter this requirement on Android, I'm not sure if it has a reference to omx.
And thank you very much for pointing out it.
Ming
> >
> > Current, it's usually used by android.
> >
> > Signed-off-by: Ming Qian <[email protected]>
> > ---
> > Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++
> > include/uapi/linux/videodev2.h | 2 ++
> > 2 files changed, 11 insertions(+)
> >
> > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst
> > b/Documentation/userspace-api/media/v4l/buffer.rst
> > index 4638ec64db00..acdc4556f4f4 100644
> > --- a/Documentation/userspace-api/media/v4l/buffer.rst
> > +++ b/Documentation/userspace-api/media/v4l/buffer.rst
> > @@ -607,6 +607,15 @@ Buffer Flags
> > the format. Any subsequent call to the
> > :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
> > but return an ``EPIPE`` error code.
> > + * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
> > +
> > + - ``V4L2_BUF_FLAG_CODECCONFIG``
> > + - 0x00200000
> > + - This flag may be set when the buffer only contains codec config
> > + header, but does not contain any frame data. Usually the codec config
> > + header is merged to the next idr frame, with the flag
> > + ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that will
> > + split the header and queue it separately.
>
> I think the documentation is clear. Now, if a driver uses this, will existing
> userland (perhaps good to check GStreamer, FFMPEG and Chromium ?) will
> break ?
> So we need existing driver to do this when flagged to, and just copy/append
> when the userland didn't opt-in that feature ?
>
> > * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
> >
> > - ``V4L2_BUF_FLAG_REQUEST_FD``
> > diff --git a/include/uapi/linux/videodev2.h
> > b/include/uapi/linux/videodev2.h index 5311ac4fde35..8708ef257710
> > 100644
> > --- a/include/uapi/linux/videodev2.h
> > +++ b/include/uapi/linux/videodev2.h
> > @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const
> struct timeval *tv)
> > #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE 0x00010000
> > /* mem2mem encoder/decoder */
> > #define V4L2_BUF_FLAG_LAST 0x00100000
> > +/* Buffer only contains codec header */
> > +#define V4L2_BUF_FLAG_CODECCONFIG 0x00200000
> > /* request_fd is valid */
> > #define V4L2_BUF_FLAG_REQUEST_FD 0x00800000
> >
>>From: Ming Qian
>>Sent: 2022年7月5日 9:52
>>To: Nicolas Dufresne <[email protected]>; [email protected];
>>[email protected]
>>Cc: [email protected]; [email protected]; [email protected];
>>[email protected]; [email protected]; dl-linux-imx
>><[email protected]>; [email protected];
>>[email protected]; [email protected]
>>Subject: RE: [EXT] Re: [PATCH] media: videobuf2: add
>>V4L2_BUF_FLAG_CODECCONFIG flag
>>
>>> From: Nicolas Dufresne <[email protected]>
>>> Sent: 2022年7月4日 23:53
>>> To: Ming Qian <[email protected]>; [email protected];
>>> [email protected]
>>> Cc: [email protected]; [email protected]; [email protected];
>>> [email protected]; [email protected]; dl-linux-imx
>>> <[email protected]>; [email protected];
>>> [email protected]; [email protected]
>>> Subject: [EXT] Re: [PATCH] media: videobuf2: add
>>> V4L2_BUF_FLAG_CODECCONFIG flag
>>>
>>> Caution: EXT Email
>>>
>>> Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit :
>>> > By setting the V4L2_BUF_FLAG_CODECCONFIG flag, user-space should be
>>> > able to hint decoder the vb2 only contains codec config header, but
>>> > does not contain any frame data.
>>> > It's only used for parsing header, and can't be decoded.
>>>
>>> This is copied from OMX specification. I think we we import this, we
>>> should at least refer to the original.
>>>
>>
>>Hi Nicolas,
>> Do you mean OMX_BUFFERFLAG_CODECCONFIG?
>> I'm sorry that I didn't notice it before.
>> Currently we only encounter this requirement on Android, I'm not sure if
>>it has a reference to omx.
>> And thank you very much for pointing out it.
>>
>
> Android media stack has been based on OMX for the last decade. They are slowly moving to CODEC2 which more or less is a similar abstraction with similar ideas. Let's research prior art, so we don't screw compatibility.
>
I got it, I'll try to study the android codec2,
and do you agree that we should add V4L2_BUF_FLAG_CODECCONFIG flag, just like OMX_BUFFERFLAG_CODECCONFIG?
Or is there any other solution that can handle this case?
>>Ming
>>
>>> >
>>> > Current, it's usually used by android.
>>> >
>>> > Signed-off-by: Ming Qian <[email protected]>
>>> > ---
>>> > Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++
>>> > include/uapi/linux/videodev2.h | 2 ++
>>> > 2 files changed, 11 insertions(+)
>>> >
>>> > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst
>>> > b/Documentation/userspace-api/media/v4l/buffer.rst
>>> > index 4638ec64db00..acdc4556f4f4 100644
>>> > --- a/Documentation/userspace-api/media/v4l/buffer.rst
>>> > +++ b/Documentation/userspace-api/media/v4l/buffer.rst
>>> > @@ -607,6 +607,15 @@ Buffer Flags
>>> > the format. Any subsequent call to the
>>> > :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
>>> > but return an ``EPIPE`` error code.
>>> > + * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
>>> > +
>>> > + - ``V4L2_BUF_FLAG_CODECCONFIG``
>>> > + - 0x00200000
>>> > + - This flag may be set when the buffer only contains codec config
>>> > + header, but does not contain any frame data. Usually the codec
>>config
>>> > + header is merged to the next idr frame, with the flag
>>> > + ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that
>>will
>>> > + split the header and queue it separately.
>>>
>>> I think the documentation is clear. Now, if a driver uses this, will
>>> existing userland (perhaps good to check GStreamer, FFMPEG and
>>> Chromium ?) will break ?
>>> So we need existing driver to do this when flagged to, and just
>>> copy/append when the userland didn't opt-in that feature ?
>>>
>>> > * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
>>> >
>>> > - ``V4L2_BUF_FLAG_REQUEST_FD`` diff --git
>>> > a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>>> > index 5311ac4fde35..8708ef257710
>>> > 100644
>>> > --- a/include/uapi/linux/videodev2.h
>>> > +++ b/include/uapi/linux/videodev2.h
>>> > @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const
>>> struct timeval *tv)
>>> > #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE 0x00010000
>>> > /* mem2mem encoder/decoder */
>>> > #define V4L2_BUF_FLAG_LAST 0x00100000
>>> > +/* Buffer only contains codec header */
>>> > +#define V4L2_BUF_FLAG_CODECCONFIG 0x00200000
>>> > /* request_fd is valid */
>>> > #define V4L2_BUF_FLAG_REQUEST_FD 0x00800000
>>> >
Hi Ming and Nicolas
On Mon, 4 Jul 2022 at 16:53, Nicolas Dufresne <[email protected]> wrote:
>
> Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit :
> > By setting the V4L2_BUF_FLAG_CODECCONFIG flag,
> > user-space should be able to hint decoder
> > the vb2 only contains codec config header,
> > but does not contain any frame data.
> > It's only used for parsing header, and can't be decoded.
>
> This is copied from OMX specification. I think we we import this, we should at
> least refer to the original.
>
> >
> > Current, it's usually used by android.
> >
> > Signed-off-by: Ming Qian <[email protected]>
> > ---
> > Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++
> > include/uapi/linux/videodev2.h | 2 ++
> > 2 files changed, 11 insertions(+)
> >
> > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst
> > index 4638ec64db00..acdc4556f4f4 100644
> > --- a/Documentation/userspace-api/media/v4l/buffer.rst
> > +++ b/Documentation/userspace-api/media/v4l/buffer.rst
> > @@ -607,6 +607,15 @@ Buffer Flags
> > the format. Any subsequent call to the
> > :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
> > but return an ``EPIPE`` error code.
> > + * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
> > +
> > + - ``V4L2_BUF_FLAG_CODECCONFIG``
> > + - 0x00200000
> > + - This flag may be set when the buffer only contains codec config
> > + header, but does not contain any frame data. Usually the codec config
> > + header is merged to the next idr frame, with the flag
> > + ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that will
> > + split the header and queue it separately.
>
> I think the documentation is clear. Now, if a driver uses this, will existing
> userland (perhaps good to check GStreamer, FFMPEG and Chromium ?) will break ?
> So we need existing driver to do this when flagged to, and just copy/append when
> the userland didn't opt-in that feature ?
The commit text says it is for userspace feeding data into a video
decoder, so it's a userspace choice instead of driver.
For encoders there is already V4L2_CID_MPEG_VIDEO_HEADER_MODE [1]
which allows for V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE or
V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME. FFmpeg selects
_SEPARATE by default [2], whilst the default is normally
_JOINED_WITH_1ST_FRAME.
It does raise the question as to whether all decoders will support
header byte only buffers, and does there need to be a capabilities
flag to denote that it is supported.
And should encoders in V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE mode set
it on the headers only buffers?
A number of undefined elements of how this should be implemented/used :-(
Dave
[1] https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/ext-ctrls-codec.html
[2] https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/v4l2_m2m_enc.c#L196
> > * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
> >
> > - ``V4L2_BUF_FLAG_REQUEST_FD``
> > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > index 5311ac4fde35..8708ef257710 100644
> > --- a/include/uapi/linux/videodev2.h
> > +++ b/include/uapi/linux/videodev2.h
> > @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const struct timeval *tv)
> > #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE 0x00010000
> > /* mem2mem encoder/decoder */
> > #define V4L2_BUF_FLAG_LAST 0x00100000
> > +/* Buffer only contains codec header */
> > +#define V4L2_BUF_FLAG_CODECCONFIG 0x00200000
> > /* request_fd is valid */
> > #define V4L2_BUF_FLAG_REQUEST_FD 0x00800000
> >
>
Le mardi 05 juillet 2022 à 11:34 +0000, Ming Qian a écrit :
> > > From: Ming Qian
> > > Sent: 2022年7月5日 9:52
> > > To: Nicolas Dufresne <[email protected]>; [email protected];
> > > [email protected]
> > > Cc: [email protected]; [email protected]; [email protected];
> > > [email protected]; [email protected]; dl-linux-imx
> > > <[email protected]>; [email protected];
> > > [email protected]; [email protected]
> > > Subject: RE: [EXT] Re: [PATCH] media: videobuf2: add
> > > V4L2_BUF_FLAG_CODECCONFIG flag
> > >
> > > > From: Nicolas Dufresne <[email protected]>
> > > > Sent: 2022年7月4日 23:53
> > > > To: Ming Qian <[email protected]>; [email protected];
> > > > [email protected]
> > > > Cc: [email protected]; [email protected]; [email protected];
> > > > [email protected]; [email protected]; dl-linux-imx
> > > > <[email protected]>; [email protected];
> > > > [email protected]; [email protected]
> > > > Subject: [EXT] Re: [PATCH] media: videobuf2: add
> > > > V4L2_BUF_FLAG_CODECCONFIG flag
> > > >
> > > > Caution: EXT Email
> > > >
> > > > Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit :
> > > > > By setting the V4L2_BUF_FLAG_CODECCONFIG flag, user-space should be
> > > > > able to hint decoder the vb2 only contains codec config header, but
> > > > > does not contain any frame data.
> > > > > It's only used for parsing header, and can't be decoded.
> > > >
> > > > This is copied from OMX specification. I think we we import this, we
> > > > should at least refer to the original.
> > > >
> > >
> > > Hi Nicolas,
> > > Do you mean OMX_BUFFERFLAG_CODECCONFIG?
> > > I'm sorry that I didn't notice it before.
> > > Currently we only encounter this requirement on Android, I'm not sure
> > > if
> > > it has a reference to omx.
> > > And thank you very much for pointing out it.
> > >
> >
> > Android media stack has been based on OMX for the last decade. They are
> > slowly moving to CODEC2 which more or less is a similar abstraction with
> > similar ideas. Let's research prior art, so we don't screw compatibility.
> >
>
> I got it, I'll try to study the android codec2,
> and do you agree that we should add V4L2_BUF_FLAG_CODECCONFIG flag, just like
> OMX_BUFFERFLAG_CODECCONFIG?
> Or is there any other solution that can handle this case?
We can probably discuss the name. CODECCONFIG is a bit strange, could be
CODEC_HEADER, HEADER_ONLY, CONFIG_ONLY, something along these lines. I'm just
wondering what is the best rule, since more specification is needed here.
Current userland expect full frames into each encoded buffer. If we start
splitting these up, we'll break non-android userland (and Android userland does
not seems to be very upstream thing, every vendor forks it).
I also think that what the CODECCONFIG contains and its format need to be
strictly documented for every CODEC that would allow it. In H.264 notably, the
headers could be packed in Annex B. or AVCc NAL headers. If we look at FFMPEG,
which uses codec_data name, they only requires this when the header are not
"inline", which means only for AVCc. Also, many codec_data is for other codecs
get wrapped into ISOMP4 or Matroska (webm) envelope.
On the other hand, I don't remember if the 1 frame per buffer is an actual rule
or simply what existing userland expect. Also, I'll be fair, there is no reason
this must come from the driver. Android OMX or CODEC2 COMPONENT is a userland
component, it could do bitstream scanning (using traditional boyer-more) to find
and split appart the config to satisfy its internal API. This can be done with
low overhead and zero-copy.
>
> > > Ming
> > >
> > > > >
> > > > > Current, it's usually used by android.
> > > > >
> > > > > Signed-off-by: Ming Qian <[email protected]>
> > > > > ---
> > > > > Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++
> > > > > include/uapi/linux/videodev2.h | 2 ++
> > > > > 2 files changed, 11 insertions(+)
> > > > >
> > > > > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst
> > > > > b/Documentation/userspace-api/media/v4l/buffer.rst
> > > > > index 4638ec64db00..acdc4556f4f4 100644
> > > > > --- a/Documentation/userspace-api/media/v4l/buffer.rst
> > > > > +++ b/Documentation/userspace-api/media/v4l/buffer.rst
> > > > > @@ -607,6 +607,15 @@ Buffer Flags
> > > > > the format. Any subsequent call to the
> > > > > :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
> > > > > but return an ``EPIPE`` error code.
> > > > > + * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
> > > > > +
> > > > > + - ``V4L2_BUF_FLAG_CODECCONFIG``
> > > > > + - 0x00200000
> > > > > + - This flag may be set when the buffer only contains codec
> > > > > config
> > > > > + header, but does not contain any frame data. Usually the codec
> > > config
> > > > > + header is merged to the next idr frame, with the flag
> > > > > + ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that
> > > will
> > > > > + split the header and queue it separately.
> > > >
> > > > I think the documentation is clear. Now, if a driver uses this, will
> > > > existing userland (perhaps good to check GStreamer, FFMPEG and
> > > > Chromium ?) will break ?
> > > > So we need existing driver to do this when flagged to, and just
> > > > copy/append when the userland didn't opt-in that feature ?
> > > >
> > > > > * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
> > > > >
> > > > > - ``V4L2_BUF_FLAG_REQUEST_FD`` diff --git
> > > > > a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > > > > index 5311ac4fde35..8708ef257710
> > > > > 100644
> > > > > --- a/include/uapi/linux/videodev2.h
> > > > > +++ b/include/uapi/linux/videodev2.h
> > > > > @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const
> > > > struct timeval *tv)
> > > > > #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE 0x00010000
> > > > > /* mem2mem encoder/decoder */
> > > > > #define V4L2_BUF_FLAG_LAST 0x00100000
> > > > > +/* Buffer only contains codec header */
> > > > > +#define V4L2_BUF_FLAG_CODECCONFIG 0x00200000
> > > > > /* request_fd is valid */
> > > > > #define V4L2_BUF_FLAG_REQUEST_FD 0x00800000
> > > > >
>
Le mardi 05 juillet 2022 à 13:34 +0100, Dave Stevenson a écrit :
> Hi Ming and Nicolas
>
> On Mon, 4 Jul 2022 at 16:53, Nicolas Dufresne <[email protected]> wrote:
> >
> > Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit :
> > > By setting the V4L2_BUF_FLAG_CODECCONFIG flag,
> > > user-space should be able to hint decoder
> > > the vb2 only contains codec config header,
> > > but does not contain any frame data.
> > > It's only used for parsing header, and can't be decoded.
> >
> > This is copied from OMX specification. I think we we import this, we should at
> > least refer to the original.
> >
> > >
> > > Current, it's usually used by android.
> > >
> > > Signed-off-by: Ming Qian <[email protected]>
> > > ---
> > > Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++
> > > include/uapi/linux/videodev2.h | 2 ++
> > > 2 files changed, 11 insertions(+)
> > >
> > > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst
> > > index 4638ec64db00..acdc4556f4f4 100644
> > > --- a/Documentation/userspace-api/media/v4l/buffer.rst
> > > +++ b/Documentation/userspace-api/media/v4l/buffer.rst
> > > @@ -607,6 +607,15 @@ Buffer Flags
> > > the format. Any subsequent call to the
> > > :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
> > > but return an ``EPIPE`` error code.
> > > + * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
> > > +
> > > + - ``V4L2_BUF_FLAG_CODECCONFIG``
> > > + - 0x00200000
> > > + - This flag may be set when the buffer only contains codec config
> > > + header, but does not contain any frame data. Usually the codec config
> > > + header is merged to the next idr frame, with the flag
> > > + ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that will
> > > + split the header and queue it separately.
> >
> > I think the documentation is clear. Now, if a driver uses this, will existing
> > userland (perhaps good to check GStreamer, FFMPEG and Chromium ?) will break ?
> > So we need existing driver to do this when flagged to, and just copy/append when
> > the userland didn't opt-in that feature ?
>
> The commit text says it is for userspace feeding data into a video
> decoder, so it's a userspace choice instead of driver.
I see, the spec needs to be more clear then.
>
> For encoders there is already V4L2_CID_MPEG_VIDEO_HEADER_MODE [1]
> which allows for V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE or
> V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME. FFmpeg selects
> _SEPARATE by default [2], whilst the default is normally
> _JOINED_WITH_1ST_FRAME.
I did miss the addition of this API, thanks for the reminder. The problem right
now is that things are being added with the needed cross-reference.
>
> It does raise the question as to whether all decoders will support
> header byte only buffers, and does there need to be a capabilities
> flag to denote that it is supported.
> And should encoders in V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE mode set
> it on the headers only buffers?
What about:
- Document better that in absence of V4L2_CID_MPEG_VIDEO_HEADER_MODE in a
driver, V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME is to be assumed
- Document that in V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE, driver should signal
the header buffer with the newly added buffer flag (I would suggest
V4L2_BUF_FLAG_HEADERS_ONLY)
- Document for each support CODECs if V4L2_CID_MPEG_VIDEO_HEADER_MODE can be
supported, and the expected headers and the packing of these
- Document that decoders that didn't implement V4L2_CID_MPEG_VIDEO_HEADER_MODE
should be assumed to only support
V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME decoding (even though it
allowed to support any alignment, even random).
- Cross-reference V4L2_CID_MPEG_VIDEO_HEADER_MODE into the new flag
documentation (whatever this new flag will be called).
>
> A number of undefined elements of how this should be implemented/used :-(
>
> Dave
>
> [1] https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/ext-ctrls-codec.html
> [2] https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/v4l2_m2m_enc.c#L196
>
> > > * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
> > >
> > > - ``V4L2_BUF_FLAG_REQUEST_FD``
> > > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > > index 5311ac4fde35..8708ef257710 100644
> > > --- a/include/uapi/linux/videodev2.h
> > > +++ b/include/uapi/linux/videodev2.h
> > > @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const struct timeval *tv)
> > > #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE 0x00010000
> > > /* mem2mem encoder/decoder */
> > > #define V4L2_BUF_FLAG_LAST 0x00100000
> > > +/* Buffer only contains codec header */
> > > +#define V4L2_BUF_FLAG_CODECCONFIG 0x00200000
> > > /* request_fd is valid */
> > > #define V4L2_BUF_FLAG_REQUEST_FD 0x00800000
> > >
> >
Hi Ming,
I love your patch! Perhaps something to improve:
[auto build test WARNING on media-tree/master]
[also build test WARNING on linus/master v5.19-rc5 next-20220705]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/intel-lab-lkp/linux/commits/Ming-Qian/media-videobuf2-add-V4L2_BUF_FLAG_CODECCONFIG-flag/20220628-103906
base: git://linuxtv.org/media_tree.git master
reproduce: make htmldocs
If you fix the issue, kindly add following tag where applicable
Reported-by: kernel test robot <[email protected]>
All warnings (new ones prefixed by >>):
>> Documentation/userspace-api/media/v4l/buffer.rst:614: WARNING: Bullet list ends without a blank line; unexpected unindent.
>> Documentation/userspace-api/media/v4l/buffer.rst:623: WARNING: Block quote ends without a blank line; unexpected unindent.
>> Documentation/userspace-api/media/v4l/buffer.rst:460: WARNING: Error parsing content block for the "flat-table" directive: exactly one bullet list expected.
vim +614 Documentation/userspace-api/media/v4l/buffer.rst
> 460
461 .. flat-table::
462 :header-rows: 0
463 :stub-columns: 0
464 :widths: 65 18 70
465
466 * .. _`V4L2-BUF-FLAG-MAPPED`:
467
468 - ``V4L2_BUF_FLAG_MAPPED``
469 - 0x00000001
470 - The buffer resides in device memory and has been mapped into the
471 application's address space, see :ref:`mmap` for details.
472 Drivers set or clear this flag when the
473 :ref:`VIDIOC_QUERYBUF`,
474 :ref:`VIDIOC_QBUF` or
475 :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl is called. Set by the
476 driver.
477 * .. _`V4L2-BUF-FLAG-QUEUED`:
478
479 - ``V4L2_BUF_FLAG_QUEUED``
480 - 0x00000002
481 - Internally drivers maintain two buffer queues, an incoming and
482 outgoing queue. When this flag is set, the buffer is currently on
483 the incoming queue. It automatically moves to the outgoing queue
484 after the buffer has been filled (capture devices) or displayed
485 (output devices). Drivers set or clear this flag when the
486 ``VIDIOC_QUERYBUF`` ioctl is called. After (successful) calling
487 the ``VIDIOC_QBUF``\ ioctl it is always set and after
488 ``VIDIOC_DQBUF`` always cleared.
489 * .. _`V4L2-BUF-FLAG-DONE`:
490
491 - ``V4L2_BUF_FLAG_DONE``
492 - 0x00000004
493 - When this flag is set, the buffer is currently on the outgoing
494 queue, ready to be dequeued from the driver. Drivers set or clear
495 this flag when the ``VIDIOC_QUERYBUF`` ioctl is called. After
496 calling the ``VIDIOC_QBUF`` or ``VIDIOC_DQBUF`` it is always
497 cleared. Of course a buffer cannot be on both queues at the same
498 time, the ``V4L2_BUF_FLAG_QUEUED`` and ``V4L2_BUF_FLAG_DONE`` flag
499 are mutually exclusive. They can be both cleared however, then the
500 buffer is in "dequeued" state, in the application domain so to
501 say.
502 * .. _`V4L2-BUF-FLAG-ERROR`:
503
504 - ``V4L2_BUF_FLAG_ERROR``
505 - 0x00000040
506 - When this flag is set, the buffer has been dequeued successfully,
507 although the data might have been corrupted. This is recoverable,
508 streaming may continue as normal and the buffer may be reused
509 normally. Drivers set this flag when the ``VIDIOC_DQBUF`` ioctl is
510 called.
511 * .. _`V4L2-BUF-FLAG-IN-REQUEST`:
512
513 - ``V4L2_BUF_FLAG_IN_REQUEST``
514 - 0x00000080
515 - This buffer is part of a request that hasn't been queued yet.
516 * .. _`V4L2-BUF-FLAG-KEYFRAME`:
517
518 - ``V4L2_BUF_FLAG_KEYFRAME``
519 - 0x00000008
520 - Drivers set or clear this flag when calling the ``VIDIOC_DQBUF``
521 ioctl. It may be set by video capture devices when the buffer
522 contains a compressed image which is a key frame (or field), i. e.
523 can be decompressed on its own. Also known as an I-frame.
524 Applications can set this bit when ``type`` refers to an output
525 stream.
526 * .. _`V4L2-BUF-FLAG-PFRAME`:
527
528 - ``V4L2_BUF_FLAG_PFRAME``
529 - 0x00000010
530 - Similar to ``V4L2_BUF_FLAG_KEYFRAME`` this flags predicted frames
531 or fields which contain only differences to a previous key frame.
532 Applications can set this bit when ``type`` refers to an output
533 stream.
534 * .. _`V4L2-BUF-FLAG-BFRAME`:
535
536 - ``V4L2_BUF_FLAG_BFRAME``
537 - 0x00000020
538 - Similar to ``V4L2_BUF_FLAG_KEYFRAME`` this flags a bi-directional
539 predicted frame or field which contains only the differences
540 between the current frame and both the preceding and following key
541 frames to specify its content. Applications can set this bit when
542 ``type`` refers to an output stream.
543 * .. _`V4L2-BUF-FLAG-TIMECODE`:
544
545 - ``V4L2_BUF_FLAG_TIMECODE``
546 - 0x00000100
547 - The ``timecode`` field is valid. Drivers set or clear this flag
548 when the ``VIDIOC_DQBUF`` ioctl is called. Applications can set
549 this bit and the corresponding ``timecode`` structure when
550 ``type`` refers to an output stream.
551 * .. _`V4L2-BUF-FLAG-PREPARED`:
552
553 - ``V4L2_BUF_FLAG_PREPARED``
554 - 0x00000400
555 - The buffer has been prepared for I/O and can be queued by the
556 application. Drivers set or clear this flag when the
557 :ref:`VIDIOC_QUERYBUF`,
558 :ref:`VIDIOC_PREPARE_BUF <VIDIOC_QBUF>`,
559 :ref:`VIDIOC_QBUF` or
560 :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl is called.
561 * .. _`V4L2-BUF-FLAG-NO-CACHE-INVALIDATE`:
562
563 - ``V4L2_BUF_FLAG_NO_CACHE_INVALIDATE``
564 - 0x00000800
565 - Caches do not have to be invalidated for this buffer. Typically
566 applications shall use this flag if the data captured in the
567 buffer is not going to be touched by the CPU, instead the buffer
568 will, probably, be passed on to a DMA-capable hardware unit for
569 further processing or output. This flag is ignored unless the
570 queue is used for :ref:`memory mapping <mmap>` streaming I/O and
571 reports :ref:`V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS
572 <V4L2-BUF-CAP-SUPPORTS-MMAP-CACHE-HINTS>` capability.
573 * .. _`V4L2-BUF-FLAG-NO-CACHE-CLEAN`:
574
575 - ``V4L2_BUF_FLAG_NO_CACHE_CLEAN``
576 - 0x00001000
577 - Caches do not have to be cleaned for this buffer. Typically
578 applications shall use this flag for output buffers if the data in
579 this buffer has not been created by the CPU but by some
580 DMA-capable unit, in which case caches have not been used. This flag
581 is ignored unless the queue is used for :ref:`memory mapping <mmap>`
582 streaming I/O and reports :ref:`V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS
583 <V4L2-BUF-CAP-SUPPORTS-MMAP-CACHE-HINTS>` capability.
584 * .. _`V4L2-BUF-FLAG-M2M-HOLD-CAPTURE-BUF`:
585
586 - ``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF``
587 - 0x00000200
588 - Only valid if struct :c:type:`v4l2_requestbuffers` flag ``V4L2_BUF_CAP_SUPPORTS_M2M_HOLD_CAPTURE_BUF`` is
589 set. It is typically used with stateless decoders where multiple
590 output buffers each decode to a slice of the decoded frame.
591 Applications can set this flag when queueing the output buffer
592 to prevent the driver from dequeueing the capture buffer after
593 the output buffer has been decoded (i.e. the capture buffer is
594 'held'). If the timestamp of this output buffer differs from that
595 of the previous output buffer, then that indicates the start of a
596 new frame and the previously held capture buffer is dequeued.
597 * .. _`V4L2-BUF-FLAG-LAST`:
598
599 - ``V4L2_BUF_FLAG_LAST``
600 - 0x00100000
601 - Last buffer produced by the hardware. mem2mem codec drivers set
602 this flag on the capture queue for the last buffer when the
603 :ref:`VIDIOC_QUERYBUF` or
604 :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl is called. Due to
605 hardware limitations, the last buffer may be empty. In this case
606 the driver will set the ``bytesused`` field to 0, regardless of
607 the format. Any subsequent call to the
608 :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
609 but return an ``EPIPE`` error code.
610 * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
611
612 - ``V4L2_BUF_FLAG_CODECCONFIG``
613 - 0x00200000
> 614 - This flag may be set when the buffer only contains codec config
615 header, but does not contain any frame data. Usually the codec config
616 header is merged to the next idr frame, with the flag
617 ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that will
618 split the header and queue it separately.
619 * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
620
621 - ``V4L2_BUF_FLAG_REQUEST_FD``
622 - 0x00800000
> 623 - The ``request_fd`` field contains a valid file descriptor.
624 * .. _`V4L2-BUF-FLAG-TIMESTAMP-MASK`:
625
626 - ``V4L2_BUF_FLAG_TIMESTAMP_MASK``
627 - 0x0000e000
628 - Mask for timestamp types below. To test the timestamp type, mask
629 out bits not belonging to timestamp type by performing a logical
630 and operation with buffer flags and timestamp mask.
631 * .. _`V4L2-BUF-FLAG-TIMESTAMP-UNKNOWN`:
632
633 - ``V4L2_BUF_FLAG_TIMESTAMP_UNKNOWN``
634 - 0x00000000
635 - Unknown timestamp type. This type is used by drivers before Linux
636 3.9 and may be either monotonic (see below) or realtime (wall
637 clock). Monotonic clock has been favoured in embedded systems
638 whereas most of the drivers use the realtime clock. Either kinds
639 of timestamps are available in user space via
640 :c:func:`clock_gettime` using clock IDs ``CLOCK_MONOTONIC``
641 and ``CLOCK_REALTIME``, respectively.
642 * .. _`V4L2-BUF-FLAG-TIMESTAMP-MONOTONIC`:
643
644 - ``V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC``
645 - 0x00002000
646 - The buffer timestamp has been taken from the ``CLOCK_MONOTONIC``
647 clock. To access the same clock outside V4L2, use
648 :c:func:`clock_gettime`.
649 * .. _`V4L2-BUF-FLAG-TIMESTAMP-COPY`:
650
651 - ``V4L2_BUF_FLAG_TIMESTAMP_COPY``
652 - 0x00004000
653 - The CAPTURE buffer timestamp has been taken from the corresponding
654 OUTPUT buffer. This flag applies only to mem2mem devices.
655 * .. _`V4L2-BUF-FLAG-TSTAMP-SRC-MASK`:
656
657 - ``V4L2_BUF_FLAG_TSTAMP_SRC_MASK``
658 - 0x00070000
659 - Mask for timestamp sources below. The timestamp source defines the
660 point of time the timestamp is taken in relation to the frame.
661 Logical 'and' operation between the ``flags`` field and
662 ``V4L2_BUF_FLAG_TSTAMP_SRC_MASK`` produces the value of the
663 timestamp source. Applications must set the timestamp source when
664 ``type`` refers to an output stream and
665 ``V4L2_BUF_FLAG_TIMESTAMP_COPY`` is set.
666 * .. _`V4L2-BUF-FLAG-TSTAMP-SRC-EOF`:
667
668 - ``V4L2_BUF_FLAG_TSTAMP_SRC_EOF``
669 - 0x00000000
670 - End Of Frame. The buffer timestamp has been taken when the last
671 pixel of the frame has been received or the last pixel of the
672 frame has been transmitted. In practice, software generated
673 timestamps will typically be read from the clock a small amount of
674 time after the last pixel has been received or transmitten,
675 depending on the system and other activity in it.
676 * .. _`V4L2-BUF-FLAG-TSTAMP-SRC-SOE`:
677
678 - ``V4L2_BUF_FLAG_TSTAMP_SRC_SOE``
679 - 0x00010000
680 - Start Of Exposure. The buffer timestamp has been taken when the
681 exposure of the frame has begun. This is only valid for the
682 ``V4L2_BUF_TYPE_VIDEO_CAPTURE`` buffer type.
683
--
0-DAY CI Kernel Test Service
https://01.org/lkp
>From: Nicolas Dufresne <[email protected]>
>Sent: 2022年7月5日 21:13
>To: Ming Qian <[email protected]>; [email protected];
>[email protected]
>Cc: [email protected]; [email protected]; [email protected];
>[email protected]; [email protected]; dl-linux-imx
><[email protected]>; [email protected];
>[email protected]; [email protected]
>Subject: Re: [EXT] Re: [PATCH] media: videobuf2: add
>V4L2_BUF_FLAG_CODECCONFIG flag
>
>Caution: EXT Email
>
>Le mardi 05 juillet 2022 à 11:34 +0000, Ming Qian a écrit :
>> > > From: Ming Qian
>> > > Sent: 2022年7月5日 9:52
>> > > To: Nicolas Dufresne <[email protected]>; [email protected];
>> > > [email protected]
>> > > Cc: [email protected]; [email protected];
>> > > [email protected]; [email protected]; [email protected];
>> > > dl-linux-imx <[email protected]>; [email protected];
>> > > [email protected]; [email protected]
>> > > Subject: RE: [EXT] Re: [PATCH] media: videobuf2: add
>> > > V4L2_BUF_FLAG_CODECCONFIG flag
>> > >
>> > > > From: Nicolas Dufresne <[email protected]>
>> > > > Sent: 2022年7月4日 23:53
>> > > > To: Ming Qian <[email protected]>; [email protected];
>> > > > [email protected]
>> > > > Cc: [email protected]; [email protected];
>> > > > [email protected]; [email protected];
>> > > > [email protected]; dl-linux-imx <[email protected]>;
>> > > > [email protected]; [email protected];
>> > > > [email protected]
>> > > > Subject: [EXT] Re: [PATCH] media: videobuf2: add
>> > > > V4L2_BUF_FLAG_CODECCONFIG flag
>> > > >
>> > > > Caution: EXT Email
>> > > >
>> > > > Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit :
>> > > > > By setting the V4L2_BUF_FLAG_CODECCONFIG flag, user-space
>> > > > > should be able to hint decoder the vb2 only contains codec
>> > > > > config header, but does not contain any frame data.
>> > > > > It's only used for parsing header, and can't be decoded.
>> > > >
>> > > > This is copied from OMX specification. I think we we import
>> > > > this, we should at least refer to the original.
>> > > >
>> > >
>> > > Hi Nicolas,
>> > > Do you mean OMX_BUFFERFLAG_CODECCONFIG?
>> > > I'm sorry that I didn't notice it before.
>> > > Currently we only encounter this requirement on Android, I'm
>> > > not sure if it has a reference to omx.
>> > > And thank you very much for pointing out it.
>> > >
>> >
>> > Android media stack has been based on OMX for the last decade. They
>> > are slowly moving to CODEC2 which more or less is a similar
>> > abstraction with similar ideas. Let's research prior art, so we don't screw
>compatibility.
>> >
>>
>> I got it, I'll try to study the android codec2, and do you agree that
>> we should add V4L2_BUF_FLAG_CODECCONFIG flag, just like
>> OMX_BUFFERFLAG_CODECCONFIG?
>> Or is there any other solution that can handle this case?
>
>We can probably discuss the name. CODECCONFIG is a bit strange, could be
>CODEC_HEADER, HEADER_ONLY, CONFIG_ONLY, something along these lines.
>I'm just wondering what is the best rule, since more specification is needed
>here.
>Current userland expect full frames into each encoded buffer. If we start
>splitting these up, we'll break non-android userland (and Android userland does
>not seems to be very upstream thing, every vendor forks it).
>
>I also think that what the CODECCONFIG contains and its format need to be
>strictly documented for every CODEC that would allow it. In H.264 notably, the
>headers could be packed in Annex B. or AVCc NAL headers. If we look at
>FFMPEG, which uses codec_data name, they only requires this when the header
>are not "inline", which means only for AVCc. Also, many codec_data is for other
>codecs get wrapped into ISOMP4 or Matroska (webm) envelope.
>
>On the other hand, I don't remember if the 1 frame per buffer is an actual rule
>or simply what existing userland expect. Also, I'll be fair, there is no reason this
>must come from the driver. Android OMX or CODEC2 COMPONENT is a userland
>component, it could do bitstream scanning (using traditional boyer-more) to
>find and split appart the config to satisfy its internal API. This can be done with
>low overhead and zero-copy.
>
OK, I'll try to change the name, maybe CODEC_HEADER is better.
And I think we add the flag just to support more scenarios, it won't break the existing userland flow.
It's not easy to change the android codec2, and we indeed want to support it.
>>
>> > > Ming
>> > >
>> > > > >
>> > > > > Current, it's usually used by android.
>> > > > >
>> > > > > Signed-off-by: Ming Qian <[email protected]>
>> > > > > ---
>> > > > > Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++
>> > > > > include/uapi/linux/videodev2.h | 2 ++
>> > > > > 2 files changed, 11 insertions(+)
>> > > > >
>> > > > > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst
>> > > > > b/Documentation/userspace-api/media/v4l/buffer.rst
>> > > > > index 4638ec64db00..acdc4556f4f4 100644
>> > > > > --- a/Documentation/userspace-api/media/v4l/buffer.rst
>> > > > > +++ b/Documentation/userspace-api/media/v4l/buffer.rst
>> > > > > @@ -607,6 +607,15 @@ Buffer Flags
>> > > > > the format. Any subsequent call to the
>> > > > > :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block
>anymore,
>> > > > > but return an ``EPIPE`` error code.
>> > > > > + * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
>> > > > > +
>> > > > > + - ``V4L2_BUF_FLAG_CODECCONFIG``
>> > > > > + - 0x00200000
>> > > > > + - This flag may be set when the buffer only contains
>> > > > > + codec
>> > > > > config
>> > > > > + header, but does not contain any frame data. Usually the
>> > > > > + codec
>> > > config
>> > > > > + header is merged to the next idr frame, with the flag
>> > > > > + ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some
>> > > > > + scenes that
>> > > will
>> > > > > + split the header and queue it separately.
>> > > >
>> > > > I think the documentation is clear. Now, if a driver uses this,
>> > > > will existing userland (perhaps good to check GStreamer, FFMPEG
>> > > > and Chromium ?) will break ?
>> > > > So we need existing driver to do this when flagged to, and just
>> > > > copy/append when the userland didn't opt-in that feature ?
>> > > >
>> > > > > * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
>> > > > >
>> > > > > - ``V4L2_BUF_FLAG_REQUEST_FD`` diff --git
>> > > > > a/include/uapi/linux/videodev2.h
>> > > > > b/include/uapi/linux/videodev2.h index
>> > > > > 5311ac4fde35..8708ef257710
>> > > > > 100644
>> > > > > --- a/include/uapi/linux/videodev2.h
>> > > > > +++ b/include/uapi/linux/videodev2.h
>> > > > > @@ -1131,6 +1131,8 @@ static inline __u64
>> > > > > v4l2_timeval_to_ns(const
>> > > > struct timeval *tv)
>> > > > > #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE 0x00010000
>> > > > > /* mem2mem encoder/decoder */
>> > > > > #define V4L2_BUF_FLAG_LAST 0x00100000
>> > > > > +/* Buffer only contains codec header */
>> > > > > +#define V4L2_BUF_FLAG_CODECCONFIG 0x00200000
>> > > > > /* request_fd is valid */
>> > > > > #define V4L2_BUF_FLAG_REQUEST_FD 0x00800000
>> > > > >
>>