Hi Steve, Philipp,
I see in the kernel sources you created a ipu-ic.c file which helps
working with the Color Space Converter on i.MX6.
We would like to use it on our board with GStreamer, as the conversion
done by the CPU is not very efficient :).
What is the easiest/best approach to this ?
Should we create a video device using a new driver, which would be
instanciated as a /dev/videoX and see as a transform element in
GStreamer ?
The idea is to have a separate element, ideally without memory copy or
anything like that.
I have a similar question, regarding the scaler part. It seems to be
limited by nature to 1024x1024 frames. In FSL BSP there is a split to
have greater frames. Is there something doable right now with the
kernel, or should we write something else ?
Thanks !
JM
On 12/15/2014 09:03 AM, Jean-Michel Hautbois wrote:
> Hi Steve, Philipp,
>
> I see in the kernel sources you created a ipu-ic.c file which helps
> working with the Color Space Converter on i.MX6.
> We would like to use it on our board with GStreamer, as the conversion
> done by the CPU is not very efficient :).
> What is the easiest/best approach to this ?
> Should we create a video device using a new driver, which would be
> instanciated as a /dev/videoX and see as a transform element in
> GStreamer ?
> The idea is to have a separate element, ideally without memory copy or
> anything like that.
Hi JM,
I've written a mem2mem driver for this. It does tiling to support
> 1024x1024 scaled output frames. I've attached it, feel free to use
it, it was pulled from a 3.14 kernel. I haven't submitted this driver to
community yet, mostly because I want to improve it, most importantly
move the tiling support into ipu-ic, so that tiling could eventually be used by
any media-device enabled pipeline elements (instead of only mem2mem).
Also Philipp probably also has a mem2mem device as well.
You would need to write a gstreamer plugin to make use of the
mem2mem device for h/w CSC and scaling.
Steve
Le 12/15/2014 01:23 PM, Steve Longerbeam a écrit :
> You would need to write a gstreamer plugin to make use of the
> mem2mem device for h/w CSC and scaling.
There is an element already (v4l2transform), though it currently only
handle color conversion. I'd like to see scaling in that too instead of
having a separate element that only do scaling like:
https://bugzilla.gnome.org/show_bug.cgi?id=726104
I think Michael and Philipp idea was that scaling could be done directly
out of the decoder, something supported by CODA driver. There is some
work require to get that in GStreamer, see:
https://bugzilla.gnome.org/show_bug.cgi?id=733827 (pixel format nego)
https://bugzilla.gnome.org/show_bug.cgi?id=733828 (display size nego)
cheers,
Nicolas
2014-12-15 19:52 GMT+01:00 Nicolas Dufresne <[email protected]>:
>
>
> Le 12/15/2014 01:23 PM, Steve Longerbeam a écrit :
>> You would need to write a gstreamer plugin to make use of the
>> mem2mem device for h/w CSC and scaling.
Thank you Steve, this is of a great help !
> There is an element already (v4l2transform), though it currently only
> handle color conversion. I'd like to see scaling in that too instead of
> having a separate element that only do scaling like:
>
> https://bugzilla.gnome.org/show_bug.cgi?id=726104
Nice idea. You are probably right Nicolas, it would make sense to be
able to scale and convert color in the same element.
> I think Michael and Philipp idea was that scaling could be done directly
> out of the decoder, something supported by CODA driver. There is some
> work require to get that in GStreamer, see:
>
> https://bugzilla.gnome.org/show_bug.cgi?id=733827 (pixel format nego)
> https://bugzilla.gnome.org/show_bug.cgi?id=733828 (display size nego)
There is a need for a separate element too. So it could be handled in
front of the encoder if needed, on decoder output if needed, but
should be a single element too, instanciable multiple times if
multiple inputs/outputs are needed.
JM
Am Montag, den 15.12.2014, 22:27 +0100 schrieb Jean-Michel Hautbois:
> 2014-12-15 19:52 GMT+01:00 Nicolas Dufresne <[email protected]>:
> > Le 12/15/2014 01:23 PM, Steve Longerbeam a écrit :
> >> You would need to write a gstreamer plugin to make use of the
> >> mem2mem device for h/w CSC and scaling.
>
> Thank you Steve, this is of a great help !
>
> > There is an element already (v4l2transform), though it currently only
> > handle color conversion. I'd like to see scaling in that too instead of
> > having a separate element that only do scaling like:
> >
> > https://bugzilla.gnome.org/show_bug.cgi?id=726104
>
> Nice idea. You are probably right Nicolas, it would make sense to be
> able to scale and convert color in the same element.
>
> > I think Michael and Philipp idea was that scaling could be done directly
> > out of the decoder, something supported by CODA driver. There is some
> > work require to get that in GStreamer, see:
> >
> > https://bugzilla.gnome.org/show_bug.cgi?id=733827 (pixel format nego)
> > https://bugzilla.gnome.org/show_bug.cgi?id=733828 (display size nego)
>
> There is a need for a separate element too. So it could be handled in
> front of the encoder if needed, on decoder output if needed, but
> should be a single element too, instanciable multiple times if
> multiple inputs/outputs are needed.
Scaling in the decoder element is desirable, but unrelated. I wanted
that for the CODA JPEG decoder, which can produce decimated output right
during the decoding process.
For the IPU IC mem2mem scaler/CSC, v4l2transform with scaling capability
would be the perfect fit.
regards
Philipp
Le 2014-12-16 08:30, Philipp Zabel a écrit :
> For the IPU IC mem2mem scaler/CSC, v4l2transform with scaling capability
> would be the perfect fit
Thanks you for clarifying. Jean-Michel, are you going to work on this ?
If so, feel free to contact me directly for pointers. You'll find me as
stormer on IRC freenode on both #v4l2 and #gstreamer.
What would need to be done:
a) Import scaler negotiation code from videoscale element, and merge it
into v4l2transform negotiation. It is likely that wtay (Wim Tayman) have
some code already for you, has on the SW side he is currently working on
merging videoscale and videoconvert together as it's faster this way.
b) Complete the handling of G_CROP (if required add G_SELECTION). Most
HW scaler will allocate aligned buffers, but can reach a 1 or 2 pixels
precision through CROP or SELECTION.
cheers,
Nicolas
Hi,
2014-12-16 15:15 GMT+01:00 Nicolas Dufresne <[email protected]>:
>
> Le 2014-12-16 08:30, Philipp Zabel a écrit :
>>
>> For the IPU IC mem2mem scaler/CSC, v4l2transform with scaling capability
>> would be the perfect fit
>
> Thanks you for clarifying. Jean-Michel, are you going to work on this ? If
> so, feel free to contact me directly for pointers. You'll find me as stormer
> on IRC freenode on both #v4l2 and #gstreamer.
Well, yes, we are going to work on it, but first we need to have
Steve's module probed. It seems to be using DT, but figuring out how
to use it properly.
> What would need to be done:
> a) Import scaler negotiation code from videoscale element, and merge it into
> v4l2transform negotiation. It is likely that wtay (Wim Tayman) have some
> code already for you, has on the SW side he is currently working on merging
> videoscale and videoconvert together as it's faster this way.
OK, so better to go on #gstreamer and talk together with Wim of this.
> b) Complete the handling of G_CROP (if required add G_SELECTION). Most HW
> scaler will allocate aligned buffers, but can reach a 1 or 2 pixels
> precision through CROP or SELECTION.
Have you already started something about that ?
JM
Le 2014-12-16 09:27, Jean-Michel Hautbois a écrit :
> Have you already started something about that ?
Yes, currently I do cropping right if there is no scaling. Doing this
with S_CROP is really ackward, but very few m2m driver has been ported
to G_SELECTION yet. The v4l2transform was tested to run with Exynos 4
FIMC driver. There was still issue with such driver doing middle
rounding for alignment. This has been reported on this list. The
proposed solution is to add flags to our internal alignment method, and
update these driver to pick a direction.
Nicolas
2014-12-16 15:50 GMT+01:00 Nicolas Dufresne <[email protected]>:
>
> Le 2014-12-16 09:27, Jean-Michel Hautbois a écrit :
>>
>> Have you already started something about that ?
>
> Yes, currently I do cropping right if there is no scaling. Doing this with
> S_CROP is really ackward, but very few m2m driver has been ported to
> G_SELECTION yet. The v4l2transform was tested to run with Exynos 4 FIMC
> driver. There was still issue with such driver doing middle rounding for
> alignment. This has been reported on this list. The proposed solution is to
> add flags to our internal alignment method, and update these driver to pick
> a direction.
Right now, I can get the driver probed (twice, as I have two IPUs) and
gstreamer creates two elements :
$> rm .cache/gstreamer-1.0/registry.arm.bin
$>
gst-inspect-1.0 |grep v4l2
video4linux2: v4l2video3videodec: V4L2 Video Decoder
video4linux2: v4l2video2h264enc: V4L2 H.264 Encoder
video4linux2: v4l2video1convert: V4L2 Video Converter
video4linux2: v4l2video0convert: V4L2 Video Converter
video4linux2: v4l2deviceprovider (GstDeviceProviderFactory)
video4linux2: v4l2radio: Radio (video4linux2) Tuner
video4linux2: v4l2sink: Video (video4linux2) Sink
video4linux2: v4l2src: Video (video4linux2) Source
$> gst-inspect-1.0 v4l2video0convert
Factory Details:
Rank none (0)
Long-name V4L2 Video Converter
Klass Filter/Converter/Video
Description Transform streams via V4L2 API
Author Nicolas Dufresne <[email protected]>
Plugin Details:
Name video4linux2
Description elements for Video 4 Linux
Filename /usr/lib/gstreamer-1.0/libgstvideo4linux2.so
Version 1.5.0.1
License LGPL
Source module gst-plugins-good
Source release date 2014-12-15 11:41 (UTC)
Binary package GStreamer Good Plug-ins git
Origin URL Unknown package origin
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstBaseTransform
+----GstV4l2Transform
+----v4l2video0convert
Pad Templates:
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-raw
format: { RGB16, RGB, BGR, xRGB, ARGB, BGRx, BGRA,
YUY2, UYVY, I420, YV12, NV12, NV21, Y42B }
width: [ 1, 32768 ]
height: [ 1, 32768 ]
framerate: [ 0/1, 100/1 ]
SRC template: 'src'
Availability: Always
Capabilities:
video/x-raw
format: { RGB16, RGB, BGR, xRGB, ARGB, BGRx, BGRA,
YUY2, UYVY, I420, YV12, NV12, NV21, Y42B }
width: [ 1, 32768 ]
height: [ 1, 32768 ]
framerate: [ 0/1, 100/1 ]
Element Flags:
no flags set
Element Implementation:
Has change_state() function: gst_v4l2_transform_change_state
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
SRC: 'src'
Pad Template: 'src'
Element Properties:
name : The name of the object
flags: readable, writable
String. Default: "v4l2video0convert0"
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
qos : Handle Quality-of-Service events
flags: readable, writable
Boolean. Default: true
device : Device location
flags: readable
String. Default: "/dev/video0"
device-name : Name of the device
flags: readable
String. Default: "mx6-m2m"
device-fd : File descriptor of the device
flags: readable
Integer. Range: -1 - 2147483647 Default: -1
output-io-mode : Output side I/O mode (matches sink pad)
flags: readable, writable
Enum "GstV4l2IOMode" Default: 0, "auto"
(0): auto - GST_V4L2_IO_AUTO
(1): rw - GST_V4L2_IO_RW
(2): mmap - GST_V4L2_IO_MMAP
(3): userptr - GST_V4L2_IO_USERPTR
(4): dmabuf - GST_V4L2_IO_DMABUF
(5): dmabuf-import - GST_V4L2_IO_DMABUF_IMPORT
capture-io-mode : Capture I/O mode (matches src pad)
flags: readable, writable
Enum "GstV4l2IOMode" Default: 0, "auto"
(0): auto - GST_V4L2_IO_AUTO
(1): rw - GST_V4L2_IO_RW
(2): mmap - GST_V4L2_IO_MMAP
(3): userptr - GST_V4L2_IO_USERPTR
(4): dmabuf - GST_V4L2_IO_DMABUF
(5): dmabuf-import - GST_V4L2_IO_DMABUF_IMPORT
extra-controls : Extra v4l2 controls (CIDs) for the device
flags: readable, writable
Boxed pointer of type "GstStructure"
When I try to use it, the driver fails to work, because S_FMT needs to
be called first.
$> gst-launch-1.0 -vve videotestsrc !
'video/x-raw,format=YUY2,width=320,height=240' ! v4l2video0convert !
'video/x-raw,format=NV12,width=320,height=240' ! fakesink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
ERROR: from element
/GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0: Internal data
flow error.
$> dmesg
[10005.548047] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.555784] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.563482] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.571225] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.578925] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.586781] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.594583] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.602292] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.610125] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.617864] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.625559] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.633970] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.641730] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.649488] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.657217] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.664927] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.672629] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.680336] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.688070] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.695770] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.703457] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.711184] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.718880] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.727170] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.734885] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.742592] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.750301] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.758205] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.765910] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.773603] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.781334] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.789028] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.796743] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.804444] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.812141] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
[10005.819862] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
determine segmentation
So, basically it should be working because using v4l2-ctl I can do :
$> v4l2-ctl -d0 --set-fmt-video=width=320,height=240,pixelformat=5
--set-fmt-video-out=width=1920,height=1080,pixelformat=NV12
[10127.455820] mx6-m2m ipum2m@ipu1: Capture format: 320x240 (1 320x240
segments), YUYV
[10127.463584] mx6-m2m ipum2m@ipu1: Output format: 1920x1080 (1
1920x1080 segments), NV12
I may need some help in order to modify the gstreamer plugin now ;-).
JM
Le 2014-12-17 08:12, Jean-Michel Hautbois a écrit :
> [10005.819862] mx6-m2m ipum2m@ipu1: call capture S_FMT first to
> determine segmentation
Please, find the part of the spec that would inforce that, if not
inforced, do the spec work first.
Nicolas