Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20221129074530.640251-1-wenst@chromium.org> <45143854.fMDQidcC6G@kista>
 <5d79ed06-15c0-3564-97b6-5fd4433acabf@xs4all.nl>
In-Reply-To: <5d79ed06-15c0-3564-97b6-5fd4433acabf@xs4all.nl>
From:   Chen-Yu Tsai <wenst@chromium.org>
Date:   Tue, 6 Dec 2022 20:23:21 +0800
Message-ID: <CAGXv+5GH==gm533P_sNiFTyaFwQXp-QLXRLWyABdsn+0p_83UQ@mail.gmail.com>
Subject: Re: [PATCH] media: cedrus: Convert to MPLANE uAPI
To:     Hans Verkuil <hverkuil-cisco@xs4all.nl>
Cc:     =?UTF-8?Q?Jernej_=C5=A0krabec?= <jernej.skrabec@gmail.com>,
        Mauro Carvalho Chehab <mchehab@kernel.org>,
        Samuel Holland <samuel@sholland.org>,
        linux-media@vger.kernel.org, linux-sunxi@lists.linux.dev,
        linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org,
        Nicolas Dufresne <nicolas.dufresne@collabora.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Tue, Dec 6, 2022 at 4:35 PM Hans Verkuil <hverkuil-cisco@xs4all.nl> wrot=
e:
>
> On 05/12/2022 22:01, Jernej =C5=A0krabec wrote:
> > Hi Chen-Yu!
> >
> > Dne torek, 29. november 2022 ob 08:45:30 CET je Chen-Yu Tsai napisal(a)=
:
> >> The majority of the V4L2 stateless video decoder drivers use the MPLAN=
E
> >> interface.
> >>
> >> On the userspace side, Gstreamer supports non-MPLANE and MPLANE
> >> interfaces. Chromium only supports the MPLANE interface, and is not ye=
t
> >> usable with standard desktop Linux. FFmpeg support for either has not
> >> landed.
> >
> > I don't like fixing userspace issues in kernel, if kernel side works fi=
ne.
> > Implementing missing non-MPLANE support in Chromium will also allow it =
to work
> > with older kernels.
> >
> > Hans, what's linux-media politics about such changes?
>
> Not keen on this. Does the cedrus HW even have support for multiple plane=
s?
> I suspect not, in which case the driver shouldn't suggest that it can do =
that.
>
> Now, if the hardware *can* support this, then there is an argument to be =
made
> for the cedrus driver to move to the multiplanar API before moving it out
> of staging to allow such future enhancements.

AFAIK it can, but has some limitations on how far apart the buffers for
the separate planes can be. Nicolas mentioned that I could support the
multiplanar API, but only allow the contiguous single planar formats,
so NV12 instead of NV12M.

And I suspect that reference frames have to be contiguous as well.
So I guess the overall answer is that it can't. But the same goes for
Hantro or rkvdec, which both use the multiplanar API.

This conversion was done so that I could use Cedrus as a testbed for
developing new codec support, as I already own an H6 device. Added
bonus is trying to get V4L2 decoder support to work for desktop Linux,
without the libraries that ChromeOS ships.

> Note that you have to choose whether to support single or multiplanar, yo=
u
> can't support both at the same time.
>
> So the decision to move to multiplanar should be led by the HW capabiliti=
es.
>
> And Chromium really needs to support non-multiplanar formats as well. I'm
> really surprised to hear that it doesn't, to be honest.

Chromium does support non-multiplanar formats, such as NV12. In fact
this is the preferred format, unless it's on MediaTek, in which it
switches to MM21 as the capture format, and does untiling in software.
This support for formats is separate from using the multiplanar API.

Support for video decoders is mostly driven by ChromeOS. Currently all the
hardware ChromeOS supports, be it stateful or stateless, use the multiplana=
r
API, so there hasn't been a real need to support it yet.


Regards
ChenYu

> Regards,
>
>         Hans
>
> >
> > Best regards,
> > Jernej
> >
> >>
> >> A fallback route using libv4l is also available. The library translate=
s
> >> MPLANE interface ioctl calls to non-MPLANE ones, provided that the pix=
el
> >> format used is single plane.
> >>
> >> Convert the Cedrus driver to the MPLANE interface, while keeping the
> >> supported formats single plane. Besides backward compatibility through
> >> the plugin, the hardware requires that different planes not be located
> >> too far apart in memory. Keeping the single plane pixel format makes
> >> this easy to enforce.
> >>
> >> Signed-off-by: Chen-Yu Tsai <wenst@chromium.org>
> >> ---
> >>
> >> This has been tested with Fluster. The score remained the same with or
> >> without the patch. This also helps with getting VP8 decoding working
> >> with Chromium's in-tree test program "video_decode_accelerator_tests",
> >> though Chromium requires other changes regarding buffer allocation and
> >> management.
> >
> >
>