Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <ede840d10677041d2c8d0a07b81b3cd55a6c92ba.camel@ndufresne.ca>
Subject: Re: [PATCH 2/2] [WIP]: media: Add Synaptics compressed tiled format
From:   Nicolas Dufresne <nicolas@ndufresne.ca>
To:     Hsia-Jun Li <Randy.Li@synaptics.com>
Cc:     Tomasz Figa <tfiga@chromium.org>, dri-devel@lists.freedesktop.org,
        maarten.lankhorst@linux.intel.com, mripard@kernel.org,
        tzimmermann@suse.de, airlied@linux.ie, daniel@ffwll.ch,
        mchehab@kernel.org, hverkuil-cisco@xs4all.nl,
        ezequiel@vanguardiasur.com.ar, sakari.ailus@linux.intel.com,
        ribalda@chromium.org,
        Laurent Pinchart <laurent.pinchart@ideasonboard.com>,
        linux-media@vger.kernel.org, linux-kernel@vger.kernel.org,
        sebastian.hesselbarth@gmail.com, jszhang@kernel.org,
        linux-arm-kernel@lists.infradead.org
Date:   Tue, 23 Aug 2022 09:44:22 -0400
In-Reply-To: <2f3c8f6d-fc01-353e-fb74-b7f9af1ed2c4@synaptics.com>
References: <20220808162750.828001-1-randy.li@synaptics.com>
         <20220808162750.828001-3-randy.li@synaptics.com>
         <CAAFQd5AKjpJ+fPAeCqdNnJbS4R7SdaHkfyW4qG1xXr-sE801pQ@mail.gmail.com>
         <13d37c15-79f3-4e16-8cf4-fc37846f4a04@synaptics.com>
         <Yv7HnHE7bLmgq5D0@pendragon.ideasonboard.com>
         <6da7faf329128312f0862f555d1a855437ae99f3.camel@ndufresne.ca>
         <50dd9b7a-8f48-0799-57f6-048d20de8dcc@synaptics.com>
         <2662ac698898f71f60b9b7e0ad4703854de1d012.camel@ndufresne.ca>
         <1f926989-eb13-14ee-e30d-ac6d01b86c52@synaptics.com>
         <e15165f60ce801dd502bfe1992ea430fe37c5a91.camel@ndufresne.ca>
         <2f3c8f6d-fc01-353e-fb74-b7f9af1ed2c4@synaptics.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.44.4 (3.44.4-1.fc36) 
MIME-Version: 1.0
Precedence: bulk

Le mardi 23 ao=C3=BBt 2022 =C3=A0 15:40 +0800, Hsia-Jun Li a =C3=A9crit=C2=
=A0:
> > In current state, If your driver can support it, userland does not stri=
ctly
> > need
> > to re-allocate if the resolution is changed to smaller. In most SVC
> > scenarios,
> > the largest resolution is known in advance, so pre-allocation can happe=
n to
> > the
> When you play a video from Youtube, you may notice that starting=20
> resolution is low, then after it received more data knowning the=20
> bandwidth is enough, it would switch to a higher resolution. I don't=20
> think it would inform the codecs2 or OMX there is a higher target=20
> resolution.
>=20
> Besides, for the case of SVC in a conference system, the remote(gatway)=
=20
> would not tell you there is a higer resolution or frame rate because you=
=20
> can't receive it in negotiate stage, it could be permanently(device=20
> capability) or just bandwidth problem. Whether we know there is a higher=
=20
> requirement video depends on the transport protocols used here.
>=20
> The basic idea of SVC is that the low layer didn't depends on the upper=
=20
> layer, we can't tell how the bitstream usually.

I'm not saying against the fact the for drivers without IOMMU (hitting dire=
ctly
into the CMA allocator), allocation latency is massive challenge, and a
mechanism to smoothly reallocate (rather then mass-reallocation) is needed =
in
the long run. This is what I'm referring to when saying that folks have
considered extending CREATE_BUFS() with a DELETE_BUFS() ioctl.

Note that there is tones of software trickery you can use to mitigate this.=
 The
most simple one is to use CREATE_BUFS() instead of REQBUFS(). Instead of
reallocating all the buffers you need in one go, you would allocate them on=
e by
one. This will distribute allocation latency. For stateful CODEC, most OMX
focused firmware needs to be modified for that, since they stick with the o=
ld
OMX spec which did not allow run-time allocation.

Another trick is to use a second codec session. Both stateful/stateless COD=
EC
have support for concurrent decoding. On the MSE requirement, is that the s=
tream
transition happens only on keyframe boundary. Meaning, there is no need to =
reuse
the same session, you can create a new decoder in parallel, and that before=
 the
drain is complete (after the event, before the last buffer). This will comp=
ress
the "setup" latency, to the cost of some extra memory usage. Specially in t=
he
MSE case, this is nearly always possible since browsers do require support =
for
more then 1 concurrent decode. This method also works with OMX style CODEC
without any modification.

regards,
Nicolas