Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Message-ID: <3ea3bf5bf9904ce877142c41f595207752172d27.camel@ndufresne.ca>
Subject: Re: [PATCH v2 1/2] media: docs-rst: Document memory-to-memory video
 decoder interface
From:   Nicolas Dufresne <nicolas@ndufresne.ca>
To:     Tomasz Figa <tfiga@chromium.org>, Hans Verkuil <hverkuil@xs4all.nl>
Cc:     Linux Media Mailing List <linux-media@vger.kernel.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Mauro Carvalho Chehab <mchehab@kernel.org>,
        Pawel Osciak <posciak@chromium.org>,
        Alexandre Courbot <acourbot@chromium.org>,
        Kamil Debski <kamil@wypas.org>,
        Andrzej Hajda <a.hajda@samsung.com>,
        Kyungmin Park <kyungmin.park@samsung.com>,
        Jeongtae Park <jtp.park@samsung.com>,
        Philipp Zabel <p.zabel@pengutronix.de>,
        Tiffany Lin =?UTF-8?Q?=28=E6=9E=97=E6=85=A7=E7=8F=8A=29?= 
        <tiffany.lin@mediatek.com>,
        Andrew-CT Chen =?UTF-8?Q?=28=E9=99=B3=E6=99=BA=E8=BF=AA=29?= 
        <andrew-ct.chen@mediatek.com>,
        Stanimir Varbanov <stanimir.varbanov@linaro.org>,
        Todor Tomov <todor.tomov@linaro.org>,
        Paul Kocialkowski <paul.kocialkowski@bootlin.com>,
        Laurent Pinchart <laurent.pinchart@ideasonboard.com>,
        dave.stevenson@raspberrypi.org,
        Ezequiel Garcia <ezequiel@collabora.com>,
        Maxime Jourdan <maxi.jourdan@wanadoo.fr>
Date:   Thu, 24 Jan 2019 14:55:16 -0500
In-Reply-To: <CAAFQd5BPJv3cbJOWrziEjz_yE32DhfZv9vb-pG1Ltx-KS2=PQg@mail.gmail.com>
References: <20181022144901.113852-1-tfiga@chromium.org>
         <20181022144901.113852-2-tfiga@chromium.org>
         <cf0fc2fc-72c6-dbca-68f7-a349879a3a14@xs4all.nl>
         <CAAFQd5AORjMjHdavdr3zM13BnyFnKnEb-0aKNjvwbB_xJEnxgQ@mail.gmail.com>
         <9b7c1385-d482-6e92-2222-2daa835dbc91@xs4all.nl>
         <CAAFQd5DwjLt8UeDohzrMausaLGnOStvrmp5p7frYbG1hbGjx3Q@mail.gmail.com>
         <CAAFQd5BPJv3cbJOWrziEjz_yE32DhfZv9vb-pG1Ltx-KS2=PQg@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
User-Agent: Evolution 3.30.4 (3.30.4-1.fc29) 
Mime-Version: 1.0
Content-Transfer-Encoding: 8bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

Le jeudi 24 janvier 2019 à 18:06 +0900, Tomasz Figa a écrit :
> > Actually I just realized the last point might not even be achievable
> > for some of the decoders (s5p-mfc, mtk-vcodec), as they don't report
> > which frame originates from which bitstream buffer and the driver just
> > picks the most recently consumed OUTPUT buffer to copy the timestamp
> > from. (s5p-mfc actually "forgets" to set the timestamp in some cases
> > too...)
> > 
> > I need to think a bit more about this.
> 
> Actually I misread the code. Both s5p-mfc and mtk-vcodec seem to
> correctly match the buffers.

Ok good, since otherwise it would have been a regression in MFC driver.
This timestamp passing thing could in theory be made optional though,
it lives under some COPY_TIMESTAMP kind of flag. What that means though
is that a driver without such a capability would need to signal dropped
frames using some other mean.

In userspace, the main use is to match the produced frame against a
userspace specific list of frames. At least this seems to be the case
in Gst and Chromium, since the userspace list contains a superset of
the metadata found in the v4l2_buffer.

Now, using the produced timestamp, userspace can deduce frame that the
driver should have produced but didn't (could be a deadline case codec,
or simply the frames where corrupted). It's quite normal for a codec to
just keep parsing until it finally find something it can decode.

That's at least one way to do it, but there is other possible
mechanism. The sequence number could be used, or even producing buffers
with the ERROR flag set. What matters is just to give userspace a way
to clear these frames, which would simply grow userspace memory usage
over time.

Nicolas