Received: by 2002:ac0:a5a7:0:0:0:0:0 with SMTP id m36-v6csp330101imm; Tue, 7 Aug 2018 20:09:00 -0700 (PDT) X-Google-Smtp-Source: AA+uWPz6dtJDJNLGrLzlsEGxqybLrkdBNZeNwcmD4yZpalr0LRbShyNOS/Gbe6Osb7pGYYGV1t8/ X-Received: by 2002:a17:902:6b4c:: with SMTP id g12-v6mr820716plt.159.1533697740608; Tue, 07 Aug 2018 20:09:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533697740; cv=none; d=google.com; s=arc-20160816; b=iJeNbDHlVTEOs9VuXd0oujUmy1QMjCeBu4fO4TNnt9f/8cNweQlQz27fK9V1hPjm9o sR0XV/+1IQFw6bjQHxw9yvOGxjP03cE8DqmUuTVy+S1I0Vku/LeEWLIfZOEnD6b3sQcj VbNy8AwIG8H9S/B5jjVZaFyGmVkrFdPr5185/SdXws2mZvuJL35VfbWB1HAXqDUigScl 5CGwUglNY3xSZdWo1NKnvFkmZakxAUXl3SgqR3xNiyI1irfIWv92I4HV+f+2hq6sphrW /Uv0Bvj7Usy4FJXiIPJifkxP83yuGFGhPdq81ZzhoduHvL4EwrzKQ6dUu523RmWGt410 2RWA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:dkim-signature :arc-authentication-results; bh=4eKTNsXmlyRVhYcZT0orNOxHkL5AlBcDabEJ6uwLNVg=; b=c7qibB8a+wBPNimDv+QJlJwY14hrvybyFMAKp7UCDWBnlQhM+Y9vCnEzk5QglLPdQW eazip54jyHJe76xwO6U4TW9vShpJW9ivp4Lz169wC5uKE8oetNNdLVlngVpN5J/dfg78 vi1OOLqooanXaL2vVrexO5fZsUUo9uOJWoqaMgniXW9csAhAopVN7PIFJs9Sltc3gzUj tMb+yY4bx5CeJF5N1sW9ZEZzQ6BrXAjysjqiFwgrg36di98dJS/sHvDFSVyL+yl06FXa BCjDrUQb1l6NVTdiDvU41+bPXdbdjLDmlNHnxZvy9erd6CYhmap/hrhCg/R/Hhz6TdFI MCFQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b=IKxF9IA8; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=chromium.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e10-v6si3263906pge.48.2018.08.07.20.08.46; Tue, 07 Aug 2018 20:09:00 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b=IKxF9IA8; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=chromium.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726945AbeHHFZU (ORCPT + 99 others); Wed, 8 Aug 2018 01:25:20 -0400 Received: from mail-yw1-f41.google.com ([209.85.161.41]:41991 "EHLO mail-yw1-f41.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726579AbeHHFZU (ORCPT ); Wed, 8 Aug 2018 01:25:20 -0400 Received: by mail-yw1-f41.google.com with SMTP id y203-v6so534764ywd.9 for ; Tue, 07 Aug 2018 20:07:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=4eKTNsXmlyRVhYcZT0orNOxHkL5AlBcDabEJ6uwLNVg=; b=IKxF9IA8wJGk8sRWiYooVuPLZJG8XV4iQavDV8YManCO5sQYnnyJga1E59gZRhacoi xe7zgCtw/X+hRtGQcc4ZjzUpbSq93RNiU27CAhQKsV5exZaqEWlqV8Mm0ZF0NZiB5H9M MPZdMwbBvnXOdcYo2SV1bBFRIdVL+F/cEuASg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=4eKTNsXmlyRVhYcZT0orNOxHkL5AlBcDabEJ6uwLNVg=; b=I8m9VS+wjsP6a2H6Oh2aKtUt+ynX38zH1UQN4ld3EOLzu3NZkP5EMSlXDPpgGyloU7 z19B7IlfeT/FuxAS68plSplsgyvMkzL45Tttey3LyHnx/dLbycEQ6dRHZUzBa5OxaAPj l6/q9Jb6MvVGFHk8/+lhON2vXhPvuvKNTBIqjTDfM+4a1ls/2lzsOMqb/ZSo+/FtaZLN dTxwbe0hSUOGiVJum3m8Tnytp+VlTFun2x+E/Elfvh8RofuZwDDFIaAAfaf4fx7708m3 R3BHOhkC7V/BIFD7mhGF8RvxLXe+tdXBOK6jlfNO00cG5XWui5c+4aXsCOJjmFOByh0M IN8Q== X-Gm-Message-State: AOUpUlHWNRrxgnk4muM9jloy275HsV+k/9mmNd5nFIYG6XCHkuYqbLHb Kf+5XhROD2QTE3wC1Gl7DxYAgJ4Clx0= X-Received: by 2002:a25:80c8:: with SMTP id c8-v6mr477225ybm.203.1533697672097; Tue, 07 Aug 2018 20:07:52 -0700 (PDT) Received: from mail-yw1-f48.google.com (mail-yw1-f48.google.com. [209.85.161.48]) by smtp.gmail.com with ESMTPSA id a124-v6sm1220681ywc.96.2018.08.07.20.07.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 07 Aug 2018 20:07:50 -0700 (PDT) Received: by mail-yw1-f48.google.com with SMTP id r184-v6so540179ywg.6 for ; Tue, 07 Aug 2018 20:07:49 -0700 (PDT) X-Received: by 2002:a25:5709:: with SMTP id l9-v6mr542289ybb.226.1533697669282; Tue, 07 Aug 2018 20:07:49 -0700 (PDT) MIME-Version: 1.0 References: <20180724140621.59624-1-tfiga@chromium.org> <20180724140621.59624-2-tfiga@chromium.org> <37a8faea-a226-2d52-36d4-f9df194623cc@xs4all.nl> In-Reply-To: From: Tomasz Figa Date: Wed, 8 Aug 2018 12:07:37 +0900 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH 1/2] media: docs-rst: Document memory-to-memory video decoder interface To: Maxime Jourdan Cc: Hans Verkuil , Linux Media Mailing List , Linux Kernel Mailing List , Stanimir Varbanov , Mauro Carvalho Chehab , Pawel Osciak , Alexandre Courbot , kamil@wypas.org, a.hajda@samsung.com, Kyungmin Park , jtp.park@samsung.com, Philipp Zabel , =?UTF-8?B?VGlmZmFueSBMaW4gKOael+aFp+ePiik=?= , =?UTF-8?B?QW5kcmV3LUNUIENoZW4gKOmZs+aZuui/qik=?= , todor.tomov@linaro.org, nicolas@ndufresne.ca, Paul Kocialkowski , Laurent Pinchart , dave.stevenson@raspberrypi.org, Ezequiel Garcia Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Aug 8, 2018 at 4:11 AM Maxime Jourdan wrote: > > 2018-08-07 9:13 GMT+02:00 Hans Verkuil : > > On 07/26/2018 12:20 PM, Tomasz Figa wrote: > >> Hi Hans, > >> > >> On Wed, Jul 25, 2018 at 8:59 PM Hans Verkuil wrote: > >>>> + > >>>> +14. Call :c:func:`VIDIOC_STREAMON` to initiate decoding frames. > >>>> + > >>>> +Decoding > >>>> +======== > >>>> + > >>>> +This state is reached after a successful initialization sequence. In this > >>>> +state, client queues and dequeues buffers to both queues via > >>>> +:c:func:`VIDIOC_QBUF` and :c:func:`VIDIOC_DQBUF`, following standard > >>>> +semantics. > >>>> + > >>>> +Both queues operate independently, following standard behavior of V4L2 > >>>> +buffer queues and memory-to-memory devices. In addition, the order of > >>>> +decoded frames dequeued from ``CAPTURE`` queue may differ from the order of > >>>> +queuing coded frames to ``OUTPUT`` queue, due to properties of selected > >>>> +coded format, e.g. frame reordering. The client must not assume any direct > >>>> +relationship between ``CAPTURE`` and ``OUTPUT`` buffers, other than > >>>> +reported by :c:type:`v4l2_buffer` ``timestamp`` field. > >>> > >>> Is there a relationship between capture and output buffers w.r.t. the timestamp > >>> field? I am not aware that there is one. > >> > >> I believe the decoder was expected to copy the timestamp of matching > >> OUTPUT buffer to respective CAPTURE buffer. Both s5p-mfc and coda seem > >> to be implementing it this way. I guess it might be a good idea to > >> specify this more explicitly. > > > > What about an output buffer producing multiple capture buffers? Or the case > > where the encoded bitstream of a frame starts at one output buffer and ends > > at another? What happens if you have B frames and the order of the capture > > buffers is different from the output buffers? > > > > In other words, for codecs there is no clear 1-to-1 relationship between an > > output buffer and a capture buffer. And we never defined what the 'copy timestamp' > > behavior should be in that case or if it even makes sense. > > > > Regards, > > > > Hans > > As it is done right now in userspace (FFmpeg, GStreamer) and most (if > not all?) drivers, it's a 1:1 between OUTPUT and CAPTURE. The only > thing that changes is the ordering since OUTPUT buffers are in > decoding order while CAPTURE buffers are in presentation order. If I understood it correctly, there is a feature in VP9 that lets one frame repeat several times, which would make one OUTPUT buffer produce multiple CAPTURE buffers. Moreover, V4L2_PIX_FMT_H264 is actually defined to be a byte stream, without any need for framing, and yes, there are drivers that follow this definition correctly (s5p-mfc and, AFAIR, coda). In that case, one OUTPUT buffer can have arbitrary amount of bitstream and lead to multiple CAPTURE frames being produced. > > This almost always implies some timestamping kung-fu to match the > OUTPUT timestamps with the corresponding CAPTURE timestamps. It's > often done indirectly by the firmware on some platforms (rpi comes to > mind iirc). I don't think there is an upstream driver for it, is there? (If not, are you aware of any work towards it?) > > The current constructions also imply one video packet per OUTPUT > buffer. If a video packet is too big to fit in a buffer, FFmpeg will > crop that packet to the maximum buffer size and will discard the > remaining packet data. GStreamer will abort the decoding. This is > unfortunately one of the shortcomings of having fixed-size buffers. > And if they were to split the packet in multiple buffers, then some > drivers in their current state wouldn't be able to handle the > timestamping issues and/or x:1 OUTPUT:CAPTURE buffer numbers. In Chromium, we just allocate OUTPUT buffers big enough to be really unlikely for a single frame not to fit inside [1]. Obviously it's a waste of memory, for formats which normally have just single frames inside buffers, but it seems to work in practice. [1] https://cs.chromium.org/chromium/src/media/gpu/v4l2/v4l2_video_decode_accelerator.h?rcl=3468d5a59e00bcb2c2e946a30694e6057fd9ab21&l=118 Best regards, Tomasz