Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <452a4d48-fd73-d539-25f6-ded6bc6212bc@synaptics.com>
Date:   Fri, 21 Jul 2023 16:56:30 +0800
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.12.0
Subject: Re: [PATCH 1/2] [RESEND] media: v4l2-mem2mem: allow device run
 without buf
Content-Language: en-GB
To:     Tomasz Figa <tfiga@chromium.org>,
        Nicolas Dufresne <nicolas@ndufresne.ca>
Cc:     linux-media@vger.kernel.org, ayaka@soulik.info,
        hans.verkuil@cisco.com, mchehab@kernel.org,
        laurent.pinchart@ideasonboard.com, hiroh@chromium.org,
        hverkuil@xs4all.nl, linux-kernel@vger.kernel.org,
        Sebastian Fricke <sebastian.fricke@collabora.com>
References: <20230704040044.681850-1-randy.li@synaptics.com>
 <20230704040044.681850-2-randy.li@synaptics.com>
 <20452e233a9a4b39b58139081d818d3b1454105a.camel@ndufresne.ca>
 <20230712093129.pdcbvlxd5zphwr5m@chromium.org>
 <17ca8c5ee31ac8e48bb2ac7f89f6bae84816c893.camel@ndufresne.ca>
From:   Hsia-Jun Li <Randy.Li@synaptics.com>
In-Reply-To: <17ca8c5ee31ac8e48bb2ac7f89f6bae84816c893.camel@ndufresne.ca>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?WjlnMFlIcEQ1YnNXSzZlZVRuRUFSVDBMTUdaazNDMTRGQUZKQnhJKzJ1L01I?=
 =?utf-8?B?V1M4MWdwQ0E5ejF1MjhPOVBZSmVSN1hUdjcya28vVkMzVWFZVzB6TE5KYXk3?=
 =?utf-8?B?bmNEeGpib1BHdERRMkZiQ2h1ekRDK0Z5blNNKzVmWjh4MkJEbDkvUFJTTlJ0?=
 =?utf-8?B?UzJSYTFPSWNkRThDZEVDMW03ZDFhN0lHNFg0NWNsZXFMckVTbWljdmg3cGE3?=
 =?utf-8?B?eHhCdjBhYnZQbHBzUFcxOE5scS9PUGNIM01WNGVYc2dSQ2NwTldDQW1CTTdY?=
 =?utf-8?B?ZWUrbTJvdGd6aXlXSUJEdVpHdHlYaktzUGVPTlVmNlBOUVp3NGtiWTV1Rlhx?=
 =?utf-8?B?QzRVdUdZdGMrS3pGbXFncEhXd1NPWHMxeGU3UkVFcjRJc2VvdkRzOWdzM3RJ?=
 =?utf-8?B?RlBHR3B3RGc0RmhIN0FtNkhkWFN1NkJ1MldCVjRlT2FHYklLMlBDMXljbDk0?=
 =?utf-8?B?YmRpV0FuNDJKS1NUK1I5U1BodXhIZVh5dWRtZTNDTXJoT0dqcGd0Y3JHOFkw?=
 =?utf-8?B?ZkxhQ25XRldyaDEwOWFVZDJOWTAwNGRZOFh1bWlKZ00rU3JrZkpKcmw4SnhD?=
 =?utf-8?B?Y213aFRoaE84ekthTklxMUtOTm4yOFBRMTEreTVLdUhSYm0xL2ptei9LMVh3?=
 =?utf-8?B?SHNkVjBONkFDbm5tdTMyM014RDBIdVBWSk5rbVVJVGxadjl2Qi9lRForMGxn?=
 =?utf-8?B?bWgxcTdnWFZjUnQwQVRJMWhhVEpKVk5zd1lWOGFwNmZpRXlDTGErMjlHOTR3?=
 =?utf-8?B?eWFIR05iQ1VvUHpBYWkrU2hQYXRxUkRKdURycXk5b3l4VUtEVElyNDRkSkh0?=
 =?utf-8?B?TXBBbFpzSjNPODkxckhmakJEVTJhbHpYTEpQcnZpeUN4MU82Z1VPamhxZHVo?=
 =?utf-8?B?NnRlTnQ5a25tUldvT3c3QkxBbEZDOXJwTWp5VnMzcFJ0SHptWUIxbkxUTjFj?=
 =?utf-8?B?MWo4OVJDMFVsNjB0S295SHVJSndkbW02c0FrRFQzK3VsdkJIU3o3SElVeFAw?=
 =?utf-8?B?Z3FiMWltclRqNEFkWVZIMy9BYy9ueHkwWkJYQlM3bXBoWlVZbXJ4Tk9tYzQv?=
 =?utf-8?B?dk9EbzJxM0l3cytqd1c4UTBpUjZCWXB5VXBLWnJzOWx3ajJvZXN2VEV5Q2R5?=
 =?utf-8?B?aU5QS2I2OTRQSThpTFErdi9mSS9NNitTWExLS2t6SDlFempORW9IR01IY2tD?=
 =?utf-8?B?cGRmbEt2bGJEanpNRWV0a3Z5a0hyc091eUZVRWNzRU5CM1VUKytIbElSRW5L?=
 =?utf-8?B?VzdROFRVM01WUDMrN1ZXTXRsYkdTVE5sQStReGFZMEpqZXp1UmNveW5ncFJZ?=
 =?utf-8?B?NUtmVVVveXVsQUlTZG9iSlA1TWxVS3I5UXR4L0wwNUQvaWhsVUUrTkZRNWtJ?=
 =?utf-8?B?b1JrV29mWXVEUlB4eDJ3cEFoSVVUMVF3Vys5cnJNNnlSRnBQZGhVdW9wZDJM?=
 =?utf-8?B?Q0dmZTVwR0h6RytRRXg3NDFjUTkyL3VmY2RXdEpwbTNFUHJZQ2QzQS9yeHNm?=
 =?utf-8?B?TTRWelhRbEpaNlFOTTdXVFgxYjBNWFh3L1JXUzJrdWNabHFSbFRNZlJxbm5V?=
 =?utf-8?B?MU9Ucm5uMVlnc2pzVXJoT29Jc0F3WHhOcDd1L1RHRmtHRTB0RTN0VHBSNmJM?=
 =?utf-8?B?WlpSNmhBVTRiblp0dDVsSytscjJ2eTdJcVA3VzNuUHNGWFdNSDczL0lTV000?=
 =?utf-8?B?VTRuK2RRNHI3cXJlY1ZFbzNmcjhiZVNBM0xKSmFjZWNIV2RhNEovQXNJN0kz?=
 =?utf-8?B?TmxKVXV2M0lSeXNqOVI1ZGQ4M2RrU1dJTHBQa0xORmhaaDBvbnB0OEQ3WjZ1?=
 =?utf-8?B?RmhpWTdsRFhjK1ZDbmdWVnZwYjNjb00xQUpGTmczeWkrZEJLcFdLeHhnNUxN?=
 =?utf-8?B?QndCZFJXaHpGSHpyQ2pKMUVDRU5ndXFxZ0N0Zm45UnZyRFE4T0Y2ZG03aFNV?=
 =?utf-8?B?OTJJMFNHeHUyc2JkdjltdERjUEF4OVVwUDhmOW82WERGTzBWY2lGN1JZWXN4?=
 =?utf-8?B?czBBQ2N4SGg0a0VkSDRNVVNmSmxHUlVPcGdGTVBSeEpqUEorQ1BPR2RwTnoy?=
 =?utf-8?B?MzRuK2wvbDV2MEhXWitKb1Rtc0c5MXBpbFJrc041b1hoaXBNRXlYd21LZjBY?=
 =?utf-8?Q?Tfzg4Jy9MExqrPz6cwLjo9hpB?=
X-OriginatorOrg: synaptics.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c4cb6a47-7a18-4128-a0eb-08db89c8644e
X-MS-Exchange-CrossTenant-AuthSource: DM6PR03MB5196.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jul 2023 08:56:37.4663
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335d1fbc-2124-4173-9863-17e7051a2a0e
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: m5tBnb/vkOhg1sZmYlI/pmjmxezeLzVC7Dvlm1n0TM9OG969rNIxwKu0crNsgLqJGE4r2tjBFEJjWJ1PxImXOQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR03MB5132
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,NICE_REPLY_A,SPF_HELO_PASS,SPF_PASS,T_SCC_BODY_TEXT_LINE,
        URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 7/17/23 22:00, Nicolas Dufresne wrote:
> CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe.
> 
> 
> Le mercredi 12 juillet 2023 à 09:31 +0000, Tomasz Figa a écrit :
>> On Fri, Jul 07, 2023 at 03:14:23PM -0400, Nicolas Dufresne wrote:
>>> Hi Randy,
>>>
>>> Le mardi 04 juillet 2023 à 12:00 +0800, Hsia-Jun Li a écrit :
>>>> From: Randy Li <ayaka@soulik.info>
>>>>
>>>> For the decoder supports Dynamic Resolution Change,
>>>> we don't need to allocate any CAPTURE or graphics buffer
>>>> for them at inital CAPTURE setup step.
>>>>
>>>> We need to make the device run or we can't get those
>>>> metadata.
>>>>
>>>> Signed-off-by: Randy Li <ayaka@soulik.info>
>>>> ---
>>>>   drivers/media/v4l2-core/v4l2-mem2mem.c | 5 +++--
>>>>   1 file changed, 3 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/drivers/media/v4l2-core/v4l2-mem2mem.c b/drivers/media/v4l2-core/v4l2-mem2mem.c
>>>> index 0cc30397fbad..c771aba42015 100644
>>>> --- a/drivers/media/v4l2-core/v4l2-mem2mem.c
>>>> +++ b/drivers/media/v4l2-core/v4l2-mem2mem.c
>>>> @@ -301,8 +301,9 @@ static void __v4l2_m2m_try_queue(struct v4l2_m2m_dev *m2m_dev,
>>>>
>>>>    dprintk("Trying to schedule a job for m2m_ctx: %p\n", m2m_ctx);
>>>>
>>>> - if (!m2m_ctx->out_q_ctx.q.streaming
>>>> -     || !m2m_ctx->cap_q_ctx.q.streaming) {
>>>> + if (!(m2m_ctx->out_q_ctx.q.streaming || m2m_ctx->out_q_ctx.buffered)
>>>> +     || !(m2m_ctx->cap_q_ctx.q.streaming
>>>> +          || m2m_ctx->cap_q_ctx.buffered)) {
>>>
>>> I have a two atches with similar goals in my wave5 tree. It will be easier to
>>> upstream with an actual user, though, I'm probably a month or two away from
>>> submitting this driver again.
>>>
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__gitlab.collabora.com_chipsnmedia_kernel_-2D_commit_ac59eafd5076c4deb3bfe1fb85b3b776586ef3eb&d=DwIFaQ&c=7dfBJ8cXbWjhc0BhImu8wVIoUFmBzj1s88r8EGyM0UY&r=P4xb2_7biqBxD4LGGPrSV6j-jf3C3xlR7PXU-mLTeZE&m=9eWwqueFnh1yZHTW11j-syNVQvema7iBzNQeX1GKUQwXZ9pm6V4HDL_R2tIYKoOw&s=Ez5AyEYFIAJmC_k00IPO_ImzVdLZjr_veRq1bN4RSNg&e=
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__gitlab.collabora.com_chipsnmedia_kernel_-2D_commit_5de4fbe0abb20b8e8d862b654f93e3efeb1ef251&d=DwIFaQ&c=7dfBJ8cXbWjhc0BhImu8wVIoUFmBzj1s88r8EGyM0UY&r=P4xb2_7biqBxD4LGGPrSV6j-jf3C3xlR7PXU-mLTeZE&m=9eWwqueFnh1yZHTW11j-syNVQvema7iBzNQeX1GKUQwXZ9pm6V4HDL_R2tIYKoOw&s=tM81gjNe-bTjpjmidZ1sAhiodMh6npcVJNOhMCi1mPo&e=
>>>
>>
>> While I'm not going to NAK this series or those 2 patches if you send
>> them, I'm not really convinced that adding more and more complexity to
>> the mem2mem helpers is a good idea, especially since all of those seem
>> to be only needed by stateful video decoders.
>>
>> The mem2mem framework started as a set of helpers to eliminate boiler
>> plate from simple drivers that always get 1 CAPTURE and 1 OUTPUT buffer,
>> run 1 processing job on them and then return both of the to the userspace
>> and I think it should stay like this.
> 
> Its a bit late to try and bring that argument. It should have been raised couple
> of years ago (before I even started helping with these CODEC). Now that all the
> newly written stately decoders uses this framework, it is logical to keep
> reducing the boiler plate for these too. In my opinion, the job_ready()
> callback, should have been a lot more flexible from the start. And allowing
> driver to make it more powerful does not really add that much complexity.
> 
> Speaking of complexity, driving the output manually (outside of the job
> workqueue) during sequence initialization is a way more complex and risky then
> this. Finally, sticking with 1:1 pattern means encoder, detilers, image
> enhancement reducing framerate, etc. would all be unwelcome to use this. Which
> in short, means no one should even use this.
> 
I think those things are m2m, but it would be hard to present in current 
m2m framework:
1. N:1 compositor(It may be implemented as a loop running 2:1 compositor).
2. AV1 film gain
3. HDR with dynamic meta data to SDR

The video things fix for m2m model could be just a little less complex 
than ISP or camera pipeline. The only difference is just ISP won't have 
multiple contexts running at the same time.
If we could design a model for the video encoder I think we could solve 
the most of problems.
A video encoder would have:
1. input graphics buffer
2. reconstruction graphics buffer
3. motion vector cache buffer(optional)
4. coded bitstream output
5. encoding statistic report
>>
>> I think we're strongly in need of a stateful video decoder framework that
>> would actually address the exact problems that those have rather than
>> bending something that wasn't designed with them in mind to work around the
>> differences.
> 
> The bend is already there, of course I'd be happy to help with any new
> framework. Specially on modern stateless, were there is a need to do better
> scheduling.
I didn't know the schedule problem about stateless codec, are they 
supposed to be in the job queue when the buffers that DPB requests are 
own by the driver and its registers are prepared except the trigger bit?
  Just ping me if you have some effort starting, I don't currently
> have a budget or bandwidth to write new drivers or port existing drivers them on
> a newly written framework.
> 
> Nicolas
> 
> 
> [...]

-- 
Hsia-Jun(Randy) Li