Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <871126cc-a3ca-2009-8f1d-a8eb5852e033@synaptics.com>
Date:   Fri, 19 Aug 2022 07:51:03 +0800
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.12.0
Subject: Re: [PATCH 2/2] [WIP]: media: Add Synaptics compressed tiled format
Content-Language: en-US
To:     Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Cc:     Tomasz Figa <tfiga@chromium.org>, dri-devel@lists.freedesktop.org,
        maarten.lankhorst@linux.intel.com, mripard@kernel.org,
        tzimmermann@suse.de, airlied@linux.ie, daniel@ffwll.ch,
        mchehab@kernel.org, hverkuil-cisco@xs4all.nl,
        ezequiel@vanguardiasur.com.ar, sakari.ailus@linux.intel.com,
        ribalda@chromium.org, linux-media@vger.kernel.org,
        linux-kernel@vger.kernel.org, sebastian.hesselbarth@gmail.com,
        jszhang@kernel.org, linux-arm-kernel@lists.infradead.org
References: <20220808162750.828001-1-randy.li@synaptics.com>
 <20220808162750.828001-3-randy.li@synaptics.com>
 <CAAFQd5AKjpJ+fPAeCqdNnJbS4R7SdaHkfyW4qG1xXr-sE801pQ@mail.gmail.com>
 <13d37c15-79f3-4e16-8cf4-fc37846f4a04@synaptics.com>
 <Yv7HnHE7bLmgq5D0@pendragon.ideasonboard.com>
From:   Hsia-Jun Li <Randy.Li@synaptics.com>
In-Reply-To: <Yv7HnHE7bLmgq5D0@pendragon.ideasonboard.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?R21tSnYrM3FsekpIMzFmV1l6VmxST2NKU0Jwc2s0RDFkS1lvTzIvc2w4dWp0?=
 =?utf-8?B?a2xOSWY1YXllcEQ5N0VMWWNzM3BuSDB1NmF4NFl3WjJ1SWdQS3d2TU10M0FH?=
 =?utf-8?B?dWkxaXFrN3hiZWh0bWZFSzh1UUdMLzZoWTUvMHFNT1VjRC9YYU02M1Ywd1Vz?=
 =?utf-8?B?ZWRscFR3YmVSUVAxVlR2ZCtVbzVYZnRrYXlUNXhveUZNT3RsWTluZjFHcFB0?=
 =?utf-8?B?MTk1QmllSGg0UUJ5TktZNHFzVWF6djdaL2lkMkRQVUViOE5Bb3J0bmx5dzhJ?=
 =?utf-8?B?a3F2UDFNWUZYNzlOQVNrd0N4blJOajI2UlRMdEdaR1ZRanh2ZWxjVWpUQ3FZ?=
 =?utf-8?B?SFNWMGNncFNld2d5dlJqNmdRTU0wRlR5OXZLR3E1REVmeWRoaDdMMFowUk9h?=
 =?utf-8?B?dFd5SzNyaytuMHRpRzcvcTZlRFFrWmV3dGtrT21EYndLbmdqaG16dW9CWmtB?=
 =?utf-8?B?bHMyMldONzdOK0lJOGFQd3ZYR01PWGdFZmtXL3FDbnZTdFEwSXhOUzVmZUds?=
 =?utf-8?B?Z1ppNHp4SEd5WFFLUWMzeGEwRnA5aVJydy95ZGRNK2hLYWdGTHl4cXZWSUgx?=
 =?utf-8?B?ZlJtOXViLytEY1lQazlDd0ZvUm9tYmtUOG9hS0RFM1VSRkg4bGFsK0wxbWhk?=
 =?utf-8?B?YkZVK2dHM1pBcVZndHovWFpyaC9vaFVmS2Q4VHdPZk1PRjhRNG5ReHVYaWNh?=
 =?utf-8?B?K01lbGdhL2xYT1NVUDBuY0pydFJXL1dHd0RYZTRXMjRZTkZsdTh3OUlDd0U3?=
 =?utf-8?B?cXY1UGJkSlY2cm13L3NtMFdvQ0U3ZUQ0cXArcXA2dXB4SmhUU1FRTFRnQzZi?=
 =?utf-8?B?TEJpelM1NHZHR3NUb3dGSnRhRW9rZUNIb0hlQStKNE52M2l5OGU0VzdaL0dO?=
 =?utf-8?B?dGFSUW5nL0xFUnZ3VlZUaFBLTk5ZMEM1OXphNkV6c3RYWi9YMUVLbkFBeWd6?=
 =?utf-8?B?NDZkY1BrNzlQYWEwek8wRHZvazhEVGo3T2N5OTg2MUpORHY5UVBtS0VrMHRU?=
 =?utf-8?B?Rm5KdC94RCtaaTBVWTcwdG5oQXZ5dDIvbVNFdzIzdFE0YW03WndoUklsa0oy?=
 =?utf-8?B?ZE1ndGVlZlhWYkUvVHFVNnhlak1CcFd0V29Zcmk0cmhjUnNSOVl0VTN5eFcy?=
 =?utf-8?B?SWt5S2NtMFhVSFZndkIydlVQc2JobXpTRnROSFpPOVoxN2t4bVE3cURpVnZr?=
 =?utf-8?B?VEljL0VmOEU4dkV4TEVMYWdxK3JNVmtwVHhRTEw1dzd0Q1gyQlZ3dzFkRWo0?=
 =?utf-8?B?YThmYjdUNUtoMWJRcmV4M1paR2lTWDIwMEd4enVWRlc2RWpzMCtpdC9rWnFh?=
 =?utf-8?B?MVhrWDBROTM3VmZ0MTR0QmlhQmkvNHVObU9aNklMTHhZL21mNWtyeEdtYXM1?=
 =?utf-8?B?SitZWHJyekt4QVY2cGpLL05sUlp1b21mWXBKSU5WdlpLaHBVZmlNNlFZaTR1?=
 =?utf-8?B?VjZkN2cvZlJqUFZ3dDVSY05tQWpwSG1IcjB3S1pwNDFDa0dLaVR3cTI1TFg3?=
 =?utf-8?B?dCtFVmtUc2gvb3kwbklBNzVDMHJjbWJHMjFubUhkRG9EZFl0N0xFa2RGM0U3?=
 =?utf-8?B?bnZTdzRTY3Mvcm5MYkZiU3FQdHkxb1R5dXV5NTV5TkkyRmxKWWQ3U2tOWm9a?=
 =?utf-8?B?a1cvUmlCT0ErUmU5SjVMUzViSzAzNGVjMWlxakZwcnl2MUtWRFphVUhpWDlE?=
 =?utf-8?B?WnFZM2dqaEFGY21jS0FENGhqQ05naVJmdGhLOFNDdmFNZUtQU09PekZpTU1Y?=
 =?utf-8?B?NGcrY3dDN3ZmQ01MTStJaHk5ZVF6QWRLSE9LcnV0aHFXdTFJM0h2dWVKbEdS?=
 =?utf-8?B?bE42amprMEM2cGxQMEhvcDFIbDJYeFpveC9BWmRYaHJvL3hEUVRRdFhyczZi?=
 =?utf-8?B?RGtZNDVxR1h6djZ2QmRXVHdTdVdFRjdBSG1FMi94L1QzU1FJTHlxa0E3aWVM?=
 =?utf-8?B?ODdJSHBGUFc4RXZnQk82OUlDdXppZXZzVEwvQjhITFFrekl4YUxWWUQ3WStI?=
 =?utf-8?B?MUhOVjlaWXA1UWdhb3FpSTBHSVkvRkFBUlRlYi9GU2RsV2NDU0lkcEdSYUpL?=
 =?utf-8?B?N3g3VC9rcFlPUTZUcnYrcDYrZGRVbmdIRTl5VWFkaDZxcDBRV1NZdUJQK09t?=
 =?utf-8?Q?5F8ONErWuhNS5O35ni/dwSlqy?=
X-OriginatorOrg: synaptics.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 05d5ea7d-a164-498b-cbd6-08da81748953
X-MS-Exchange-CrossTenant-AuthSource: DM6PR03MB5196.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Aug 2022 23:51:14.7443
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 335d1fbc-2124-4173-9863-17e7051a2a0e
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: evDWh7Z9KAAANG6NbOXOggInHAqQBkR0jr7jWoGa3cibmle6QN5ieFm38nMyPCbcPPP4ORJT6Ewk5f6h/XT4pw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR03MB5744
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,
        SPF_HELO_PASS,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 8/19/22 07:13, Laurent Pinchart wrote:
> CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe.
> 
> 
> On Thu, Aug 18, 2022 at 02:33:42PM +0800, Hsia-Jun Li wrote:
>> On 8/18/22 14:06, Tomasz Figa wrote:
>>> On Tue, Aug 9, 2022 at 1:28 AM Hsia-Jun Li <randy.li@synaptics.com> wrote:
>>>>
>>>> From: "Hsia-Jun(Randy) Li" <randy.li@synaptics.com>
>>>>
>>>> The most of detail has been written in the drm.
> 
> This patch still needs a description of the format, which should go to
> Documentation/userspace-api/media/v4l/.

I just want t tell people we need an extra plane for MVTP and I don't 
have enough space here to place all the pixel formats.

Besides, I was thinking a modifer in v4l2_ext_pix_format is not enough.
Let's take a compression NV12 tile format as an example, we need
1. luma planes
2. chroma planes
3. compression meta data for luma
4. compression meta data for chroma
5. mvtp
and a single data planer version would be
1. luma and chroma data
2. compression meta data
3. mvtp

You see, we actually have 3 kind of data here(not including the 
compression options that I am thinking of storing them somewhere else).
> 
>>>> Please notice that the tiled formats here request
>>>> one more plane for storing the motion vector metadata.
>>>> This buffer won't be compressed, so you can't append
>>>> it to luma or chroma plane.
>>>
>>> Does the motion vector buffer need to be exposed to userspace? Is the
>>> decoder stateless (requires userspace to specify the reference frames)
>>> or stateful (manages the entire decoding process internally)?
>>
>> No, users don't need to access them at all. Just they need a different
>> dma-heap.
>>
>> You would only get the stateful version of both encoder and decoder.
> 
> Shouldn't the motion vectors be stored in a separate V4L2 buffer,
> submitted through a different queue then ?
Yes, I like that.
Proposal: A third buffer type for the reconstruction buffers in V4L2 M2M 
encoder
https://www.spinics.net/lists/linux-media/msg214565.html

Although the major usage for the decoder here is producing the non-tile 
buffers, the decoder of us could product the NV12 or the pixel formats 
that GPU likes, but it must happen at the same time a frame is decoded.
Still the reference buffer or we call them the real decoded frame would 
stay in a tiled format. More than one queue would be need here.
> 
>>>> Signed-off-by: Hsia-Jun(Randy) Li <randy.li@synaptics.com>
>>>> ---
>>>>    drivers/media/v4l2-core/v4l2-common.c | 1 +
>>>>    drivers/media/v4l2-core/v4l2-ioctl.c  | 2 ++
>>>>    include/uapi/linux/videodev2.h        | 2 ++
>>>>    3 files changed, 5 insertions(+)
>>>>
>>>> diff --git a/drivers/media/v4l2-core/v4l2-common.c b/drivers/media/v4l2-core/v4l2-common.c
>>>> index e0fbe6ba4b6c..f645278b3055 100644
>>>> --- a/drivers/media/v4l2-core/v4l2-common.c
>>>> +++ b/drivers/media/v4l2-core/v4l2-common.c
>>>> @@ -314,6 +314,7 @@ const struct v4l2_format_info *v4l2_format_info(u32 format)
>>>>                   { .format = V4L2_PIX_FMT_SGBRG12,       .pixel_enc = V4L2_PIXEL_ENC_BAYER, .mem_planes = 1, .comp_planes = 1, .bpp = { 2, 0, 0, 0 }, .hdiv = 1, .vdiv = 1 },
>>>>                   { .format = V4L2_PIX_FMT_SGRBG12,       .pixel_enc = V4L2_PIXEL_ENC_BAYER, .mem_planes = 1, .comp_planes = 1, .bpp = { 2, 0, 0, 0 }, .hdiv = 1, .vdiv = 1 },
>>>>                   { .format = V4L2_PIX_FMT_SRGGB12,       .pixel_enc = V4L2_PIXEL_ENC_BAYER, .mem_planes = 1, .comp_planes = 1, .bpp = { 2, 0, 0, 0 }, .hdiv = 1, .vdiv = 1 },
>>>> +               { .format = V4L2_PIX_FMT_NV12M_V4H1C, .pixel_enc = V4L2_PIXEL_ENC_YUV, .mem_planes = 5, .comp_planes = 2, .bpp = { 1, 2, 0, 0 }, .hdiv = 2, .vdiv = 2, .block_w = { 128, 128 }, .block_h = { 128, 128 } },
>>>>           };
>>>>           unsigned int i;
>>>>
>>>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
>>>> index e6fd355a2e92..8f65964aff08 100644
>>>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
>>>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
>>>> @@ -1497,6 +1497,8 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
>>>>                   case V4L2_PIX_FMT_MT21C:        descr = "Mediatek Compressed Format"; break;
>>>>                   case V4L2_PIX_FMT_QC08C:        descr = "QCOM Compressed 8-bit Format"; break;
>>>>                   case V4L2_PIX_FMT_QC10C:        descr = "QCOM Compressed 10-bit Format"; break;
>>>> +               case V4L2_PIX_FMT_NV12M_V4H1C:  descr = "Synaptics Compressed 8-bit tiled Format";break;
>>>> +               case V4L2_PIX_FMT_NV12M_10_V4H3P8C:     descr = "Synaptics Compressed 10-bit tiled Format";break;
>>>>                   default:
>>>>                           if (fmt->description[0])
>>>>                                   return;
>>>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>>>> index 01e630f2ec78..7e928cb69e7c 100644
>>>> --- a/include/uapi/linux/videodev2.h
>>>> +++ b/include/uapi/linux/videodev2.h
>>>> @@ -661,6 +661,8 @@ struct v4l2_pix_format {
>>>>    #define V4L2_PIX_FMT_NV12MT_16X16 v4l2_fourcc('V', 'M', '1', '2') /* 12  Y/CbCr 4:2:0 16x16 tiles */
>>>>    #define V4L2_PIX_FMT_NV12M_8L128      v4l2_fourcc('N', 'A', '1', '2') /* Y/CbCr 4:2:0 8x128 tiles */
>>>>    #define V4L2_PIX_FMT_NV12M_10BE_8L128 v4l2_fourcc_be('N', 'T', '1', '2') /* Y/CbCr 4:2:0 10-bit 8x128 tiles */
>>>> +#define V4L2_PIX_FMT_NV12M_V4H1C v4l2_fourcc('S', 'Y', '1', '2')   /* 12  Y/CbCr 4:2:0 tiles */
>>>> +#define V4L2_PIX_FMT_NV12M_10_V4H3P8C v4l2_fourcc('S', 'Y', '1', '0')   /* 12  Y/CbCr 4:2:0 10-bits tiles */
>>>>
>>>>    /* Bayer formats - see https://urldefense.proofpoint.com/v2/url?u=http-3A__www.siliconimaging.com_RGB-2520Bayer.htm&d=DwIBaQ&c=7dfBJ8cXbWjhc0BhImu8wVIoUFmBzj1s88r8EGyM0UY&r=P4xb2_7biqBxD4LGGPrSV6j-jf3C3xlR7PXU-mLTeZE&m=87M5Aa5fG3kdTTjlJrLIgv0E7O10QAj_4RqDlVsCAFdbfJzJ_P0s8wkBqaR0VBUO&s=8AsoiLPt9hQnn4ta51tT-RUXRLoKKYrePdAwtdvxuDo&e=   */
>>>>    #define V4L2_PIX_FMT_SBGGR8  v4l2_fourcc('B', 'A', '8', '1') /*  8  BGBG.. GRGR.. */
> 
> --
> Regards,
> 
> Laurent Pinchart

-- 
Hsia-Jun(Randy) Li