Received-SPF: pass (google.com: domain of linux-kernel+bounces-129431-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
From: Yunfei Dong <yunfei.dong@mediatek.com>
To: =?UTF-8?q?N=C3=ADcolas=20F=20=2E=20R=20=2E=20A=20=2E=20Prado?=
	<nfraprado@collabora.com>, Sebastian Fricke <sebastian.fricke@collabora.com>,
	Nicolas Dufresne <nicolas.dufresne@collabora.com>, Hans Verkuil
	<hverkuil-cisco@xs4all.nl>, AngeloGioacchino Del Regno
	<angelogioacchino.delregno@collabora.com>, Benjamin Gaignard
	<benjamin.gaignard@collabora.com>, Nathan Hebert <nhebert@chromium.org>
CC: Hsin-Yi Wang <hsinyi@chromium.org>, Fritz Koenig <frkoenig@chromium.org>,
	Daniel Vetter <daniel@ffwll.ch>, Steve Cho <stevecho@chromium.org>, Yunfei
 Dong <yunfei.dong@mediatek.com>, <linux-media@vger.kernel.org>,
	<devicetree@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
	<linux-arm-kernel@lists.infradead.org>, <linux-mediatek@lists.infradead.org>,
	<Project_Global_Chrome_Upstream_Group@mediatek.com>
Subject: [PATCH] media: mediatek: vcodec: fix the error sizeimage for 10bit bitstream
Date: Wed, 3 Apr 2024 17:30:18 +0800
Message-ID: <20240403093018.13168-1-yunfei.dong@mediatek.com>
Precedence: bulk
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Type: text/plain

The sizeimage of each plane are calculated the same way for 8bit and
10bit bitstream. Need to enlarge the sizeimage with simeimage*5/4 for
10bit bitstream when try and set fmt.

Fixes: 9d86be9bda6c ("media: mediatek: vcodec: Add driver to support 10bit")
Signed-off-by: Yunfei Dong <yunfei.dong@mediatek.com>
---
 .../mediatek/vcodec/decoder/mtk_vcodec_dec.c  | 47 ++++++++++++++-----
 1 file changed, 34 insertions(+), 13 deletions(-)

diff --git a/drivers/media/platform/mediatek/vcodec/decoder/mtk_vcodec_dec.c b/drivers/media/platform/mediatek/vcodec/decoder/mtk_vcodec_dec.c
index 9107707de6c4..45209894f1fe 100644
--- a/drivers/media/platform/mediatek/vcodec/decoder/mtk_vcodec_dec.c
+++ b/drivers/media/platform/mediatek/vcodec/decoder/mtk_vcodec_dec.c
@@ -259,6 +259,7 @@ static int vidioc_try_fmt(struct mtk_vcodec_dec_ctx *ctx, struct v4l2_format *f,
 		pix_fmt_mp->num_planes = 1;
 		pix_fmt_mp->plane_fmt[0].bytesperline = 0;
 	} else if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
+		unsigned int dram_y, dram_c, dram_y_10bit, dram_c_10bit;
 		int tmp_w, tmp_h;
 
 		/*
@@ -280,22 +281,42 @@ static int vidioc_try_fmt(struct mtk_vcodec_dec_ctx *ctx, struct v4l2_format *f,
 		    (pix_fmt_mp->height + 64) <= frmsize->max_height)
 			pix_fmt_mp->height += 64;
 
-		mtk_v4l2_vdec_dbg(0, ctx,
-				  "before resize wxh=%dx%d, after resize wxh=%dx%d, sizeimage=%d",
-				  tmp_w, tmp_h, pix_fmt_mp->width, pix_fmt_mp->height,
-				  pix_fmt_mp->width * pix_fmt_mp->height);
+		dram_y = pix_fmt_mp->width * pix_fmt_mp->height;
+		dram_c = dram_y / 2;
+
+		dram_y_10bit = dram_y * 5 / 4;
+		dram_c_10bit = dram_y_10bit / 2;
 
 		pix_fmt_mp->num_planes = fmt->num_planes;
-		pix_fmt_mp->plane_fmt[0].sizeimage =
-				pix_fmt_mp->width * pix_fmt_mp->height;
-		pix_fmt_mp->plane_fmt[0].bytesperline = pix_fmt_mp->width;
-
-		if (pix_fmt_mp->num_planes == 2) {
-			pix_fmt_mp->plane_fmt[1].sizeimage =
-				(pix_fmt_mp->width * pix_fmt_mp->height) / 2;
-			pix_fmt_mp->plane_fmt[1].bytesperline =
-				pix_fmt_mp->width;
+		if (pix_fmt_mp->num_planes == 1) {
+			if (ctx->is_10bit_bitstream) {
+				pix_fmt_mp->plane_fmt[0].bytesperline = pix_fmt_mp->width * 5 / 4;
+				pix_fmt_mp->plane_fmt[0].sizeimage = dram_y_10bit + dram_c_10bit;
+			} else {
+				pix_fmt_mp->plane_fmt[0].bytesperline = pix_fmt_mp->width;
+				pix_fmt_mp->plane_fmt[0].sizeimage = dram_y + dram_c;
+			}
+		} else {
+			if (ctx->is_10bit_bitstream) {
+				pix_fmt_mp->plane_fmt[0].bytesperline = pix_fmt_mp->width * 5 / 4;
+				pix_fmt_mp->plane_fmt[1].bytesperline = pix_fmt_mp->width * 5 / 4;
+
+				pix_fmt_mp->plane_fmt[0].sizeimage = dram_y_10bit;
+				pix_fmt_mp->plane_fmt[1].sizeimage = dram_c_10bit;
+			} else {
+				pix_fmt_mp->plane_fmt[0].bytesperline = pix_fmt_mp->width;
+				pix_fmt_mp->plane_fmt[1].bytesperline = pix_fmt_mp->width;
+
+				pix_fmt_mp->plane_fmt[0].sizeimage = dram_y;
+				pix_fmt_mp->plane_fmt[1].sizeimage = dram_c;
+			}
 		}
+
+		mtk_v4l2_vdec_dbg(0, ctx,
+				  "before resize:%dx%d, after resize:%dx%d, sizeimage=0x%x_0x%x",
+				  tmp_w, tmp_h, pix_fmt_mp->width, pix_fmt_mp->height,
+				  pix_fmt_mp->plane_fmt[0].sizeimage,
+				  pix_fmt_mp->plane_fmt[1].sizeimage);
 	}
 
 	pix_fmt_mp->flags = 0;
-- 
2.25.1