Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757425AbcDDXwi (ORCPT ); Mon, 4 Apr 2016 19:52:38 -0400 Received: from youngberry.canonical.com ([91.189.89.112]:34940 "EHLO youngberry.canonical.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756499AbcDDX1m (ORCPT ); Mon, 4 Apr 2016 19:27:42 -0400 From: Kamal Mostafa To: linux-kernel@vger.kernel.org, stable@vger.kernel.org, kernel-team@lists.ubuntu.com Cc: Hans de Goede , Hans Verkuil , Mauro Carvalho Chehab , Kamal Mostafa Subject: [PATCH 3.13.y-ckt 13/97] [media] saa7134: Fix bytesperline not being set correctly for planar formats Date: Mon, 4 Apr 2016 16:25:37 -0700 Message-Id: <1459812421-933-14-git-send-email-kamal@canonical.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1459812421-933-1-git-send-email-kamal@canonical.com> References: <1459812421-933-1-git-send-email-kamal@canonical.com> X-Extended-Stable: 3.13 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2392 Lines: 64 3.13.11-ckt38 -stable review patch. If anyone has any objections, please let me know. ---8<------------------------------------------------------------ From: Hans de Goede commit 3e71da19f9dc22e39a755d6ae9678661abb66adc upstream. bytesperline should be the bytesperline for the first plane for planar formats, not that of all planes combined. This fixes a crash in xawtv caused by the wrong bpl. BugLink: https://bugzilla.redhat.com/show_bug.cgi?id=1305389 Reported-and-tested-by: Stas Sergeev Signed-off-by: Hans de Goede Signed-off-by: Hans Verkuil Signed-off-by: Mauro Carvalho Chehab Signed-off-by: Kamal Mostafa --- drivers/media/pci/saa7134/saa7134-video.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/drivers/media/pci/saa7134/saa7134-video.c b/drivers/media/pci/saa7134/saa7134-video.c index fb60da8..d11fdfb 100644 --- a/drivers/media/pci/saa7134/saa7134-video.c +++ b/drivers/media/pci/saa7134/saa7134-video.c @@ -1562,10 +1562,13 @@ static int saa7134_g_fmt_vid_cap(struct file *file, void *priv, f->fmt.pix.height = dev->height; f->fmt.pix.field = fh->cap.field; f->fmt.pix.pixelformat = dev->fmt->fourcc; - f->fmt.pix.bytesperline = - (f->fmt.pix.width * dev->fmt->depth) >> 3; + if (dev->fmt->planar) + f->fmt.pix.bytesperline = f->fmt.pix.width; + else + f->fmt.pix.bytesperline = + (f->fmt.pix.width * dev->fmt->depth) / 8; f->fmt.pix.sizeimage = - f->fmt.pix.height * f->fmt.pix.bytesperline; + (f->fmt.pix.height * f->fmt.pix.width * dev->fmt->depth) / 8; f->fmt.pix.colorspace = V4L2_COLORSPACE_SMPTE170M; f->fmt.pix.priv = 0; return 0; @@ -1646,10 +1649,13 @@ static int saa7134_try_fmt_vid_cap(struct file *file, void *priv, if (f->fmt.pix.height > maxh) f->fmt.pix.height = maxh; f->fmt.pix.width &= ~0x03; - f->fmt.pix.bytesperline = - (f->fmt.pix.width * fmt->depth) >> 3; + if (fmt->planar) + f->fmt.pix.bytesperline = f->fmt.pix.width; + else + f->fmt.pix.bytesperline = + (f->fmt.pix.width * fmt->depth) / 8; f->fmt.pix.sizeimage = - f->fmt.pix.height * f->fmt.pix.bytesperline; + (f->fmt.pix.height * f->fmt.pix.width * fmt->depth) / 8; f->fmt.pix.colorspace = V4L2_COLORSPACE_SMPTE170M; f->fmt.pix.priv = 0; -- 2.7.4