Received: by 2002:a25:4158:0:0:0:0:0 with SMTP id o85csp3792504yba; Tue, 23 Apr 2019 09:40:50 -0700 (PDT) X-Google-Smtp-Source: APXvYqxO8jNKgo3ieFVrbP8AT8e8etS4CiS303p78eP6kTb/b/McdorskQSfZogJ2Q1Hp03mukQy X-Received: by 2002:a63:6941:: with SMTP id e62mr23276246pgc.99.1556037650642; Tue, 23 Apr 2019 09:40:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556037650; cv=none; d=google.com; s=arc-20160816; b=0E1uqSG3iX2vv2cYWFND/9ZRGGJg5E2Ie4AM7ayifhANj1NZQm8yVTIlFwJy5BPZ/z JunaF8FR4J5kuLujai+wcpqkpSWsOaCXZaOBA7WR6lz99M1RbpCraXp/iaHOzohDqJL+ MScnZ1LEGENG4Lj9z5O5mROlHbzndXh1nPMSTe6Ewu84MSctvdYyV6Zg3MCZIXicm07/ o6O+MR6yqV2MNzd/tYfgtqN3T7siFMTdfn3G4hZSHC2ccnvq0w9UkW2tR8EJjhAzrtdg v+fuOwxGrM8EaYsYQuPsnEzq+P6ZliGuzgXII60iAgkQbCD3wi8+kS+0vo2EBoiooDr1 KatA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:date:cc:to:from:subject :message-id; bh=zVHOhB7lgARBbvMMBjhD7iww9E6WUSppYoaB2ABLKCM=; b=X7fwsKwM9iFsmsHlmW4td6rZnEKwzysNKSRJG8XlrWVrQia7x3OtLW1qHlisL8BUI9 rEKbSYwcSunRae+zaZ0RoM8x2VTdHbGJLUertPmV7u7za0KmF5gJAIW/AVjb/Zvt0A2h CfmXSGB6RS7nEHsodacztFIFp6fIvh6NwyvkPBFUvvXFZliihMEmSqgf0KZw6xnfS59l yBjjOyV7gURJ5w6ublJpoiie3uVKBckS12TREqvGgksjlEF7v/AZx6I2oo1cqEOc91hJ 46QR9/CRUhD/fJwvrtxBchFuNDFryFL4/orQjo6xVHNU9LiD7gDO/4Ay8/Yi5yS0KISv n0nA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id t17si15447480pgi.180.2019.04.23.09.40.35; Tue, 23 Apr 2019 09:40:50 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729577AbfDWQix (ORCPT + 99 others); Tue, 23 Apr 2019 12:38:53 -0400 Received: from relay6-d.mail.gandi.net ([217.70.183.198]:58409 "EHLO relay6-d.mail.gandi.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729557AbfDWQiv (ORCPT ); Tue, 23 Apr 2019 12:38:51 -0400 X-Originating-IP: 93.29.109.196 Received: from collins (196.109.29.93.rev.sfr.net [93.29.109.196]) (Authenticated sender: paul.kocialkowski@bootlin.com) by relay6-d.mail.gandi.net (Postfix) with ESMTPSA id AEFF9C0007; Tue, 23 Apr 2019 16:38:46 +0000 (UTC) Message-ID: Subject: Re: [PATCH 00/20] drm: Split out the formats API and move it to a common place From: Paul Kocialkowski To: Daniel Stone , Laurent Pinchart Cc: Maxime Ripard , Daniel Vetter , David Airlie , Maarten Lankhorst , Sean Paul , Mauro Carvalho Chehab , Sakari Ailus , Linux Kernel Mailing List , dri-devel , Hans Verkuil , Thomas Petazzoni , "open list:DMA BUFFER SHARING FRAMEWORK" Date: Tue, 23 Apr 2019 18:38:46 +0200 In-Reply-To: References: <20190417154121.GJ13337@phenom.ffwll.local> <20190418062229.eyog4i62eg4pr6uf@flea> <20190418090221.e57dogn4yx5nwdni@flea> <20190420225904.GZ4964@pendragon.ideasonboard.com> <20190423072554.GW13337@phenom.ffwll.local> <20190423155416.GI16111@pendragon.ideasonboard.com> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.32.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi, Le mardi 23 avril 2019 à 17:02 +0100, Daniel Stone a écrit : > Hi Laurent, > > On Tue, 23 Apr 2019 at 16:54, Laurent Pinchart > wrote: > > On Tue, Apr 23, 2019 at 09:59:37AM +0100, Daniel Stone wrote: > > > On Tue, 23 Apr 2019 at 08:26, Daniel Vetter wrote: > > > Totally. Let's take DRM_FORMAT_XRGB8888 + I915_FORMAT_MOD_Y_TILED as > > > an example. [... details ...] > > > > Looks like we have different kinds of metadata to consider. On the V4L2 > > side metadata usually refers to the context in which a frame was > > captured, it doesn't tell how to interpret the contents of the pixels. > > In the case you just described, the metadata is part of the frame > > contents. I agree that this is a proper use case for storing such > > metadata in a plane. What I wouldn't like to see being stored in a plane > > is for instance gamma tables or similar data that configures the > > processing pipeline in the display engine (I know we have an API for > > gamma tables, this is just an example). > > > > > It would be good to understand what you had in mind when you said that > > > using multiple planes created a mess. I haven't touched media > > > encode/decode units at a low level for quite a while (hooray for > > > gst-v4l2!), but I remember that they often used padding areas around > > > the buffer for scratch space - maybe motion vectors or similar? That > > > case is quite different to something like CCS, since the data is only > > > meaningful to the media engine and must be ignored (but preserved) by > > > everyone else. Using multiple planes in that case isn't appropriate, > > > since it's very specific to how that hardware unit deals with that > > > buffer, instead of something that every consumer needs to understand > > > in order to use it. > > > > With metadata unrelated to the pixel content, using a separate plane in > > the same buffer resulted in an explosion of the number of combinations > > that we would need to support, and ultimately led to a very ill-defined > > API. We decided to convey metadata related to the frame capture context > > (e.g. what exposure time was used for the frame) and processing pipeline > > configuration data in different buffers than the frame itself. > > Yeah, that makes sense. It's not really that different from what > happens with GPUs either: the final colour buffer the display > controller gets from a game is the product of a _lot_ of other work > which is invisible to the display controller, including things like > depth and stencil buffers, command buffers, etc etc. Those are closely > related to the frame production, but totally irrelevant for exchanging > the colour buffer with other subsystems. > > I think we should look at the metadata buffers you're describing in > the same way. Perhaps each V4L2 buffer could have driver-private > auxiliary buffer storage, or perhaps it's something you need to > separately expose to userspace as auxiliary data which must be > preserved but ignored. But modifiers are really only about what you > need when exchanging image colour buffers between subsystems, not > anything else. > > You're pretty close with gamma tables as well; for HDR and other kinds > of complex colour management, we need to carry a fair bit of auxiliary > information in order to display the image correctly. These have quite > different uses though: normally the colour buffer is produced by the > hardware and the primaries/whitepoints/etc are produced by software, > with the colour-management details remaining static across the life of > a swapchain even as you flip between multiple buffers. Given that, it > doesn't really make sense to try to stuff them into the same storage. I agree that we need to keep things minimal and clearly distinguish between what constitutes the description of the pixel buffer in terms of how it's laid out in memory and information about how the data should be interpreted. And there's indeed a fair share of things to consider there. Adding to the list above, I'm also thinking of the YUV colorspace information which must be passed along when displaying a buffer. But that's essentially not required to have a common description of buffers unified accross subsystems. Thinking about that, it would be interesting to have a common base structure for buffers used in v4l2 and drm. Ideally, that could be shared when doing dma-buf to avoid having userspace describe buffers and memory each time a buffer is imported into another subsystem. This could also help us solve the ambiguity related to the YUV M-suffixed formats. Another idea could be having common ioctls to get a unified buffer description from userspace and e.g. mmap on a per-plane basis (with virtual mappings like DRM does). What do you think? Cheers, Paul