From:   "Wu, Wentong" <wentong.wu@intel.com>
To:     Sakari Ailus <sakari.ailus@linux.intel.com>
CC:     Bingbu Cao <bingbu.cao@linux.intel.com>,
        Laurent Pinchart <laurent.pinchart@ideasonboard.com>,
        "mchehab@kernel.org" <mchehab@kernel.org>,
        "linux-media@vger.kernel.org" <linux-media@vger.kernel.org>,
        "Pandruvada, Srinivas" <srinivas.pandruvada@intel.com>,
        "pierre-louis.bossart@linux.intel.com" 
        <pierre-louis.bossart@linux.intel.com>,
        "Wang, Zhifeng" <zhifeng.wang@intel.com>,
        "Ye, Xiang" <xiang.ye@intel.com>,
        "Qiu, Tian Shu" <tian.shu.qiu@intel.com>,
        "Cao, Bingbu" <bingbu.cao@intel.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: RE: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel
 Visual Sensing Controller(IVSC)
Thread-Topic: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel
 Visual Sensing Controller(IVSC)
Thread-Index: AQHZP1HlDdnw2M/ymEOJ6dDL3JcZGa7PxSqAgAApAACAAaPNgIAA0d4AgABNrbCAAErlAIAEKgKg
Date:   Mon, 20 Feb 2023 03:45:27 +0000
Message-ID: <DM6PR11MB4316B7A714EA1ABE62A4A46E8DA49@DM6PR11MB4316.namprd11.prod.outlook.com>
References: <20230213022347.2480307-1-wentong.wu@intel.com>
 <Y+ypKYI4c6fHL4Eu@pendragon.ideasonboard.com>
 <00c5c16e-c94e-a328-06d9-5f09ad35258d@linux.intel.com>
 <Y+4rtSMArg5ow0Sh@kekkonen.localdomain>
 <8022b9da-e033-b77b-914c-d35a31aac2e2@linux.intel.com>
 <DM6PR11MB4316F7C9854B113556466BF58DA19@DM6PR11MB4316.namprd11.prod.outlook.com>
 <Y+9bvl5SA2a1nXhe@kekkonen.localdomain>
In-Reply-To: <Y+9bvl5SA2a1nXhe@kekkonen.localdomain>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?vAZffxRedT/GIPYpTFvkQx4pxdlVLnI4O/UHz1AdNhPpG8gU8v5WTxKB7pxV?=
 =?us-ascii?Q?m3ZQQQxq4n6q16kjyoChQsYULCb5w0qBS4Ejk7l6WxCY5eoP1GdYSnUJWAIL?=
 =?us-ascii?Q?PThy+phcnHtXeGPAyNyhUuv1k+WCqmDThfj4rMAbz3fkmuI7kyzm9q/n+Q4Q?=
 =?us-ascii?Q?7U7vgaVzBfcnwhEqgMkEM2rPeyVWEG9JffXtdU5E/NCMOGbbxk9xTJ8MVP5Z?=
 =?us-ascii?Q?x0/KQqrk/O1sakD2l18xaiLIepSG/F/h6R8eCjq1tULdjAafMDKSi4gWXNXq?=
 =?us-ascii?Q?XvljNxkiJmlqGcxmRdAL8O0Wi1G+kKKEO+Dogc5QiYyt3tqGRNhBF8YLiRUy?=
 =?us-ascii?Q?C84eTHwSZadFQehpHKWKYE0hhB+7I3vHpIJC6ibXWnJtZgBFlxmxZG/PHfE7?=
 =?us-ascii?Q?ZdguXQCHIQvqghoubWi8XjnGRbUSkuAdUxhjhWMJ6rsqShvGtt5iXvtfMfje?=
 =?us-ascii?Q?8FZElttJcBPqypd3j04UQJBHT/DvGRVgLhr1baZGjDHmzQirEv3tnWkErHeL?=
 =?us-ascii?Q?+xW1wpHTQZg4TJP3FmYSEmmIjDMkewzipWnqh55ECF4AD9ov/Ijfve7CMJIG?=
 =?us-ascii?Q?oEmZQ5ZtLbyEPDCC2lXs36bLw8N9vcYh5lsw8Ao2Nfu0rW0xrWHmFyKHtEno?=
 =?us-ascii?Q?YUFVRNo50yWDASMdIErpl6wt8piaHUETKinZGghyRoLeQNNdPRi6v4awPoYL?=
 =?us-ascii?Q?2n4RJbs4AAx/KDDTxiSp9ihcmZNYYxpofwo4lwdEtmxEnH2bjKgFO59BxuMM?=
 =?us-ascii?Q?4+jZdG4++Z6Y8DpNjdQ9g03LxoEhuC9E01jFmVae1voRUvMajdvsGXRKp0N+?=
 =?us-ascii?Q?duD58Zp6La1yscAxLxOJr2O1sO4pLEeEgpYoJwpwCIBesCP7rbabgcm0USxv?=
 =?us-ascii?Q?xHflmsqQeGugWWFbqaJozUbz/HoNdBTB1GXLfKNkWfWK2pF3GEJWWf30tpsh?=
 =?us-ascii?Q?+T3WOdETOynZxMVvnbx0Do5DOfZeyRW65QnRWO69BHGg1vpy8CdiDkSq3agL?=
 =?us-ascii?Q?rKbueY73OPPrCeqQDMm765y7HbRbPq/10LptlkXvisEPfAbNAYNkbfXoZYtj?=
 =?us-ascii?Q?TJhLVzD/foyx/WdCliiAXLj8gkKOL9WBqZEsc4s42dtycLp4iwMOyWFXbiP+?=
 =?us-ascii?Q?dCEi/c0DbZXpm/i7+Cf/MQbO8me4nltkc2vnIssK5+XBN5tOqmpDf/px62Xp?=
 =?us-ascii?Q?LYNZPX4ZK33D9zirfI0u9zHWSH0nTV346aFMynKHDXGRlpO0zFKTd54vAyRA?=
 =?us-ascii?Q?5m6JXRPgbqtZbx1oEaq6W2id1q1RIu+7V9Gzxn7pQjgrxpro+he36c/LET3G?=
 =?us-ascii?Q?xxr4e7Ybvclfiz255IQFw8UXA9MMBLsuk/SY6nuy+/e5PGs+7JlwaHcFwQ8k?=
 =?us-ascii?Q?/XiFwaBVs+yc8Sm1iI3QOQFXCGxLCw48HYg8L3sxX0JWZAwiGVfPAgaFTGQc?=
 =?us-ascii?Q?nUPYRRHrj756F4vJAb/5++ZKy8uYz5GTy7shW4g4iU/jKHiMgQMIz7IjHe+k?=
 =?us-ascii?Q?5ezIeED/0YUQLT6yboXYFe7J2GOaiMxmixBbBwsfwOdwCammVztR7mpCGovO?=
 =?us-ascii?Q?aKmF5lR7KSP0rfPBIN9yaHkWuYYTyc6tPuRmzd1h?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: DM6PR11MB4316.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e6e20920-e1a2-4dfa-647c-08db12f4e7c2
X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Feb 2023 03:45:27.2497
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: eTSczhm1jKiJgOnb26IuHEvgHL4JV/CEiKsGNPeZLUWCXlnp5VMKitXUIrKfbzraG/I3xquhADE5ysaJVB+ywQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW3PR11MB4745
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Sakari,

> -----Original Message-----
> From: Sakari Ailus <sakari.ailus@linux.intel.com>
> Sent: Friday, February 17, 2023 6:50 PM
>=20
> Hi Wentong,
>=20
> On Fri, Feb 17, 2023 at 06:28:32AM +0000, Wu, Wentong wrote:
> >
> >
> > > -----Original Message-----
> > > From: Bingbu Cao <bingbu.cao@linux.intel.com>
> > > Sent: Friday, February 17, 2023 9:44 AM
> > >
> > > Hi, Sakari,
> > >
> > > On 2/16/23 9:12 PM, Sakari Ailus wrote:
> > > > Hi Bingbu, Wentong,
> > > >
> > > > On Wed, Feb 15, 2023 at 08:09:50PM +0800, Bingbu Cao wrote:
> > > >>
> > > >> Hi, Wentong,
> > > >>
> > > >> On 2/15/23 5:43 PM, Laurent Pinchart wrote:
> > > >>> Hello Wentong,
> > > >>>
> > > >>> On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
> > > >>>> Intel Visual Sensing Controller (IVSC), codenamed "Clover
> > > >>>> Falls", is a companion chip designed to provide secure and low
> > > >>>> power vision capability to IA platforms. IVSC is available in
> > > >>>> existing commercial platforms from multiple OEMs.
> > > >>>>
> > > >>>> The primary use case of IVSC is to bring in context awareness.
> > > >>>> IVSC interfaces directly with the platform main camera sensor
> > > >>>> via a
> > > >>>> CSI-2 link and processes the image data with the embedded AI
> > > >>>> engine. The detected events are sent over I2C to ISH (Intel
> > > >>>> Sensor
> > > >>>> Hub) for additional data fusion from multiple sensors. The
> > > >>>> fusion results are used to implement advanced use cases like:
> > > >>>>  - Face detection to unlock screen
> > > >>>>  - Detect user presence to manage backlight setting or waking
> > > >>>> up system
> > > >>>
> > > >>> Do you have plan to support these features in the ivsc driver in
> > > >>> the future ?
> > > >>>
> > > >>>> Since the Image Processing Unit(IPU) used on the host processor
> > > >>>> needs to configure the CSI-2 link in normal camera usages, the
> > > >>>> CSI-2 link and camera sensor can only be used in
> > > >>>> mutually-exclusive ways by host IPU and IVSC. By default the
> > > >>>> IVSC owns the CSI-2 link and camera sensor. The IPU driver can
> > > >>>> take ownership of the CSI-2 link and camera sensor using interfa=
ces
> provided by this IVSC driver.
> > > >>>>
> > > >>>> Switching ownership requires an interface with two different
> > > >>>> hardware modules inside IVSC. The software interface to these
> > > >>>> modules is via Intel MEI (The Intel Management Engine) commands.
> > > >>>> These two hardware modules have two different MEI UUIDs to
> enumerate.
> > > These hardware modules are:
> > > >>>>  - ACE (Algorithm Context Engine): This module is for algorithm
> > > >>>> computing when IVSC owns camera sensor. Also ACE module
> > > >>>> controls camera sensor's ownership. This hardware module is
> > > >>>> used to set ownership
> > > of camera sensor.
> > > >>>>  - CSI (Camera Serial Interface): This module is used to route
> > > >>>> camera sensor data either to IVSC or to host for IPU driver and
> application.
> > > >>>>
> > > >>>> IVSC also provides a privacy mode. When privacy mode is turned
> > > >>>> on, camera sensor can't be used. This means that both ACE and
> > > >>>> host IPU can't get image data. And when this mode is turned on,
> > > >>>> host IPU driver is informed via a registered callback, so that u=
ser can be
> notified.
> > > >>>
> > > >>> How does the privacy mode work, and how can the user trust that
> > > >>> the closed-source IVSC and IME firmwares will honour the privacy
> settings ?
> > >
> > > As I know, without IVSC, once user enable the privacy mode, the
> > > Intel Converged Security Engine will configure the IPU camera mask
> > > (security register), which will mask the specific CSI2 port and
> > > produce dummy imaging data. For the case with IVSC, there is no final
> solution on Linux so far I think.
> > >
> > > Wentong, is IVSC trying to cut off the stream and then notify user an=
d IPU?
> >
> > yes
>=20
> Does the CSI-2 transmitter on IVCS go to some LP mode during this time,=20

Yes, The low power mode is following the D-Phy spec.

> or does
> the receiver need to initialise the bus again when the stream resuems?

No, it's the same link.

>=20
> >
> > >
> > > >>>
> > > >>
> > > >> Continue with question from Laurent,
> > > >>
> > > >> How IVSC handle the privacy request from user? Is there some
> > > >> notification mechanism to user-space?
> >
> > IVSC has already defined privacy callback for host IPU/camera driver.
> >
> > > > I'd have concern if IVSC driver
> > > >> need private callback to request back-end(e.g. ISP driver) to
> > > >> handle stream
> > > cutting.
> > > >
> > > > How does the privacy mode work, does it just pass zeroes (or other
> > > > dummy
> > > > data) towards the host or nothing?
> >
> > No data on CSI transmitter side
>=20
> Can it stop in the middle of the frame?

No,

 Or is it guaranteed to produce full frames
> (assuming the sensor does)?

Yes, full frame will be guaranteed as camera sensor does.

>=20
> >
> > > >
> > > > A V4L2 control can be used for the purpose of passing the
> > > > information to the user space at least.
> >
> > I will take some time to review V4L2 sub-device and control mechanism,
> > and then update the driver.
>=20
> --
> Kind regards,
>=20
> Sakari Ailus