Subject: Re: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel
 Visual Sensing Controller(IVSC)
To:     Laurent Pinchart <laurent.pinchart@ideasonboard.com>,
        Wentong Wu <wentong.wu@intel.com>
Cc:     mchehab@kernel.org, sakari.ailus@linux.intel.com,
        linux-media@vger.kernel.org, srinivas.pandruvada@intel.com,
        pierre-louis.bossart@linux.intel.com, zhifeng.wang@intel.com,
        xiang.ye@intel.com, tian.shu.qiu@intel.com, bingbu.cao@intel.com,
        linux-kernel@vger.kernel.org
References: <20230213022347.2480307-1-wentong.wu@intel.com>
 <Y+ypKYI4c6fHL4Eu@pendragon.ideasonboard.com>
From:   Bingbu Cao <bingbu.cao@linux.intel.com>
Message-ID: <00c5c16e-c94e-a328-06d9-5f09ad35258d@linux.intel.com>
Date:   Wed, 15 Feb 2023 20:09:50 +0800
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.10.0
MIME-Version: 1.0
In-Reply-To: <Y+ypKYI4c6fHL4Eu@pendragon.ideasonboard.com>
Content-Type: text/plain; charset=windows-1252
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk


Hi, Wentong,

On 2/15/23 5:43 PM, Laurent Pinchart wrote:
> Hello Wentong,
> 
> On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
>> Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a
>> companion chip designed to provide secure and low power vision capability
>> to IA platforms. IVSC is available in existing commercial platforms from
>> multiple OEMs.
>>
>> The primary use case of IVSC is to bring in context awareness. IVSC
>> interfaces directly with the platform main camera sensor via a CSI-2 link
>> and processes the image data with the embedded AI engine. The detected
>> events are sent over I2C to ISH (Intel Sensor Hub) for additional data
>> fusion from multiple sensors. The fusion results are used to implement
>> advanced use cases like:
>>  - Face detection to unlock screen
>>  - Detect user presence to manage backlight setting or waking up system
> 
> Do you have plan to support these features in the ivsc driver in the
> future ?
> 
>> Since the Image Processing Unit(IPU) used on the host processor needs to
>> configure the CSI-2 link in normal camera usages, the CSI-2 link and
>> camera sensor can only be used in mutually-exclusive ways by host IPU and
>> IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU
>> driver can take ownership of the CSI-2 link and camera sensor using
>> interfaces provided by this IVSC driver.
>>
>> Switching ownership requires an interface with two different hardware
>> modules inside IVSC. The software interface to these modules is via Intel
>> MEI (The Intel Management Engine) commands. These two hardware modules
>> have two different MEI UUIDs to enumerate. These hardware modules are:
>>  - ACE (Algorithm Context Engine): This module is for algorithm computing
>> when IVSC owns camera sensor. Also ACE module controls camera sensor's
>> ownership. This hardware module is used to set ownership of camera sensor.
>>  - CSI (Camera Serial Interface): This module is used to route camera
>> sensor data either to IVSC or to host for IPU driver and application.
>>
>> IVSC also provides a privacy mode. When privacy mode is turned on,
>> camera sensor can't be used. This means that both ACE and host IPU can't
>> get image data. And when this mode is turned on, host IPU driver is
>> informed via a registered callback, so that user can be notified.
> 
> How does the privacy mode work, and how can the user trust that the
> closed-source IVSC and IME firmwares will honour the privacy settings ?
>

Continue with question from Laurent,

How IVSC handle the privacy request from user? Is there some notification
mechanism to user-space? I'd have concern if IVSC driver need private callback
to request back-end(e.g. ISP driver) to handle stream cutting.

>> In summary, to acquire ownership of camera by IPU driver, first ACE
>> module needs to be informed of ownership and then to setup MIPI CSI-2
>> link for the camera sensor and IPU.
>>
>> Implementation:
>> There are two different drivers to handle ACE and CSI hardware modules
>> inside IVSC.
>>  - mei_csi: MEI client driver to send commands and receive notifications
>> from CSI module.
>>  - mei_ace: MEI client driver to send commands and get status from ACE
>> module.
>> Interface is exposed via ivsc.h to acquire and release camera sensor and
>> CSI-2 link.
> 
> Do I understand correctly, from your diagram below, that the
> communication between the IME and IVSC goes through SPI ?
> 
>> Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
>> -----------------------------------------------------------------------------
>> | Host Processor                                                            |
>> |                                                                           |
>> |       -----------------       -----------------       ---------------     |
>> |       |               |       |               |       |             | I2C |
>> |       |      IPU      |       |      ISH      |       |camera driver|--|  |
>> |       |               |       |               |       |             |  |  |
>> |       -----------------       -----------------       ---------------  |  |
>> |               |                       |                      |         |  |
>> |               |                       |               ---------------  |  |
>> |               |                       |               |             |  |  |
>> |               |                       |               | IVSC driver |  |  |
>> |               |                       |               |             |  |  |
>> |               |                       |               ---------------  |  |
>> |               |                       |                      |         |  |
>> ----------------|-----------------------|----------------------|---------|---
>>                 | CSI                   | I2C                  |SPI      |
>>                 |                       |                      |         |
>> ----------------|-----------------------|----------------      |         |
>> | IVSC          |                                       |      |         |
>> |               |                                       |      |         |
>> |       -----------------       -----------------       |      |         |
>> |       |               |       |               |       |      |         |
>> |       |      CSI      |       |      ACE      |       |------|         |
>> |       |               |       |               |       |                |
>> |       -----------------       -----------------       |                |
>> |               |                       | I2C           |                |
>> ----------------|-----------------------|----------------                |
>>                 | CSI                   |                                |
>>                 |                       |                                |
>>             --------------------------------                             |
>>             |                              | I2C                         |
>>             |         camera sensor        |-----------------------------|
>>             |                              |
>>             --------------------------------
>>
>> Wentong Wu (3):
>>   media: pci: intel: ivsc: Add CSI submodule
>>   media: pci: intel: ivsc: Add ACE submodule
>>   media: pci: intel: ivsc: Add acquire/release API for ivsc
>>
>>  drivers/media/pci/Kconfig              |   1 +
>>  drivers/media/pci/intel/Makefile       |   2 +
>>  drivers/media/pci/intel/ivsc/Kconfig   |  12 +
>>  drivers/media/pci/intel/ivsc/Makefile  |   7 +
>>  drivers/media/pci/intel/ivsc/ivsc.c    |  84 +++++
>>  drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++
>>  drivers/media/pci/intel/ivsc/mei_ace.h |  36 ++
>>  drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++
>>  drivers/media/pci/intel/ivsc/mei_csi.h |  60 ++++
>>  include/linux/ivsc.h                   |  74 ++++
>>  10 files changed, 1090 insertions(+)
>>  create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
>>  create mode 100644 drivers/media/pci/intel/ivsc/Makefile
>>  create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
>>  create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
>>  create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
>>  create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
>>  create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
>>  create mode 100644 include/linux/ivsc.h
> 

-- 
Best regards,
Bingbu Cao