Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a
companion chip designed to provide secure and low power vision capability
to IA platforms. IVSC is available in existing commercial platforms from
multiple OEMs.
The primary use case of IVSC is to bring in context awareness. IVSC
interfaces directly with the platform main camera sensor via a CSI-2 link
and processes the image data with the embedded AI engine. The detected
events are sent over I2C to ISH (Intel Sensor Hub) for additional data
fusion from multiple sensors. The fusion results are used to implement
advanced use cases like:
- Face detection to unlock screen
- Detect user presence to manage backlight setting or waking up system
Since the Image Processing Unit(IPU) used on the host processor needs to
configure the CSI-2 link in normal camera usages, the CSI-2 link and
camera sensor can only be used in mutually-exclusive ways by host IPU and
IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU
driver can take ownership of the CSI-2 link and camera sensor using
interfaces provided by this IVSC driver.
Switching ownership requires an interface with two different hardware
modules inside IVSC. The software interface to these modules is via Intel
MEI (The Intel Management Engine) commands. These two hardware modules
have two different MEI UUIDs to enumerate. These hardware modules are:
- ACE (Algorithm Context Engine): This module is for algorithm computing
when IVSC owns camera sensor. Also ACE module controls camera sensor's
ownership. This hardware module is used to set ownership of camera sensor.
- CSI (Camera Serial Interface): This module is used to route camera
sensor data either to IVSC or to host for IPU driver and application.
IVSC also provides a privacy mode. When privacy mode is turned on,
camera sensor can't be used. This means that both ACE and host IPU can't
get image data. And when this mode is turned on, host IPU driver is
informed via a registered callback, so that user can be notified.
In summary, to acquire ownership of camera by IPU driver, first ACE
module needs to be informed of ownership and then to setup MIPI CSI-2
link for the camera sensor and IPU.
Implementation:
There are two different drivers to handle ACE and CSI hardware modules
inside IVSC.
- mei_csi: MEI client driver to send commands and receive notifications
from CSI module.
- mei_ace: MEI client driver to send commands and get status from ACE
module.
Interface is exposed via ivsc.h to acquire and release camera sensor and
CSI-2 link.
Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
-----------------------------------------------------------------------------
| Host Processor |
| |
| ----------------- ----------------- --------------- |
| | | | | | | I2C |
| | IPU | | ISH | |camera driver|--| |
| | | | | | | | |
| ----------------- ----------------- --------------- | |
| | | | | |
| | | --------------- | |
| | | | | | |
| | | | IVSC driver | | |
| | | | | | |
| | | --------------- | |
| | | | | |
----------------|-----------------------|----------------------|---------|---
| CSI | I2C |SPI |
| | | |
----------------|-----------------------|---------------- | |
| IVSC | | | |
| | | | |
| ----------------- ----------------- | | |
| | | | | | | |
| | CSI | | ACE | |------| |
| | | | | | |
| ----------------- ----------------- | |
| | | I2C | |
----------------|-----------------------|---------------- |
| CSI | |
| | |
-------------------------------- |
| | I2C |
| camera sensor |-----------------------------|
| |
--------------------------------
Wentong Wu (3):
media: pci: intel: ivsc: Add CSI submodule
media: pci: intel: ivsc: Add ACE submodule
media: pci: intel: ivsc: Add acquire/release API for ivsc
drivers/media/pci/Kconfig | 1 +
drivers/media/pci/intel/Makefile | 2 +
drivers/media/pci/intel/ivsc/Kconfig | 12 +
drivers/media/pci/intel/ivsc/Makefile | 7 +
drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++
drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++
drivers/media/pci/intel/ivsc/mei_ace.h | 36 ++
drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++
drivers/media/pci/intel/ivsc/mei_csi.h | 60 ++++
include/linux/ivsc.h | 74 ++++
10 files changed, 1090 insertions(+)
create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
create mode 100644 drivers/media/pci/intel/ivsc/Makefile
create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
create mode 100644 include/linux/ivsc.h
--
2.25.1
CSI is a submodule of IVSC which can route camera sensor data
to the outbound MIPI CSI-2 interface.
The interface communicating with firmware is via MEI. There is
a separate MEI UUID, which this driver uses to enumerate.
To route camera sensor data to host, the caller specifies link
frequency and number of data lanes. This information is sent to
firmware by sending MEI command.
Two APIs are exported: csi_set_link_owner is used to switch
ownership of CSI-2 link, and csi_set_link_cfg is to configure
CSI-2 link when routing camera sensor data to host.
CSI also provides a privacy mode. When privacy mode is turned
on, camera sensor can't be used. This means that both IVSC and
host Image Processing Unit(IPU) can't get image data. And when
this mode is turned on, host Image Processing Unit(IPU) driver
is informed via the registered callback, so that user can be
notified.
Signed-off-by: Wentong Wu <[email protected]>
---
drivers/media/pci/Kconfig | 1 +
drivers/media/pci/intel/Makefile | 2 +
drivers/media/pci/intel/ivsc/Kconfig | 12 +
drivers/media/pci/intel/ivsc/Makefile | 5 +
drivers/media/pci/intel/ivsc/mei_csi.c | 342 +++++++++++++++++++++++++
drivers/media/pci/intel/ivsc/mei_csi.h | 60 +++++
include/linux/ivsc.h | 19 ++
7 files changed, 441 insertions(+)
create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
create mode 100644 drivers/media/pci/intel/ivsc/Makefile
create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
create mode 100644 include/linux/ivsc.h
diff --git a/drivers/media/pci/Kconfig b/drivers/media/pci/Kconfig
index 1224d908713a..16661ab0b816 100644
--- a/drivers/media/pci/Kconfig
+++ b/drivers/media/pci/Kconfig
@@ -74,6 +74,7 @@ config VIDEO_PCI_SKELETON
when developing new drivers.
source "drivers/media/pci/intel/ipu3/Kconfig"
+source "drivers/media/pci/intel/ivsc/Kconfig"
endif #MEDIA_PCI_SUPPORT
endif #PCI
diff --git a/drivers/media/pci/intel/Makefile b/drivers/media/pci/intel/Makefile
index 0b4236c4db49..d27ca636c860 100644
--- a/drivers/media/pci/intel/Makefile
+++ b/drivers/media/pci/intel/Makefile
@@ -4,3 +4,5 @@
#
obj-y += ipu3/
+
+obj-$(CONFIG_INTEL_VSC) += ivsc/
diff --git a/drivers/media/pci/intel/ivsc/Kconfig b/drivers/media/pci/intel/ivsc/Kconfig
new file mode 100644
index 000000000000..9535ac10f4f7
--- /dev/null
+++ b/drivers/media/pci/intel/ivsc/Kconfig
@@ -0,0 +1,12 @@
+# SPDX-License-Identifier: GPL-2.0-only
+# Copyright (C) 2023, Intel Corporation. All rights reserved.
+
+config INTEL_VSC
+ tristate "Intel Visual Sensing Controller"
+ depends on INTEL_MEI
+ help
+ This adds support for Intel Visual Sensing Controller (IVSC).
+
+ Enables the IVSC firmware services required for controlling
+ camera sensor ownership and CSI-2 link through Image Processing
+ Unit(IPU) driver of Intel.
diff --git a/drivers/media/pci/intel/ivsc/Makefile b/drivers/media/pci/intel/ivsc/Makefile
new file mode 100644
index 000000000000..1825aad45cff
--- /dev/null
+++ b/drivers/media/pci/intel/ivsc/Makefile
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2023, Intel Corporation. All rights reserved.
+
+obj-$(CONFIG_INTEL_VSC) += mei_csi.o
diff --git a/drivers/media/pci/intel/ivsc/mei_csi.c b/drivers/media/pci/intel/ivsc/mei_csi.c
new file mode 100644
index 000000000000..309e52387502
--- /dev/null
+++ b/drivers/media/pci/intel/ivsc/mei_csi.c
@@ -0,0 +1,342 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2023 Intel Corporation. All rights reserved.
+ * Intel Visual Sensing Controller CSI Linux driver
+ */
+
+/*
+ * To set ownership of CSI-2 link and to configure CSI-2 link, there
+ * are specific commands, which are sent via MEI protocol. The send
+ * command function uses "completion" as a synchronization mechanism.
+ * The response for command is received via a mei callback which wakes
+ * up the caller. There can be only one outstanding command at a time.
+ */
+
+#include <linux/completion.h>
+#include <linux/delay.h>
+#include <linux/kernel.h>
+#include <linux/math64.h>
+#include <linux/mei_cl_bus.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/slab.h>
+#include <linux/units.h>
+#include <linux/uuid.h>
+#include <linux/workqueue.h>
+
+#include "mei_csi.h"
+
+#define MEI_CSI_DRIVER_NAME "ivsc_csi"
+
+/* the 5s used here is based on experiment */
+#define CSI_CMD_TIMEOUT (5 * HZ)
+/* to setup CSI-2 link an extra delay needed and determined experimentally */
+#define CSI_FW_READY_DELAY_MS 100
+/* link frequency unit is 100kHz */
+#define CSI_LINK_FREQ(x) ((u32)(div_u64(x, 100 * HZ_PER_KHZ)))
+
+/*
+ * identify the command id supported by firmware
+ * IPC, as well as the privacy notification id
+ * used when processing privacy event.
+ */
+enum csi_cmd_id {
+ /* used to set csi ownership */
+ CSI_SET_OWNER = 0,
+
+ /* used to configure CSI-2 link */
+ CSI_SET_CONF = 2,
+
+ /* privacy notification id used when privacy state changes */
+ CSI_PRIVACY_NOTIF = 6,
+};
+
+/* configuration of the CSI-2 link between host and IVSC */
+struct csi_link_cfg {
+ /* number of data lanes used on the CSI-2 link */
+ u32 nr_of_lanes;
+
+ /* frequency of the CSI-2 link */
+ u32 link_freq;
+
+ /* for future use */
+ u32 rsvd[2];
+} __packed;
+
+/* CSI command structure */
+struct csi_cmd {
+ u32 cmd_id;
+ union _cmd_param {
+ u32 param;
+ struct csi_link_cfg conf;
+ } param;
+} __packed;
+
+/* CSI notification structure */
+struct csi_notif {
+ u32 cmd_id;
+ int status;
+ union _resp_cont {
+ u32 cont;
+ struct csi_link_cfg conf;
+ } cont;
+} __packed;
+
+struct mei_csi {
+ struct mei_cl_device *cldev;
+
+ /* command response */
+ struct csi_notif cmd_response;
+ /* used to wait for command response from firmware */
+ struct completion cmd_completion;
+
+ /* work element used to handle firmware event */
+ struct work_struct event_work;
+
+ /* privacy status */
+ enum ivsc_privacy_status status;
+ /* privacy callback */
+ void (*callback)(void *, enum ivsc_privacy_status);
+ /* privacy callback runtime context */
+ void *context;
+};
+
+/* only one for now */
+static struct mei_csi *csi;
+/* lock used to prevent multiple call to csi */
+static DEFINE_MUTEX(csi_mutex);
+
+/* send a command to firmware and mutex must be held by caller */
+static int mei_csi_send(u8 *buf, size_t len)
+{
+ struct csi_cmd *cmd = (struct csi_cmd *)buf;
+ int ret;
+
+ reinit_completion(&csi->cmd_completion);
+
+ ret = mei_cldev_send(csi->cldev, buf, len);
+ if (ret < 0)
+ goto out;
+
+ ret = wait_for_completion_killable_timeout(&csi->cmd_completion,
+ CSI_CMD_TIMEOUT);
+ if (ret < 0) {
+ goto out;
+ } else if (!ret) {
+ ret = -ETIMEDOUT;
+ goto out;
+ }
+
+ /* command response status */
+ ret = csi->cmd_response.status;
+ if (ret) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ if (csi->cmd_response.cmd_id != cmd->cmd_id)
+ ret = -EINVAL;
+
+out:
+ return ret;
+}
+
+int csi_set_link_owner(enum csi_link_owner owner,
+ void (*callback)(void *, enum ivsc_privacy_status),
+ void *context)
+{
+ struct csi_cmd cmd = { 0 };
+ size_t cmd_size;
+ int ret;
+
+ cmd.cmd_id = CSI_SET_OWNER;
+ cmd.param.param = owner;
+ cmd_size = sizeof(cmd.cmd_id) + sizeof(cmd.param.param);
+
+ mutex_lock(&csi_mutex);
+ if (unlikely(!csi)) {
+ mutex_unlock(&csi_mutex);
+ return -EAGAIN;
+ }
+
+ ret = mei_csi_send((u8 *)&cmd, cmd_size);
+
+ /*
+ * cancel possible event work and wait for it to finish
+ * to make sure no work running for the ongoing changes
+ * to callback and context.
+ */
+ cancel_work_sync(&csi->event_work);
+ csi->callback = callback;
+ csi->context = context;
+ mutex_unlock(&csi_mutex);
+
+ return ret;
+}
+EXPORT_SYMBOL_NS_GPL(csi_set_link_owner, IVSC);
+
+int csi_set_link_cfg(u32 nr_of_lanes, u64 link_freq)
+{
+ struct csi_cmd cmd = { 0 };
+ size_t cmd_size;
+ int ret;
+
+ cmd.cmd_id = CSI_SET_CONF;
+ cmd.param.conf.nr_of_lanes = nr_of_lanes;
+ cmd.param.conf.link_freq = CSI_LINK_FREQ(link_freq);
+ cmd_size = sizeof(cmd.cmd_id) + sizeof(cmd.param.conf);
+
+ mutex_lock(&csi_mutex);
+ if (unlikely(!csi)) {
+ mutex_unlock(&csi_mutex);
+ return -EAGAIN;
+ }
+
+ ret = mei_csi_send((u8 *)&cmd, cmd_size);
+ /*
+ * wait configuration ready if download success. placing
+ * delay under mutex is to make sure current command flow
+ * completed before starting a possible new one.
+ */
+ if (!ret)
+ msleep(CSI_FW_READY_DELAY_MS);
+ mutex_unlock(&csi_mutex);
+
+ return ret;
+}
+EXPORT_SYMBOL_NS_GPL(csi_set_link_cfg, IVSC);
+
+/* event handling routine */
+static void mei_csi_event_work(struct work_struct *work)
+{
+ if (csi->callback)
+ csi->callback(csi->context, READ_ONCE(csi->status));
+}
+
+/* callback for receive */
+static void mei_csi_rx(struct mei_cl_device *cldev)
+{
+ struct csi_notif notif = { 0 };
+ int ret;
+
+ ret = mei_cldev_recv(cldev, (u8 *)¬if, sizeof(notif));
+ if (ret < 0) {
+ dev_err(&cldev->dev, "recv error: %d\n", ret);
+ return;
+ }
+
+ switch (notif.cmd_id) {
+ case CSI_PRIVACY_NOTIF:
+ if (notif.cont.cont < IVSC_PRIVACY_MAX) {
+ WRITE_ONCE(csi->status, notif.cont.cont);
+
+ schedule_work(&csi->event_work);
+ }
+ break;
+ case CSI_SET_OWNER:
+ case CSI_SET_CONF:
+ memcpy(&csi->cmd_response, ¬if, ret);
+
+ complete(&csi->cmd_completion);
+ break;
+ default:
+ break;
+ }
+}
+
+static int mei_csi_probe(struct mei_cl_device *cldev,
+ const struct mei_cl_device_id *id)
+{
+ int ret;
+
+ mutex_lock(&csi_mutex);
+ /*
+ * only instance is possible in the current hardware,
+ * but adding protection for future hardware.
+ */
+ if (csi) {
+ ret = -EBUSY;
+ goto err_unlock;
+ }
+
+ csi = kzalloc(sizeof(*csi), GFP_KERNEL);
+ if (!csi) {
+ ret = -ENOMEM;
+ goto err_unlock;
+ }
+
+ csi->cldev = cldev;
+ init_completion(&csi->cmd_completion);
+ INIT_WORK(&csi->event_work, mei_csi_event_work);
+
+ mei_cldev_set_drvdata(cldev, csi);
+
+ ret = mei_cldev_enable(cldev);
+ if (ret < 0) {
+ dev_err(&cldev->dev, "mei_cldev_enable failed: %d\n", ret);
+ goto err_free;
+ }
+
+ ret = mei_cldev_register_rx_cb(cldev, mei_csi_rx);
+ if (ret) {
+ dev_err(&cldev->dev, "event cb registration failed: %d\n", ret);
+ goto err_disable;
+ }
+ mutex_unlock(&csi_mutex);
+
+ return 0;
+
+err_disable:
+ mei_cldev_disable(cldev);
+
+err_free:
+ kfree(csi);
+ /* disable csi */
+ csi = NULL;
+
+err_unlock:
+ mutex_unlock(&csi_mutex);
+
+ return ret;
+}
+
+static void mei_csi_remove(struct mei_cl_device *cldev)
+{
+ mutex_lock(&csi_mutex);
+ /* disable mei csi client device */
+ mei_cldev_disable(cldev);
+
+ /* cancel event work and wait for it to finish */
+ cancel_work_sync(&csi->event_work);
+
+ kfree(csi);
+ /* disable csi */
+ csi = NULL;
+ mutex_unlock(&csi_mutex);
+}
+
+#define MEI_CSI_UUID UUID_LE(0x92335FCF, 0x3203, 0x4472, \
+ 0xAF, 0x93, 0x7b, 0x44, 0x53, 0xAC, 0x29, 0xDA)
+
+static const struct mei_cl_device_id mei_csi_tbl[] = {
+ { MEI_CSI_DRIVER_NAME, MEI_CSI_UUID, MEI_CL_VERSION_ANY },
+
+ /* required last entry */
+ { }
+};
+MODULE_DEVICE_TABLE(mei, mei_csi_tbl);
+
+static struct mei_cl_driver mei_csi_driver = {
+ .id_table = mei_csi_tbl,
+ .name = MEI_CSI_DRIVER_NAME,
+
+ .probe = mei_csi_probe,
+ .remove = mei_csi_remove,
+};
+
+module_mei_cl_driver(mei_csi_driver);
+
+MODULE_AUTHOR("Wentong Wu <[email protected]>");
+MODULE_AUTHOR("Zhifeng Wang <[email protected]>");
+MODULE_DESCRIPTION("Device driver for IVSC CSI");
+MODULE_LICENSE("GPL");
diff --git a/drivers/media/pci/intel/ivsc/mei_csi.h b/drivers/media/pci/intel/ivsc/mei_csi.h
new file mode 100644
index 000000000000..0cc51e6a5250
--- /dev/null
+++ b/drivers/media/pci/intel/ivsc/mei_csi.h
@@ -0,0 +1,60 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2023 Intel Corporation. All rights reserved.
+ * Intel Visual Sensing Controller CSI external interface
+ */
+
+#ifndef _MEI_CSI_H_
+#define _MEI_CSI_H_
+
+#include <linux/ivsc.h>
+
+/* CSI-2 link ownership definition */
+enum csi_link_owner {
+ CSI_LINK_IVSC,
+ CSI_LINK_HOST,
+};
+
+#if IS_ENABLED(CONFIG_INTEL_VSC)
+/*
+ * @brief set CSI-2 link ownership
+ *
+ * @param owner The csi ownership being set
+ * @param callback The pointer of privacy callback being set
+ * @param context Privacy callback runtime context
+ *
+ * @return 0 on success, negative on failure
+ */
+int csi_set_link_owner(enum csi_link_owner owner,
+ void (*callback)(void *, enum ivsc_privacy_status),
+ void *context);
+
+/*
+ * @brief configure CSI-2 link between host and IVSC
+ * with provided parameters
+ *
+ * @param nr_of_lanes The number of data lanes being used
+ * on the CSI-2 link
+ * @param link_freq The frequency being set on the CSI-2 link
+ *
+ * @return 0 on success, negative on failure
+ */
+int csi_set_link_cfg(u32 nr_of_lanes, u64 link_freq);
+
+#else
+static inline
+int csi_set_link_owner(enum csi_link_owner owner,
+ void (*callback)(void *, enum ivsc_privacy_status),
+ void *context)
+{
+ return 0;
+}
+
+static inline int csi_set_link_cfg(u32 nr_of_lanes, u64 link_freq)
+{
+ return 0;
+}
+
+#endif
+
+#endif
diff --git a/include/linux/ivsc.h b/include/linux/ivsc.h
new file mode 100644
index 000000000000..6572ca4f340c
--- /dev/null
+++ b/include/linux/ivsc.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2023 Intel Corporation. All rights reserved.
+ * Intel Visual Sensing Controller interface
+ */
+
+#ifndef _LINUX_IVSC_H_
+#define _LINUX_IVSC_H_
+
+#include <linux/types.h>
+
+/* IVSC privacy status definition */
+enum ivsc_privacy_status {
+ IVSC_PRIVACY_OFF,
+ IVSC_PRIVACY_ON,
+ IVSC_PRIVACY_MAX,
+};
+
+#endif
--
2.25.1
ACE is a submodule of IVSC which controls camera sensor's
ownership, belonging to host or IVSC. When IVSC owns camera
sensor, it is for algorithm computing. When host wants to
control camera sensor, ACE module needs to be informed of
ownership with defined interface.
The interface is via MEI. There is a separate MEI UUID, which
this driver uses to enumerate.
To switch ownership of camera sensor between IVSC and host,
the caller specifies the defined ownership information which
will be sent to firmware by sending MEI command.
The only exported API ace_set_camera_owner is to switch
ownership of camera sensor.
Signed-off-by: Wentong Wu <[email protected]>
---
drivers/media/pci/intel/ivsc/Makefile | 1 +
drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++
drivers/media/pci/intel/ivsc/mei_ace.h | 36 ++
3 files changed, 509 insertions(+)
create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
diff --git a/drivers/media/pci/intel/ivsc/Makefile b/drivers/media/pci/intel/ivsc/Makefile
index 1825aad45cff..de0a425c22c2 100644
--- a/drivers/media/pci/intel/ivsc/Makefile
+++ b/drivers/media/pci/intel/ivsc/Makefile
@@ -3,3 +3,4 @@
# Copyright (C) 2023, Intel Corporation. All rights reserved.
obj-$(CONFIG_INTEL_VSC) += mei_csi.o
+obj-$(CONFIG_INTEL_VSC) += mei_ace.o
diff --git a/drivers/media/pci/intel/ivsc/mei_ace.c b/drivers/media/pci/intel/ivsc/mei_ace.c
new file mode 100644
index 000000000000..3dbd885d11a1
--- /dev/null
+++ b/drivers/media/pci/intel/ivsc/mei_ace.c
@@ -0,0 +1,472 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2023 Intel Corporation. All rights reserved.
+ * Intel Visual Sensing Controller ACE Linux driver
+ */
+
+/*
+ * To set ownership of camera sensor, there is specific command, which
+ * is sent via MEI protocol. That's a two-step scheme where the firmware
+ * first acks receipt of the command and later responses the command was
+ * executed. The command sending function uses "completion" as the
+ * synchronization mechanism. The notification for command is received
+ * via a mei callback which wakes up the caller. There can be only one
+ * outstanding command at a time.
+ */
+
+#include <linux/completion.h>
+#include <linux/delay.h>
+#include <linux/ivsc.h>
+#include <linux/kernel.h>
+#include <linux/mei_cl_bus.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/slab.h>
+#include <linux/uuid.h>
+
+#include "mei_ace.h"
+
+#define MEI_ACE_DRIVER_NAME "ivsc_ace"
+
+/* indicating driver message */
+#define ACE_DRV_MSG 1
+/* indicating set command */
+#define ACE_CMD_SET 4
+/* command timeout determined experimentally */
+#define ACE_CMD_TIMEOUT (5 * HZ)
+/* indicating the first command block */
+#define ACE_CMD_INIT_BLOCK 1
+/* indicating the last command block */
+#define ACE_CMD_FINAL_BLOCK 1
+/* size of camera status notification content */
+#define ACE_CAMERA_STATUS_SIZE 5
+
+/* UUID used to get firmware id */
+#define ACE_GET_FW_ID_UUID UUID_LE(0x6167DCFB, 0x72F1, 0x4584, 0xBF, \
+ 0xE3, 0x84, 0x17, 0x71, 0xAA, 0x79, 0x0B)
+
+/* identify firmware event type */
+enum ace_event_type {
+ /* firmware ready */
+ ACE_FW_READY = 0x8,
+
+ /* command response */
+ ACE_CMD_RESPONSE = 0x10,
+};
+
+/* identify the command id supported by firmware IPC */
+enum ace_cmd_id {
+ /* used to switch camera sensor to host */
+ ACE_SWITCH_CAMERA_TO_HOST = 0x13,
+
+ /* used to switch camera sensor to IVSC */
+ ACE_SWITCH_CAMERA_TO_IVSC = 0x14,
+
+ /* used to get firmware id */
+ ACE_GET_FW_ID = 0x1A,
+};
+
+/* ACE command header structure */
+struct ace_cmd_hdr {
+ u32 firmware_id : 16;
+ u32 instance_id : 8;
+ u32 type : 5;
+ u32 rsp : 1;
+ u32 msg_tgt : 1;
+ u32 _hw_rsvd_1 : 1;
+ u32 param_size : 20;
+ u32 cmd_id : 8;
+ u32 final_block : 1;
+ u32 init_block : 1;
+ u32 _hw_rsvd_2 : 2;
+} __packed;
+
+/* ACE command parameter structure */
+union ace_cmd_param {
+ uuid_le uuid;
+ u32 param;
+};
+
+/* ACE command structure */
+struct ace_cmd {
+ struct ace_cmd_hdr hdr;
+ union ace_cmd_param param;
+} __packed;
+
+/* ACE notification header */
+union ace_notif_hdr {
+ struct _confirm {
+ u32 status : 24;
+ u32 type : 5;
+ u32 rsp : 1;
+ u32 msg_tgt : 1;
+ u32 _hw_rsvd_1 : 1;
+ u32 param_size : 20;
+ u32 cmd_id : 8;
+ u32 final_block : 1;
+ u32 init_block : 1;
+ u32 _hw_rsvd_2 : 2;
+ } __packed ack;
+
+ struct _event {
+ u32 rsvd1 : 16;
+ u32 event_type : 8;
+ u32 type : 5;
+ u32 ack : 1;
+ u32 msg_tgt : 1;
+ u32 _hw_rsvd_1 : 1;
+ u32 rsvd2 : 30;
+ u32 _hw_rsvd_2 : 2;
+ } __packed event;
+
+ struct _response {
+ u32 event_id : 16;
+ u32 notif_type : 8;
+ u32 type : 5;
+ u32 rsp : 1;
+ u32 msg_tgt : 1;
+ u32 _hw_rsvd_1 : 1;
+ u32 event_data_size : 16;
+ u32 request_target : 1;
+ u32 request_type : 5;
+ u32 cmd_id : 8;
+ u32 _hw_rsvd_2 : 2;
+ } __packed response;
+};
+
+/* ACE notification content */
+union ace_notif_cont {
+ u16 firmware_id;
+ u8 state_notif;
+ u8 camera_status[ACE_CAMERA_STATUS_SIZE];
+};
+
+/* ACE notification structure */
+struct ace_notif {
+ union ace_notif_hdr hdr;
+ union ace_notif_cont cont;
+} __packed;
+
+struct mei_ace {
+ struct mei_cl_device *cldev;
+
+ /* command ack */
+ struct ace_notif cmd_ack;
+ /* command response */
+ struct ace_notif cmd_response;
+ /* used to wait for command ack and response */
+ struct completion cmd_completion;
+
+ /* used to construct command */
+ u16 firmware_id;
+};
+
+/* only one for now */
+static struct mei_ace *ace;
+/* lock used to prevent multiple call to ace */
+static DEFINE_MUTEX(ace_mutex);
+
+static inline void init_cmd_hdr(struct ace_cmd_hdr *hdr)
+{
+ memset(hdr, 0, sizeof(struct ace_cmd_hdr));
+
+ hdr->type = ACE_CMD_SET;
+ hdr->msg_tgt = ACE_DRV_MSG;
+ hdr->init_block = ACE_CMD_INIT_BLOCK;
+ hdr->final_block = ACE_CMD_FINAL_BLOCK;
+}
+
+static int construct_command(struct ace_cmd *cmd, enum ace_cmd_id cmd_id)
+{
+ union ace_cmd_param *param = &cmd->param;
+ struct ace_cmd_hdr *hdr = &cmd->hdr;
+
+ init_cmd_hdr(hdr);
+
+ hdr->cmd_id = cmd_id;
+ switch (cmd_id) {
+ case ACE_GET_FW_ID:
+ param->uuid = ACE_GET_FW_ID_UUID;
+ hdr->param_size = sizeof(param->uuid);
+ break;
+ case ACE_SWITCH_CAMERA_TO_IVSC:
+ param->param = 0;
+ hdr->firmware_id = ace->firmware_id;
+ hdr->param_size = sizeof(param->param);
+ break;
+ case ACE_SWITCH_CAMERA_TO_HOST:
+ hdr->firmware_id = ace->firmware_id;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return hdr->param_size + sizeof(cmd->hdr);
+}
+
+/* send a command to firmware and mutex must be held by caller */
+static int mei_ace_send(struct ace_cmd *cmd, size_t len, bool only_ack)
+{
+ union ace_notif_hdr *resp_hdr = &ace->cmd_response.hdr;
+ union ace_notif_hdr *ack_hdr = &ace->cmd_ack.hdr;
+ struct ace_cmd_hdr *cmd_hdr = &cmd->hdr;
+ int ret;
+
+ reinit_completion(&ace->cmd_completion);
+
+ ret = mei_cldev_send(ace->cldev, (u8 *)cmd, len);
+ if (ret < 0)
+ goto out;
+
+ ret = wait_for_completion_killable_timeout(&ace->cmd_completion,
+ ACE_CMD_TIMEOUT);
+ if (ret < 0) {
+ goto out;
+ } else if (!ret) {
+ ret = -ETIMEDOUT;
+ goto out;
+ }
+
+ if (ack_hdr->ack.cmd_id != cmd_hdr->cmd_id) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ /* command ack status */
+ ret = ack_hdr->ack.status;
+ if (ret) {
+ ret = -EIO;
+ goto out;
+ }
+
+ if (only_ack)
+ goto out;
+
+ ret = wait_for_completion_killable_timeout(&ace->cmd_completion,
+ ACE_CMD_TIMEOUT);
+ if (ret < 0) {
+ goto out;
+ } else if (!ret) {
+ ret = -ETIMEDOUT;
+ goto out;
+ }
+
+ if (resp_hdr->response.cmd_id != cmd_hdr->cmd_id)
+ ret = -EINVAL;
+
+out:
+ return ret;
+}
+
+int ace_set_camera_owner(enum ace_camera_owner owner)
+{
+ enum ace_cmd_id cmd_id;
+ struct ace_cmd cmd;
+ int cmd_size;
+ int ret;
+
+ if (owner == ACE_CAMERA_IVSC)
+ cmd_id = ACE_SWITCH_CAMERA_TO_IVSC;
+ else
+ cmd_id = ACE_SWITCH_CAMERA_TO_HOST;
+
+ mutex_lock(&ace_mutex);
+ if (unlikely(!ace)) {
+ mutex_unlock(&ace_mutex);
+ return -EAGAIN;
+ }
+
+ cmd_size = construct_command(&cmd, cmd_id);
+ if (cmd_size >= 0)
+ ret = mei_ace_send(&cmd, cmd_size, false);
+ else
+ ret = cmd_size;
+ mutex_unlock(&ace_mutex);
+
+ return ret;
+}
+EXPORT_SYMBOL_NS_GPL(ace_set_camera_owner, IVSC);
+
+/* the first command downloaded to firmware */
+static inline int ace_get_firmware_id(void)
+{
+ struct ace_cmd cmd;
+ int cmd_size;
+ int ret;
+
+ cmd_size = construct_command(&cmd, ACE_GET_FW_ID);
+ if (cmd_size >= 0)
+ ret = mei_ace_send(&cmd, cmd_size, true);
+ else
+ ret = cmd_size;
+
+ return ret;
+}
+
+static void handle_command_response(struct ace_notif *resp, int len)
+{
+ union ace_notif_hdr *hdr = &resp->hdr;
+
+ switch (hdr->response.cmd_id) {
+ case ACE_SWITCH_CAMERA_TO_IVSC:
+ case ACE_SWITCH_CAMERA_TO_HOST:
+ memcpy(&ace->cmd_response, resp, len);
+ complete(&ace->cmd_completion);
+ break;
+ case ACE_GET_FW_ID:
+ break;
+ default:
+ break;
+ }
+}
+
+static void handle_command_ack(struct ace_notif *ack, int len)
+{
+ union ace_notif_hdr *hdr = &ack->hdr;
+
+ switch (hdr->ack.cmd_id) {
+ case ACE_GET_FW_ID:
+ ace->firmware_id = ack->cont.firmware_id;
+ fallthrough;
+ case ACE_SWITCH_CAMERA_TO_IVSC:
+ case ACE_SWITCH_CAMERA_TO_HOST:
+ memcpy(&ace->cmd_ack, ack, len);
+ complete(&ace->cmd_completion);
+ break;
+ default:
+ break;
+ }
+}
+
+/* callback for receive */
+static void mei_ace_rx(struct mei_cl_device *cldev)
+{
+ struct ace_notif event;
+ union ace_notif_hdr *hdr = &event.hdr;
+ int ret;
+
+ ret = mei_cldev_recv(cldev, (u8 *)&event, sizeof(event));
+ if (ret < 0) {
+ dev_err(&cldev->dev, "recv error: %d\n", ret);
+ return;
+ }
+
+ if (hdr->event.ack) {
+ handle_command_ack(&event, ret);
+ return;
+ }
+
+ switch (hdr->event.event_type) {
+ case ACE_CMD_RESPONSE:
+ handle_command_response(&event, ret);
+ break;
+ case ACE_FW_READY:
+ /*
+ * firmware ready notification sent to driver
+ * after HECI client connected with firmware.
+ */
+ dev_dbg(&cldev->dev, "firmware ready\n");
+ break;
+ default:
+ break;
+ }
+}
+
+static int mei_ace_probe(struct mei_cl_device *cldev,
+ const struct mei_cl_device_id *id)
+{
+ int ret;
+
+ mutex_lock(&ace_mutex);
+ /*
+ * only instance is possible in the current hardware,
+ * but adding protection for future hardware.
+ */
+ if (ace) {
+ ret = -EBUSY;
+ goto err_unlock;
+ }
+
+ ace = kzalloc(sizeof(struct mei_ace), GFP_KERNEL);
+ if (!ace) {
+ ret = -ENOMEM;
+ goto err_unlock;
+ }
+
+ ace->cldev = cldev;
+ init_completion(&ace->cmd_completion);
+
+ mei_cldev_set_drvdata(cldev, ace);
+
+ ret = mei_cldev_enable(cldev);
+ if (ret < 0) {
+ dev_err(&cldev->dev, "mei_cldev_enable failed: %d\n", ret);
+ goto err_free;
+ }
+
+ ret = mei_cldev_register_rx_cb(cldev, mei_ace_rx);
+ if (ret) {
+ dev_err(&cldev->dev, "event cb registration failed: %d\n", ret);
+ goto err_disable;
+ }
+
+ ret = ace_get_firmware_id();
+ if (ret) {
+ dev_err(&cldev->dev, "get firmware id failed: %d\n", ret);
+ goto err_disable;
+ }
+ mutex_unlock(&ace_mutex);
+
+ return 0;
+
+err_disable:
+ mei_cldev_disable(cldev);
+
+err_free:
+ kfree(ace);
+ /* disable ace */
+ ace = NULL;
+
+err_unlock:
+ mutex_unlock(&ace_mutex);
+
+ return ret;
+}
+
+static void mei_ace_remove(struct mei_cl_device *cldev)
+{
+ mutex_lock(&ace_mutex);
+ /* disable mei ace client device */
+ mei_cldev_disable(cldev);
+
+ kfree(ace);
+ /* disable ace */
+ ace = NULL;
+ mutex_unlock(&ace_mutex);
+}
+
+#define MEI_ACE_UUID UUID_LE(0x5DB76CF6, 0x0A68, 0x4ED6, \
+ 0x9B, 0x78, 0x03, 0x61, 0x63, 0x5E, 0x24, 0x47)
+
+static const struct mei_cl_device_id mei_ace_tbl[] = {
+ { MEI_ACE_DRIVER_NAME, MEI_ACE_UUID, MEI_CL_VERSION_ANY },
+
+ /* required last entry */
+ { }
+};
+MODULE_DEVICE_TABLE(mei, mei_ace_tbl);
+
+static struct mei_cl_driver mei_ace_driver = {
+ .id_table = mei_ace_tbl,
+ .name = MEI_ACE_DRIVER_NAME,
+
+ .probe = mei_ace_probe,
+ .remove = mei_ace_remove,
+};
+
+module_mei_cl_driver(mei_ace_driver);
+
+MODULE_AUTHOR("Wentong Wu <[email protected]>");
+MODULE_AUTHOR("Zhifeng Wang <[email protected]>");
+MODULE_DESCRIPTION("Device driver for IVSC ACE");
+MODULE_LICENSE("GPL");
diff --git a/drivers/media/pci/intel/ivsc/mei_ace.h b/drivers/media/pci/intel/ivsc/mei_ace.h
new file mode 100644
index 000000000000..0302bc899521
--- /dev/null
+++ b/drivers/media/pci/intel/ivsc/mei_ace.h
@@ -0,0 +1,36 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2023 Intel Corporation. All rights reserved.
+ * Intel Visual Sensing Controller ACE external interface
+ */
+
+#ifndef _MEI_ACE_H_
+#define _MEI_ACE_H_
+
+#include <linux/types.h>
+
+/* identify camera sensor ownership */
+enum ace_camera_owner {
+ ACE_CAMERA_IVSC,
+ ACE_CAMERA_HOST,
+};
+
+#if IS_ENABLED(CONFIG_INTEL_VSC)
+/*
+ * @brief set camera sensor ownership
+ *
+ * @param owner Camera sensor ownership being set
+ *
+ * @return 0 on success, negative on failure
+ */
+int ace_set_camera_owner(enum ace_camera_owner owner);
+
+#else
+static inline int ace_set_camera_owner(enum ace_camera_owner owner)
+{
+ return 0;
+}
+
+#endif
+
+#endif
--
2.25.1
IVSC directly connects to camera sensor on source side, and on
output side it not only connects ISH via I2C, but also exposes
MIPI CSI-2 interface to output camera sensor data. IVSC can use
the camera sensor data to do AI algorithm, and send the results
to ISH. On the other end, IVSC can share camera sensor to host
by routing the raw camera sensor data to the exposed MIPI CSI-2
interface. But they can not work at the same time, so software
APIs are defined to sync the ownership.
This commit defines the interfaces between IVSC and camera sensor
driver in include/linux/ivsc.h. The camera driver controls
ownership of the CSI-2 link and sensor with the acquire/release
APIs. When acquiring camera, lane number and link freq are also
required by IVSC frame router.
Signed-off-by: Wentong Wu <[email protected]>
---
drivers/media/pci/intel/ivsc/Makefile | 1 +
drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++++++++++++++++++++++++
include/linux/ivsc.h | 55 ++++++++++++++++++
3 files changed, 140 insertions(+)
create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
diff --git a/drivers/media/pci/intel/ivsc/Makefile b/drivers/media/pci/intel/ivsc/Makefile
index de0a425c22c2..b8b6fc1083be 100644
--- a/drivers/media/pci/intel/ivsc/Makefile
+++ b/drivers/media/pci/intel/ivsc/Makefile
@@ -4,3 +4,4 @@
obj-$(CONFIG_INTEL_VSC) += mei_csi.o
obj-$(CONFIG_INTEL_VSC) += mei_ace.o
+obj-$(CONFIG_INTEL_VSC) += ivsc.o
diff --git a/drivers/media/pci/intel/ivsc/ivsc.c b/drivers/media/pci/intel/ivsc/ivsc.c
new file mode 100644
index 000000000000..12996b587639
--- /dev/null
+++ b/drivers/media/pci/intel/ivsc/ivsc.c
@@ -0,0 +1,84 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2023 Intel Corporation. All rights reserved.
+ * Intel Visual Sensing Controller interface
+ */
+
+#include <linux/delay.h>
+#include <linux/ivsc.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+
+#include "mei_ace.h"
+#include "mei_csi.h"
+
+/* lock for ivsc APIs */
+static DEFINE_MUTEX(ivsc_mutex);
+
+int ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
+ void (*callback)(void *, enum ivsc_privacy_status),
+ void *context)
+{
+ int ret;
+
+ mutex_lock(&ivsc_mutex);
+
+ /* switch camera sensor ownership to host */
+ ret = ace_set_camera_owner(ACE_CAMERA_HOST);
+ if (ret)
+ goto error;
+
+ /* switch CSI-2 link to host */
+ ret = csi_set_link_owner(CSI_LINK_HOST, callback, context);
+ if (ret)
+ goto release_camera;
+
+ /* configure CSI-2 link */
+ ret = csi_set_link_cfg(nr_of_lanes, link_freq);
+ if (ret)
+ goto release_csi;
+
+ mutex_unlock(&ivsc_mutex);
+
+ return 0;
+
+release_csi:
+ csi_set_link_owner(CSI_LINK_IVSC, NULL, NULL);
+
+release_camera:
+ ace_set_camera_owner(ACE_CAMERA_IVSC);
+
+error:
+ mutex_unlock(&ivsc_mutex);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(ivsc_acquire_camera);
+
+int ivsc_release_camera(void)
+{
+ int ret;
+
+ mutex_lock(&ivsc_mutex);
+
+ /* switch CSI-2 link to IVSC */
+ ret = csi_set_link_owner(CSI_LINK_IVSC, NULL, NULL);
+ if (ret)
+ goto error;
+
+ /* switch camera sensor ownership to IVSC */
+ ret = ace_set_camera_owner(ACE_CAMERA_IVSC);
+
+error:
+ mutex_unlock(&ivsc_mutex);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(ivsc_release_camera);
+
+MODULE_AUTHOR("Wentong Wu <[email protected]>");
+MODULE_AUTHOR("Zhifeng Wang <[email protected]>");
+MODULE_SOFTDEP("pre: mei_csi mei_ace");
+MODULE_DESCRIPTION("IVSC interface");
+MODULE_LICENSE("GPL");
+MODULE_IMPORT_NS(IVSC);
diff --git a/include/linux/ivsc.h b/include/linux/ivsc.h
index 6572ca4f340c..bc9006cd6efc 100644
--- a/include/linux/ivsc.h
+++ b/include/linux/ivsc.h
@@ -16,4 +16,59 @@ enum ivsc_privacy_status {
IVSC_PRIVACY_MAX,
};
+#if IS_ENABLED(CONFIG_INTEL_VSC)
+/*
+ * @brief Acquire camera sensor ownership to host and setup
+ * the CSI-2 link between host and IVSC
+ *
+ * IVSC provides a privacy mode. When the privacy mode is turned
+ * on, camera sensor can't be used. This means that both IVSC and
+ * host Image Processing Unit(IPU) can't get image data. And when
+ * this mode is turned on, host Image Processing Unit(IPU) driver
+ * is informed via the registered callback, so that user can be
+ * notified.
+ *
+ * @param nr_of_lanes Number of data lanes used on the CSI-2 link
+ * @param link_freq Frequency of the CSI-2 link
+ * @param callback The pointer of privacy callback function
+ * @param context Privacy callback function runtime context
+ *
+ * @retval 0 If success
+ * @retval -EIO IO error
+ * @retval -EINVAL Invalid argument
+ * @retval -EAGAIN IVSC device not ready
+ * @retval negative values for other errors
+ */
+int ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
+ void (*callback)(void *, enum ivsc_privacy_status),
+ void *context);
+
+/*
+ * @brief Release camera sensor ownership and stop the CSI-2
+ * link between host and IVSC
+ *
+ * @retval 0 If success
+ * @retval -EIO IO error
+ * @retval -EINVAL Invalid argument
+ * @retval -EAGAIN IVSC device not ready
+ * @retval negative values for other errors
+ */
+int ivsc_release_camera(void);
+
+#else
+static inline
+int ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
+ void (*callback)(void *, enum ivsc_privacy_status),
+ void *context)
+{
+ return 0;
+}
+
+static inline int ivsc_release_camera(void)
+{
+ return 0;
+}
+
+#endif
+
#endif
--
2.25.1
On Mon, 2023-02-13 at 10:23 +0800, Wentong Wu wrote:
>
v2 without change log, why?
> Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is
> a
> companion chip designed to provide secure and low power vision
> capability
> to IA platforms. IVSC is available in existing commercial platforms
> from
> multiple OEMs.
>
> The primary use case of IVSC is to bring in context awareness. IVSC
> interfaces directly with the platform main camera sensor via a CSI-2
> link
> and processes the image data with the embedded AI engine. The
> detected
> events are sent over I2C to ISH (Intel Sensor Hub) for additional
> data
> fusion from multiple sensors. The fusion results are used to
> implement
> advanced use cases like:
> - Face detection to unlock screen
> - Detect user presence to manage backlight setting or waking up
> system
>
> Since the Image Processing Unit(IPU) used on the host processor needs
> to
> configure the CSI-2 link in normal camera usages, the CSI-2 link and
> camera sensor can only be used in mutually-exclusive ways by host IPU
> and
> IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The
> IPU
> driver can take ownership of the CSI-2 link and camera sensor using
> interfaces provided by this IVSC driver.
>
> Switching ownership requires an interface with two different hardware
> modules inside IVSC. The software interface to these modules is via
> Intel
> MEI (The Intel Management Engine) commands. These two hardware
> modules
> have two different MEI UUIDs to enumerate. These hardware modules
> are:
> - ACE (Algorithm Context Engine): This module is for algorithm
> computing
> when IVSC owns camera sensor. Also ACE module controls camera
> sensor's
> ownership. This hardware module is used to set ownership of camera
> sensor.
> - CSI (Camera Serial Interface): This module is used to route camera
> sensor data either to IVSC or to host for IPU driver and application.
>
> IVSC also provides a privacy mode. When privacy mode is turned on,
> camera sensor can't be used. This means that both ACE and host IPU
> can't
> get image data. And when this mode is turned on, host IPU driver is
> informed via a registered callback, so that user can be notified.
>
> In summary, to acquire ownership of camera by IPU driver, first ACE
> module needs to be informed of ownership and then to setup MIPI CSI-2
> link for the camera sensor and IPU.
>
> Implementation:
> There are two different drivers to handle ACE and CSI hardware
> modules
> inside IVSC.
> - mei_csi: MEI client driver to send commands and receive
> notifications
> from CSI module.
> - mei_ace: MEI client driver to send commands and get status from
> ACE
> module.
> Interface is exposed via ivsc.h to acquire and release camera sensor
> and
> CSI-2 link.
>
> Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
> ---------------------------------------------------------------------
> --------
> > Host
> > Processor
> > |
> >
> > |
> > ----------------- ----------------- -------------
> > -- |
> > | | | | |
> > | I2C |
> > | IPU | | ISH | |camera
> > driver|--| |
> > | | | | |
> > | | |
> > ----------------- ----------------- -------------
> > -- | |
> > | |
> > | | |
> > | | -------------
> > -- | |
> > | | |
> > | | |
> > | | | IVSC driver
> > | | |
> > | | |
> > | | |
> > | | -------------
> > -- | |
> > | |
> > | | |
> ----------------|-----------------------|----------------------|-----
> ----|---
> | CSI | I2C
> |SPI |
> | |
> | |
> ----------------|-----------------------|----------------
> | |
> > IVSC | |
> > | |
> > | |
> > | |
> > ----------------- ----------------- |
> > | |
> > | | | | |
> > | |
> > | CSI | | ACE | |------
> > | |
> > | | | |
> > | |
> > ----------------- -----------------
> > | |
> > | | I2C
> > | |
> ----------------|-----------------------|----------------
> |
> | CSI
> | |
> |
> | |
> --------------------------------
> |
> | |
> I2C |
> | camera sensor |-------------------------
> ----|
> | |
> --------------------------------
>
> Wentong Wu (3):
> media: pci: intel: ivsc: Add CSI submodule
> media: pci: intel: ivsc: Add ACE submodule
> media: pci: intel: ivsc: Add acquire/release API for ivsc
>
> drivers/media/pci/Kconfig | 1 +
> drivers/media/pci/intel/Makefile | 2 +
> drivers/media/pci/intel/ivsc/Kconfig | 12 +
> drivers/media/pci/intel/ivsc/Makefile | 7 +
> drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++
> drivers/media/pci/intel/ivsc/mei_ace.c | 472
> +++++++++++++++++++++++++
> drivers/media/pci/intel/ivsc/mei_ace.h | 36 ++
> drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++
> drivers/media/pci/intel/ivsc/mei_csi.h | 60 ++++
> include/linux/ivsc.h | 74 ++++
> 10 files changed, 1090 insertions(+)
> create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
> create mode 100644 drivers/media/pci/intel/ivsc/Makefile
> create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
> create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
> create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
> create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
> create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
> create mode 100644 include/linux/ivsc.h
>
Hi Hillf Danton,
Thanks for your review
> -----Original Message-----
> From: Hillf Danton <[email protected]>
> Sent: Monday, February 13, 2023 11:42 AM
>
> On Mon, 13 Feb 2023 10:23:45 +0800 Wentong Wu <[email protected]>
> > +
> > +/* send a command to firmware and mutex must be held by caller */
> > +static int mei_csi_send(u8 *buf, size_t len) {
> > + struct csi_cmd *cmd = (struct csi_cmd *)buf;
> > + int ret;
> > +
> > + reinit_completion(&csi->cmd_completion);
>
> Could you specify why reinit is needed here?
This allows new command(e.g. 'get status' command to check current firmware status) to be downloaded to firmware for debugging purpose in case no response for current command though I have never saw this happen.
> What is hurt without it?
No hurt for current implementation. I can remove it.
>
> Same question for the reinit in 2/3.
same ack for 2/3
>
> > +
> > + ret = mei_cldev_send(csi->cldev, buf, len);
> > + if (ret < 0)
> > + goto out;
> > +
> > + ret = wait_for_completion_killable_timeout(&csi->cmd_completion,
> > + CSI_CMD_TIMEOUT);
> > + if (ret < 0) {
> > + goto out;
> > + } else if (!ret) {
> > + ret = -ETIMEDOUT;
> > + goto out;
> > + }
Hi Srinivas,
Thanks for your review
> -----Original Message-----
> From: Pandruvada, Srinivas <[email protected]>
> Sent: Monday, February 13, 2023 11:21 AM
>
> On Mon, 2023-02-13 at 10:23 +0800, Wentong Wu wrote:
> >
>
> v2 without change log, why?
This addressed 'mei_csi.c:(.text+0x1a6): undefined reference to __udivdi3' issue on i386 arch. I will add change log on following version.
>
> > Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a
> > companion chip designed to provide secure and low power vision
> > capability to IA platforms. IVSC is available in existing commercial
> > platforms from multiple OEMs.
> >
> > The primary use case of IVSC is to bring in context awareness. IVSC
> > interfaces directly with the platform main camera sensor via a CSI-2
> > link and processes the image data with the embedded AI engine. The
> > detected events are sent over I2C to ISH (Intel Sensor Hub) for
> > additional data fusion from multiple sensors. The fusion results are
> > used to implement advanced use cases like:
> > - Face detection to unlock screen
> > - Detect user presence to manage backlight setting or waking up
> > system
> >
> > Since the Image Processing Unit(IPU) used on the host processor needs
> > to configure the CSI-2 link in normal camera usages, the CSI-2 link
> > and camera sensor can only be used in mutually-exclusive ways by host
> > IPU and IVSC. By default the IVSC owns the CSI-2 link and camera
> > sensor. The IPU driver can take ownership of the CSI-2 link and camera
> > sensor using interfaces provided by this IVSC driver.
> >
> > Switching ownership requires an interface with two different hardware
> > modules inside IVSC. The software interface to these modules is via
> > Intel MEI (The Intel Management Engine) commands. These two hardware
> > modules have two different MEI UUIDs to enumerate. These hardware
> > modules
> > are:
> > - ACE (Algorithm Context Engine): This module is for algorithm
> > computing when IVSC owns camera sensor. Also ACE module controls
> > camera sensor's ownership. This hardware module is used to set
> > ownership of camera sensor.
> > - CSI (Camera Serial Interface): This module is used to route camera
> > sensor data either to IVSC or to host for IPU driver and application.
> >
> > IVSC also provides a privacy mode. When privacy mode is turned on,
> > camera sensor can't be used. This means that both ACE and host IPU
> > can't get image data. And when this mode is turned on, host IPU driver
> > is informed via a registered callback, so that user can be notified.
> >
> > In summary, to acquire ownership of camera by IPU driver, first ACE
> > module needs to be informed of ownership and then to setup MIPI CSI-2
> > link for the camera sensor and IPU.
> >
> > Implementation:
> > There are two different drivers to handle ACE and CSI hardware modules
> > inside IVSC.
> > - mei_csi: MEI client driver to send commands and receive
> > notifications from CSI module.
> > - mei_ace: MEI client driver to send commands and get status from ACE
> > module.
> > Interface is exposed via ivsc.h to acquire and release camera sensor
> > and
> > CSI-2 link.
> >
> > Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
> > ---------------------------------------------------------------------
> > --------
> > > Host
> > > Processor
> > > |
> > >
> > > |
> > > ----------------- ----------------- -------------
> > > -- |
> > > | | | | |
> > > | I2C |
> > > | IPU | | ISH | |camera
> > > driver|--| |
> > > | | | | |
> > > | | |
> > > ----------------- ----------------- -------------
> > > -- | |
> > > | |
> > > | | |
> > > | | -------------
> > > -- | |
> > > | | |
> > > | | |
> > > | | | IVSC driver
> > > | | |
> > > | | |
> > > | | |
> > > | | -------------
> > > -- | |
> > > | |
> > > | | |
> > ----------------|-----------------------|----------------------|-----
> > ----|---
> > | CSI | I2C
> > |SPI |
> > | |
> > | |
> > ----------------|-----------------------|----------------
> > | |
> > > IVSC | |
> > > | |
> > > | |
> > > | |
> > > ----------------- ----------------- |
> > > | |
> > > | | | | |
> > > | |
> > > | CSI | | ACE | |------
> > > | |
> > > | | | |
> > > | |
> > > ----------------- -----------------
> > > | |
> > > | | I2C
> > > | |
> > ----------------|-----------------------|----------------
> > |
> > | CSI
> > | |
> > |
> > | |
> > --------------------------------
> > |
> > | | I2C
> > |
> > | camera sensor |-------------------------
> > ----|
> > | |
> > --------------------------------
> >
> > Wentong Wu (3):
> > media: pci: intel: ivsc: Add CSI submodule
> > media: pci: intel: ivsc: Add ACE submodule
> > media: pci: intel: ivsc: Add acquire/release API for ivsc
> >
> > drivers/media/pci/Kconfig | 1 +
> > drivers/media/pci/intel/Makefile | 2 +
> > drivers/media/pci/intel/ivsc/Kconfig | 12 +
> > drivers/media/pci/intel/ivsc/Makefile | 7 +
> > drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++
> > drivers/media/pci/intel/ivsc/mei_ace.c | 472
> > +++++++++++++++++++++++++
> > drivers/media/pci/intel/ivsc/mei_ace.h | 36 ++
> > drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++
> > drivers/media/pci/intel/ivsc/mei_csi.h | 60 ++++
> > include/linux/ivsc.h | 74 ++++
> > 10 files changed, 1090 insertions(+)
> > create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
> > create mode 100644 drivers/media/pci/intel/ivsc/Makefile
> > create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
> > create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
> > create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
> > create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
> > create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
> > create mode 100644 include/linux/ivsc.h
> >
> -----Original Message-----
> From: Hillf Danton <[email protected]>
> Sent: Tuesday, February 14, 2023 4:36 PM
>
> On Tue, 14 Feb 2023 06:25:35 +0000 Wentong Wu <[email protected]>
> >
> > This allows new command(e.g. 'get status' command to check current
> > firmware=
> > status) to be downloaded to firmware for debugging purpose in case no
> > resp= onse for current command though I have never saw this happen.
> > =20
>
> Thanks for your specification.
>
> Another question, is it likely for the wakeup to come after reinit because of for
> instance timeout on the waiter side?
Thanks, first I want to clarify why we have timeout here, in case firmware stuck somehow or IVSC device is removed suddenly, calling thread can't be wakeup if wait forever here.
And firmware gets high priority to handle command and response, also there is timeout mechanism inside firmware, I think there will be some problem with firmware if driver gets command response after 5s which will be captured by driver code(if (csi->cmd_response.cmd_id != cmd->cmd_id)) but I have never saw this happen.
BR,
Wentong
Hi Wentong,
Thanks for the patchset.
On Mon, Feb 13, 2023 at 10:23:47AM +0800, Wentong Wu wrote:
> IVSC directly connects to camera sensor on source side, and on
> output side it not only connects ISH via I2C, but also exposes
> MIPI CSI-2 interface to output camera sensor data. IVSC can use
> the camera sensor data to do AI algorithm, and send the results
> to ISH. On the other end, IVSC can share camera sensor to host
> by routing the raw camera sensor data to the exposed MIPI CSI-2
> interface. But they can not work at the same time, so software
> APIs are defined to sync the ownership.
>
> This commit defines the interfaces between IVSC and camera sensor
> driver in include/linux/ivsc.h. The camera driver controls
> ownership of the CSI-2 link and sensor with the acquire/release
> APIs. When acquiring camera, lane number and link freq are also
> required by IVSC frame router.
The more I learn about this system, the more I'm inclined to think this
functionality should be exposed as a V4L2 sub-device. IVSC doesn't really
do anything to the data (as long as it directs it towards the CSI-2
receiver in the SoC), but it is definitely part of the image pipeline.
I suppose the intended use cases assume a single instance of IVSC (as well
as MEI) but there can, and often are, be multiple camera sensors in the
system. The decision whether to request pass-through from IVCS can't be
done in the camera sensor driver, and should not be visible to the camera
sensor driver. Exposing IVSC as a V4L2 sub-device makes this trivial to
address, as the IVSC driver's V4L2 sub-device video s_stream() operation
gets called before streaming is started.
The information whether IVSC is found between the camera sensor and the
host's CSI-2 receiver (IPU in this case) should come from system firmware
and accessed most probably by what is called cio2-bridge at the moment.
The privacy status can be a V4L2 control.
Also cc Hans.
>
> Signed-off-by: Wentong Wu <[email protected]>
> ---
> drivers/media/pci/intel/ivsc/Makefile | 1 +
> drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++++++++++++++++++++++++
> include/linux/ivsc.h | 55 ++++++++++++++++++
> 3 files changed, 140 insertions(+)
> create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
>
> diff --git a/drivers/media/pci/intel/ivsc/Makefile b/drivers/media/pci/intel/ivsc/Makefile
> index de0a425c22c2..b8b6fc1083be 100644
> --- a/drivers/media/pci/intel/ivsc/Makefile
> +++ b/drivers/media/pci/intel/ivsc/Makefile
> @@ -4,3 +4,4 @@
>
> obj-$(CONFIG_INTEL_VSC) += mei_csi.o
> obj-$(CONFIG_INTEL_VSC) += mei_ace.o
> +obj-$(CONFIG_INTEL_VSC) += ivsc.o
> diff --git a/drivers/media/pci/intel/ivsc/ivsc.c b/drivers/media/pci/intel/ivsc/ivsc.c
> new file mode 100644
> index 000000000000..12996b587639
> --- /dev/null
> +++ b/drivers/media/pci/intel/ivsc/ivsc.c
> @@ -0,0 +1,84 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * Copyright (C) 2023 Intel Corporation. All rights reserved.
> + * Intel Visual Sensing Controller interface
> + */
> +
> +#include <linux/delay.h>
> +#include <linux/ivsc.h>
> +#include <linux/module.h>
> +#include <linux/mutex.h>
> +
> +#include "mei_ace.h"
> +#include "mei_csi.h"
> +
> +/* lock for ivsc APIs */
> +static DEFINE_MUTEX(ivsc_mutex);
> +
> +int ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
> + void (*callback)(void *, enum ivsc_privacy_status),
> + void *context)
> +{
> + int ret;
> +
> + mutex_lock(&ivsc_mutex);
> +
> + /* switch camera sensor ownership to host */
> + ret = ace_set_camera_owner(ACE_CAMERA_HOST);
> + if (ret)
> + goto error;
> +
> + /* switch CSI-2 link to host */
> + ret = csi_set_link_owner(CSI_LINK_HOST, callback, context);
> + if (ret)
> + goto release_camera;
> +
> + /* configure CSI-2 link */
> + ret = csi_set_link_cfg(nr_of_lanes, link_freq);
> + if (ret)
> + goto release_csi;
> +
> + mutex_unlock(&ivsc_mutex);
> +
> + return 0;
> +
> +release_csi:
> + csi_set_link_owner(CSI_LINK_IVSC, NULL, NULL);
> +
> +release_camera:
> + ace_set_camera_owner(ACE_CAMERA_IVSC);
> +
> +error:
> + mutex_unlock(&ivsc_mutex);
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(ivsc_acquire_camera);
> +
> +int ivsc_release_camera(void)
> +{
> + int ret;
> +
> + mutex_lock(&ivsc_mutex);
> +
> + /* switch CSI-2 link to IVSC */
> + ret = csi_set_link_owner(CSI_LINK_IVSC, NULL, NULL);
> + if (ret)
> + goto error;
> +
> + /* switch camera sensor ownership to IVSC */
> + ret = ace_set_camera_owner(ACE_CAMERA_IVSC);
> +
> +error:
> + mutex_unlock(&ivsc_mutex);
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(ivsc_release_camera);
> +
> +MODULE_AUTHOR("Wentong Wu <[email protected]>");
> +MODULE_AUTHOR("Zhifeng Wang <[email protected]>");
> +MODULE_SOFTDEP("pre: mei_csi mei_ace");
> +MODULE_DESCRIPTION("IVSC interface");
> +MODULE_LICENSE("GPL");
> +MODULE_IMPORT_NS(IVSC);
> diff --git a/include/linux/ivsc.h b/include/linux/ivsc.h
> index 6572ca4f340c..bc9006cd6efc 100644
> --- a/include/linux/ivsc.h
> +++ b/include/linux/ivsc.h
> @@ -16,4 +16,59 @@ enum ivsc_privacy_status {
> IVSC_PRIVACY_MAX,
> };
>
> +#if IS_ENABLED(CONFIG_INTEL_VSC)
> +/*
> + * @brief Acquire camera sensor ownership to host and setup
> + * the CSI-2 link between host and IVSC
> + *
> + * IVSC provides a privacy mode. When the privacy mode is turned
> + * on, camera sensor can't be used. This means that both IVSC and
> + * host Image Processing Unit(IPU) can't get image data. And when
> + * this mode is turned on, host Image Processing Unit(IPU) driver
> + * is informed via the registered callback, so that user can be
> + * notified.
> + *
> + * @param nr_of_lanes Number of data lanes used on the CSI-2 link
> + * @param link_freq Frequency of the CSI-2 link
> + * @param callback The pointer of privacy callback function
> + * @param context Privacy callback function runtime context
> + *
> + * @retval 0 If success
> + * @retval -EIO IO error
> + * @retval -EINVAL Invalid argument
> + * @retval -EAGAIN IVSC device not ready
> + * @retval negative values for other errors
> + */
> +int ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
> + void (*callback)(void *, enum ivsc_privacy_status),
> + void *context);
> +
> +/*
> + * @brief Release camera sensor ownership and stop the CSI-2
> + * link between host and IVSC
> + *
> + * @retval 0 If success
> + * @retval -EIO IO error
> + * @retval -EINVAL Invalid argument
> + * @retval -EAGAIN IVSC device not ready
> + * @retval negative values for other errors
> + */
> +int ivsc_release_camera(void);
> +
> +#else
> +static inline
> +int ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
> + void (*callback)(void *, enum ivsc_privacy_status),
> + void *context)
> +{
> + return 0;
> +}
> +
> +static inline int ivsc_release_camera(void)
> +{
> + return 0;
> +}
> +
> +#endif
> +
> #endif
--
Kind regards,
Sakari Ailus
Hi Wentong,
Cc'ing Hans as well.
On Mon, Feb 13, 2023 at 10:23:45AM +0800, Wentong Wu wrote:
> CSI is a submodule of IVSC which can route camera sensor data
> to the outbound MIPI CSI-2 interface.
>
> The interface communicating with firmware is via MEI. There is
> a separate MEI UUID, which this driver uses to enumerate.
>
> To route camera sensor data to host, the caller specifies link
> frequency and number of data lanes. This information is sent to
> firmware by sending MEI command.
>
> Two APIs are exported: csi_set_link_owner is used to switch
> ownership of CSI-2 link, and csi_set_link_cfg is to configure
> CSI-2 link when routing camera sensor data to host.
>
> CSI also provides a privacy mode. When privacy mode is turned
> on, camera sensor can't be used. This means that both IVSC and
> host Image Processing Unit(IPU) can't get image data. And when
> this mode is turned on, host Image Processing Unit(IPU) driver
> is informed via the registered callback, so that user can be
> notified.
>
> Signed-off-by: Wentong Wu <[email protected]>
> ---
> drivers/media/pci/Kconfig | 1 +
> drivers/media/pci/intel/Makefile | 2 +
> drivers/media/pci/intel/ivsc/Kconfig | 12 +
> drivers/media/pci/intel/ivsc/Makefile | 5 +
> drivers/media/pci/intel/ivsc/mei_csi.c | 342 +++++++++++++++++++++++++
> drivers/media/pci/intel/ivsc/mei_csi.h | 60 +++++
> include/linux/ivsc.h | 19 ++
Please add a MAINTAINERS entry for this (ivsc dir).
> 7 files changed, 441 insertions(+)
> create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
> create mode 100644 drivers/media/pci/intel/ivsc/Makefile
> create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
> create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
> create mode 100644 include/linux/ivsc.h
>
> diff --git a/drivers/media/pci/Kconfig b/drivers/media/pci/Kconfig
> index 1224d908713a..16661ab0b816 100644
> --- a/drivers/media/pci/Kconfig
> +++ b/drivers/media/pci/Kconfig
> @@ -74,6 +74,7 @@ config VIDEO_PCI_SKELETON
> when developing new drivers.
>
> source "drivers/media/pci/intel/ipu3/Kconfig"
> +source "drivers/media/pci/intel/ivsc/Kconfig"
>
> endif #MEDIA_PCI_SUPPORT
> endif #PCI
> diff --git a/drivers/media/pci/intel/Makefile b/drivers/media/pci/intel/Makefile
> index 0b4236c4db49..d27ca636c860 100644
> --- a/drivers/media/pci/intel/Makefile
> +++ b/drivers/media/pci/intel/Makefile
> @@ -4,3 +4,5 @@
> #
>
> obj-y += ipu3/
> +
> +obj-$(CONFIG_INTEL_VSC) += ivsc/
> diff --git a/drivers/media/pci/intel/ivsc/Kconfig b/drivers/media/pci/intel/ivsc/Kconfig
> new file mode 100644
> index 000000000000..9535ac10f4f7
> --- /dev/null
> +++ b/drivers/media/pci/intel/ivsc/Kconfig
> @@ -0,0 +1,12 @@
> +# SPDX-License-Identifier: GPL-2.0-only
> +# Copyright (C) 2023, Intel Corporation. All rights reserved.
> +
> +config INTEL_VSC
> + tristate "Intel Visual Sensing Controller"
> + depends on INTEL_MEI
> + help
> + This adds support for Intel Visual Sensing Controller (IVSC).
> +
> + Enables the IVSC firmware services required for controlling
> + camera sensor ownership and CSI-2 link through Image Processing
> + Unit(IPU) driver of Intel.
> diff --git a/drivers/media/pci/intel/ivsc/Makefile b/drivers/media/pci/intel/ivsc/Makefile
> new file mode 100644
> index 000000000000..1825aad45cff
> --- /dev/null
> +++ b/drivers/media/pci/intel/ivsc/Makefile
> @@ -0,0 +1,5 @@
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +# Copyright (C) 2023, Intel Corporation. All rights reserved.
> +
> +obj-$(CONFIG_INTEL_VSC) += mei_csi.o
> diff --git a/drivers/media/pci/intel/ivsc/mei_csi.c b/drivers/media/pci/intel/ivsc/mei_csi.c
> new file mode 100644
> index 000000000000..309e52387502
> --- /dev/null
> +++ b/drivers/media/pci/intel/ivsc/mei_csi.c
> @@ -0,0 +1,342 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * Copyright (C) 2023 Intel Corporation. All rights reserved.
Do you still need the "all rights reserved" part?
> + * Intel Visual Sensing Controller CSI Linux driver
> + */
> +
> +/*
You can merge the two comments.
> + * To set ownership of CSI-2 link and to configure CSI-2 link, there
> + * are specific commands, which are sent via MEI protocol. The send
> + * command function uses "completion" as a synchronization mechanism.
> + * The response for command is received via a mei callback which wakes
> + * up the caller. There can be only one outstanding command at a time.
> + */
> +
> +#include <linux/completion.h>
> +#include <linux/delay.h>
> +#include <linux/kernel.h>
> +#include <linux/math64.h>
> +#include <linux/mei_cl_bus.h>
> +#include <linux/module.h>
> +#include <linux/mutex.h>
> +#include <linux/slab.h>
> +#include <linux/units.h>
> +#include <linux/uuid.h>
> +#include <linux/workqueue.h>
> +
> +#include "mei_csi.h"
> +
> +#define MEI_CSI_DRIVER_NAME "ivsc_csi"
> +
> +/* the 5s used here is based on experiment */
> +#define CSI_CMD_TIMEOUT (5 * HZ)
> +/* to setup CSI-2 link an extra delay needed and determined experimentally */
> +#define CSI_FW_READY_DELAY_MS 100
> +/* link frequency unit is 100kHz */
> +#define CSI_LINK_FREQ(x) ((u32)(div_u64(x, 100 * HZ_PER_KHZ)))
> +
> +/*
> + * identify the command id supported by firmware
> + * IPC, as well as the privacy notification id
> + * used when processing privacy event.
> + */
> +enum csi_cmd_id {
> + /* used to set csi ownership */
> + CSI_SET_OWNER = 0,
> +
> + /* used to configure CSI-2 link */
> + CSI_SET_CONF = 2,
> +
> + /* privacy notification id used when privacy state changes */
> + CSI_PRIVACY_NOTIF = 6,
> +};
> +
> +/* configuration of the CSI-2 link between host and IVSC */
> +struct csi_link_cfg {
> + /* number of data lanes used on the CSI-2 link */
> + u32 nr_of_lanes;
> +
> + /* frequency of the CSI-2 link */
> + u32 link_freq;
> +
> + /* for future use */
> + u32 rsvd[2];
> +} __packed;
> +
> +/* CSI command structure */
> +struct csi_cmd {
> + u32 cmd_id;
> + union _cmd_param {
> + u32 param;
> + struct csi_link_cfg conf;
> + } param;
> +} __packed;
> +
> +/* CSI notification structure */
> +struct csi_notif {
> + u32 cmd_id;
> + int status;
> + union _resp_cont {
> + u32 cont;
> + struct csi_link_cfg conf;
> + } cont;
> +} __packed;
> +
> +struct mei_csi {
> + struct mei_cl_device *cldev;
> +
> + /* command response */
> + struct csi_notif cmd_response;
> + /* used to wait for command response from firmware */
> + struct completion cmd_completion;
> +
> + /* work element used to handle firmware event */
> + struct work_struct event_work;
> +
> + /* privacy status */
> + enum ivsc_privacy_status status;
> + /* privacy callback */
> + void (*callback)(void *, enum ivsc_privacy_status);
> + /* privacy callback runtime context */
> + void *context;
> +};
> +
> +/* only one for now */
> +static struct mei_csi *csi;
> +/* lock used to prevent multiple call to csi */
> +static DEFINE_MUTEX(csi_mutex);
> +
> +/* send a command to firmware and mutex must be held by caller */
> +static int mei_csi_send(u8 *buf, size_t len)
> +{
> + struct csi_cmd *cmd = (struct csi_cmd *)buf;
> + int ret;
> +
> + reinit_completion(&csi->cmd_completion);
> +
> + ret = mei_cldev_send(csi->cldev, buf, len);
> + if (ret < 0)
> + goto out;
> +
> + ret = wait_for_completion_killable_timeout(&csi->cmd_completion,
> + CSI_CMD_TIMEOUT);
> + if (ret < 0) {
> + goto out;
> + } else if (!ret) {
> + ret = -ETIMEDOUT;
> + goto out;
> + }
> +
> + /* command response status */
> + ret = csi->cmd_response.status;
> + if (ret) {
> + ret = -EINVAL;
> + goto out;
> + }
> +
> + if (csi->cmd_response.cmd_id != cmd->cmd_id)
> + ret = -EINVAL;
> +
> +out:
> + return ret;
> +}
> +
> +int csi_set_link_owner(enum csi_link_owner owner,
> + void (*callback)(void *, enum ivsc_privacy_status),
> + void *context)
> +{
> + struct csi_cmd cmd = { 0 };
> + size_t cmd_size;
> + int ret;
> +
> + cmd.cmd_id = CSI_SET_OWNER;
> + cmd.param.param = owner;
> + cmd_size = sizeof(cmd.cmd_id) + sizeof(cmd.param.param);
You could initialise these in variable declaration.
> +
> + mutex_lock(&csi_mutex);
> + if (unlikely(!csi)) {
> + mutex_unlock(&csi_mutex);
> + return -EAGAIN;
> + }
> +
> + ret = mei_csi_send((u8 *)&cmd, cmd_size);
> +
> + /*
> + * cancel possible event work and wait for it to finish
> + * to make sure no work running for the ongoing changes
> + * to callback and context.
> + */
> + cancel_work_sync(&csi->event_work);
> + csi->callback = callback;
> + csi->context = context;
> + mutex_unlock(&csi_mutex);
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_NS_GPL(csi_set_link_owner, IVSC);
> +
> +int csi_set_link_cfg(u32 nr_of_lanes, u64 link_freq)
> +{
> + struct csi_cmd cmd = { 0 };
> + size_t cmd_size;
> + int ret;
> +
> + cmd.cmd_id = CSI_SET_CONF;
> + cmd.param.conf.nr_of_lanes = nr_of_lanes;
> + cmd.param.conf.link_freq = CSI_LINK_FREQ(link_freq);
> + cmd_size = sizeof(cmd.cmd_id) + sizeof(cmd.param.conf);
> +
> + mutex_lock(&csi_mutex);
> + if (unlikely(!csi)) {
> + mutex_unlock(&csi_mutex);
> + return -EAGAIN;
> + }
> +
> + ret = mei_csi_send((u8 *)&cmd, cmd_size);
> + /*
> + * wait configuration ready if download success. placing
> + * delay under mutex is to make sure current command flow
> + * completed before starting a possible new one.
> + */
> + if (!ret)
> + msleep(CSI_FW_READY_DELAY_MS);
> + mutex_unlock(&csi_mutex);
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_NS_GPL(csi_set_link_cfg, IVSC);
> +
> +/* event handling routine */
> +static void mei_csi_event_work(struct work_struct *work)
> +{
> + if (csi->callback)
> + csi->callback(csi->context, READ_ONCE(csi->status));
> +}
> +
> +/* callback for receive */
> +static void mei_csi_rx(struct mei_cl_device *cldev)
> +{
> + struct csi_notif notif = { 0 };
> + int ret;
> +
> + ret = mei_cldev_recv(cldev, (u8 *)¬if, sizeof(notif));
> + if (ret < 0) {
> + dev_err(&cldev->dev, "recv error: %d\n", ret);
> + return;
> + }
> +
> + switch (notif.cmd_id) {
> + case CSI_PRIVACY_NOTIF:
> + if (notif.cont.cont < IVSC_PRIVACY_MAX) {
> + WRITE_ONCE(csi->status, notif.cont.cont);
> +
> + schedule_work(&csi->event_work);
> + }
> + break;
> + case CSI_SET_OWNER:
> + case CSI_SET_CONF:
> + memcpy(&csi->cmd_response, ¬if, ret);
> +
> + complete(&csi->cmd_completion);
> + break;
> + default:
> + break;
> + }
> +}
> +
> +static int mei_csi_probe(struct mei_cl_device *cldev,
> + const struct mei_cl_device_id *id)
> +{
> + int ret;
> +
> + mutex_lock(&csi_mutex);
> + /*
> + * only instance is possible in the current hardware,
> + * but adding protection for future hardware.
> + */
> + if (csi) {
> + ret = -EBUSY;
> + goto err_unlock;
> + }
> +
> + csi = kzalloc(sizeof(*csi), GFP_KERNEL);
How do you know the MEI CSI and the MEI ACE devices are related to a given
camera sensor? I'd expect this information to come from system firmware but
I don't see this being done in this patchset.
Ideally there should be also a way to obtain the appropriate MEI CSI and
MEI ACE devices without assuming there's a single one in the system.
> + if (!csi) {
> + ret = -ENOMEM;
> + goto err_unlock;
> + }
> +
> + csi->cldev = cldev;
> + init_completion(&csi->cmd_completion);
> + INIT_WORK(&csi->event_work, mei_csi_event_work);
> +
> + mei_cldev_set_drvdata(cldev, csi);
> +
> + ret = mei_cldev_enable(cldev);
> + if (ret < 0) {
> + dev_err(&cldev->dev, "mei_cldev_enable failed: %d\n", ret);
> + goto err_free;
> + }
> +
> + ret = mei_cldev_register_rx_cb(cldev, mei_csi_rx);
> + if (ret) {
> + dev_err(&cldev->dev, "event cb registration failed: %d\n", ret);
> + goto err_disable;
> + }
> + mutex_unlock(&csi_mutex);
> +
> + return 0;
> +
> +err_disable:
> + mei_cldev_disable(cldev);
> +
> +err_free:
> + kfree(csi);
> + /* disable csi */
> + csi = NULL;
> +
> +err_unlock:
> + mutex_unlock(&csi_mutex);
> +
> + return ret;
> +}
> +
> +static void mei_csi_remove(struct mei_cl_device *cldev)
> +{
> + mutex_lock(&csi_mutex);
> + /* disable mei csi client device */
> + mei_cldev_disable(cldev);
> +
> + /* cancel event work and wait for it to finish */
> + cancel_work_sync(&csi->event_work);
> +
> + kfree(csi);
> + /* disable csi */
> + csi = NULL;
> + mutex_unlock(&csi_mutex);
> +}
> +
> +#define MEI_CSI_UUID UUID_LE(0x92335FCF, 0x3203, 0x4472, \
> + 0xAF, 0x93, 0x7b, 0x44, 0x53, 0xAC, 0x29, 0xDA)
> +
> +static const struct mei_cl_device_id mei_csi_tbl[] = {
> + { MEI_CSI_DRIVER_NAME, MEI_CSI_UUID, MEI_CL_VERSION_ANY },
> +
> + /* required last entry */
> + { }
> +};
> +MODULE_DEVICE_TABLE(mei, mei_csi_tbl);
> +
> +static struct mei_cl_driver mei_csi_driver = {
> + .id_table = mei_csi_tbl,
> + .name = MEI_CSI_DRIVER_NAME,
> +
> + .probe = mei_csi_probe,
> + .remove = mei_csi_remove,
> +};
> +
> +module_mei_cl_driver(mei_csi_driver);
> +
> +MODULE_AUTHOR("Wentong Wu <[email protected]>");
> +MODULE_AUTHOR("Zhifeng Wang <[email protected]>");
> +MODULE_DESCRIPTION("Device driver for IVSC CSI");
> +MODULE_LICENSE("GPL");
> diff --git a/drivers/media/pci/intel/ivsc/mei_csi.h b/drivers/media/pci/intel/ivsc/mei_csi.h
> new file mode 100644
> index 000000000000..0cc51e6a5250
> --- /dev/null
> +++ b/drivers/media/pci/intel/ivsc/mei_csi.h
> @@ -0,0 +1,60 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Copyright (C) 2023 Intel Corporation. All rights reserved.
> + * Intel Visual Sensing Controller CSI external interface
> + */
> +
> +#ifndef _MEI_CSI_H_
> +#define _MEI_CSI_H_
> +
> +#include <linux/ivsc.h>
> +
> +/* CSI-2 link ownership definition */
> +enum csi_link_owner {
> + CSI_LINK_IVSC,
> + CSI_LINK_HOST,
> +};
> +
> +#if IS_ENABLED(CONFIG_INTEL_VSC)
> +/*
> + * @brief set CSI-2 link ownership
> + *
> + * @param owner The csi ownership being set
> + * @param callback The pointer of privacy callback being set
> + * @param context Privacy callback runtime context
> + *
> + * @return 0 on success, negative on failure
> + */
> +int csi_set_link_owner(enum csi_link_owner owner,
> + void (*callback)(void *, enum ivsc_privacy_status),
> + void *context);
> +
> +/*
> + * @brief configure CSI-2 link between host and IVSC
> + * with provided parameters
> + *
> + * @param nr_of_lanes The number of data lanes being used
> + * on the CSI-2 link
> + * @param link_freq The frequency being set on the CSI-2 link
> + *
> + * @return 0 on success, negative on failure
> + */
> +int csi_set_link_cfg(u32 nr_of_lanes, u64 link_freq);
> +
> +#else
> +static inline
> +int csi_set_link_owner(enum csi_link_owner owner,
> + void (*callback)(void *, enum ivsc_privacy_status),
> + void *context)
> +{
> + return 0;
> +}
> +
> +static inline int csi_set_link_cfg(u32 nr_of_lanes, u64 link_freq)
> +{
> + return 0;
> +}
> +
> +#endif
> +
> +#endif
> diff --git a/include/linux/ivsc.h b/include/linux/ivsc.h
> new file mode 100644
> index 000000000000..6572ca4f340c
> --- /dev/null
> +++ b/include/linux/ivsc.h
> @@ -0,0 +1,19 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Copyright (C) 2023 Intel Corporation. All rights reserved.
> + * Intel Visual Sensing Controller interface
> + */
> +
> +#ifndef _LINUX_IVSC_H_
> +#define _LINUX_IVSC_H_
> +
> +#include <linux/types.h>
> +
> +/* IVSC privacy status definition */
> +enum ivsc_privacy_status {
> + IVSC_PRIVACY_OFF,
> + IVSC_PRIVACY_ON,
> + IVSC_PRIVACY_MAX,
> +};
> +
> +#endif
--
Kind regards,
Sakari Ailus
Hi Sakari,
On 2/14/23 17:06, Sakari Ailus wrote:
> Hi Wentong,
>
> Thanks for the patchset.
>
> On Mon, Feb 13, 2023 at 10:23:47AM +0800, Wentong Wu wrote:
>> IVSC directly connects to camera sensor on source side, and on
>> output side it not only connects ISH via I2C, but also exposes
>> MIPI CSI-2 interface to output camera sensor data. IVSC can use
>> the camera sensor data to do AI algorithm, and send the results
>> to ISH. On the other end, IVSC can share camera sensor to host
>> by routing the raw camera sensor data to the exposed MIPI CSI-2
>> interface. But they can not work at the same time, so software
>> APIs are defined to sync the ownership.
>>
>> This commit defines the interfaces between IVSC and camera sensor
>> driver in include/linux/ivsc.h. The camera driver controls
>> ownership of the CSI-2 link and sensor with the acquire/release
>> APIs. When acquiring camera, lane number and link freq are also
>> required by IVSC frame router.
>
> The more I learn about this system, the more I'm inclined to think this
> functionality should be exposed as a V4L2 sub-device. IVSC doesn't really
> do anything to the data (as long as it directs it towards the CSI-2
> receiver in the SoC), but it is definitely part of the image pipeline.
Yes I happened to discuss this exact same thing with Laurent at FOSDEM
and we also came to the conclusion that the IVSC chip should be modeled
as a V4L2 sub-device.
Regards,
Hans
>
> I suppose the intended use cases assume a single instance of IVSC (as well
> as MEI) but there can, and often are, be multiple camera sensors in the
> system. The decision whether to request pass-through from IVCS can't be
> done in the camera sensor driver, and should not be visible to the camera
> sensor driver. Exposing IVSC as a V4L2 sub-device makes this trivial to
> address, as the IVSC driver's V4L2 sub-device video s_stream() operation
> gets called before streaming is started.
>
> The information whether IVSC is found between the camera sensor and the
> host's CSI-2 receiver (IPU in this case) should come from system firmware
> and accessed most probably by what is called cio2-bridge at the moment.
>
> The privacy status can be a V4L2 control.
>
> Also cc Hans.
>
>>
>> Signed-off-by: Wentong Wu <[email protected]>
>> ---
>> drivers/media/pci/intel/ivsc/Makefile | 1 +
>> drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++++++++++++++++++++++++
>> include/linux/ivsc.h | 55 ++++++++++++++++++
>> 3 files changed, 140 insertions(+)
>> create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
>>
>> diff --git a/drivers/media/pci/intel/ivsc/Makefile b/drivers/media/pci/intel/ivsc/Makefile
>> index de0a425c22c2..b8b6fc1083be 100644
>> --- a/drivers/media/pci/intel/ivsc/Makefile
>> +++ b/drivers/media/pci/intel/ivsc/Makefile
>> @@ -4,3 +4,4 @@
>>
>> obj-$(CONFIG_INTEL_VSC) += mei_csi.o
>> obj-$(CONFIG_INTEL_VSC) += mei_ace.o
>> +obj-$(CONFIG_INTEL_VSC) += ivsc.o
>> diff --git a/drivers/media/pci/intel/ivsc/ivsc.c b/drivers/media/pci/intel/ivsc/ivsc.c
>> new file mode 100644
>> index 000000000000..12996b587639
>> --- /dev/null
>> +++ b/drivers/media/pci/intel/ivsc/ivsc.c
>> @@ -0,0 +1,84 @@
>> +// SPDX-License-Identifier: GPL-2.0-only
>> +/*
>> + * Copyright (C) 2023 Intel Corporation. All rights reserved.
>> + * Intel Visual Sensing Controller interface
>> + */
>> +
>> +#include <linux/delay.h>
>> +#include <linux/ivsc.h>
>> +#include <linux/module.h>
>> +#include <linux/mutex.h>
>> +
>> +#include "mei_ace.h"
>> +#include "mei_csi.h"
>> +
>> +/* lock for ivsc APIs */
>> +static DEFINE_MUTEX(ivsc_mutex);
>> +
>> +int ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
>> + void (*callback)(void *, enum ivsc_privacy_status),
>> + void *context)
>> +{
>> + int ret;
>> +
>> + mutex_lock(&ivsc_mutex);
>> +
>> + /* switch camera sensor ownership to host */
>> + ret = ace_set_camera_owner(ACE_CAMERA_HOST);
>> + if (ret)
>> + goto error;
>> +
>> + /* switch CSI-2 link to host */
>> + ret = csi_set_link_owner(CSI_LINK_HOST, callback, context);
>> + if (ret)
>> + goto release_camera;
>> +
>> + /* configure CSI-2 link */
>> + ret = csi_set_link_cfg(nr_of_lanes, link_freq);
>> + if (ret)
>> + goto release_csi;
>> +
>> + mutex_unlock(&ivsc_mutex);
>> +
>> + return 0;
>> +
>> +release_csi:
>> + csi_set_link_owner(CSI_LINK_IVSC, NULL, NULL);
>> +
>> +release_camera:
>> + ace_set_camera_owner(ACE_CAMERA_IVSC);
>> +
>> +error:
>> + mutex_unlock(&ivsc_mutex);
>> +
>> + return ret;
>> +}
>> +EXPORT_SYMBOL_GPL(ivsc_acquire_camera);
>> +
>> +int ivsc_release_camera(void)
>> +{
>> + int ret;
>> +
>> + mutex_lock(&ivsc_mutex);
>> +
>> + /* switch CSI-2 link to IVSC */
>> + ret = csi_set_link_owner(CSI_LINK_IVSC, NULL, NULL);
>> + if (ret)
>> + goto error;
>> +
>> + /* switch camera sensor ownership to IVSC */
>> + ret = ace_set_camera_owner(ACE_CAMERA_IVSC);
>> +
>> +error:
>> + mutex_unlock(&ivsc_mutex);
>> +
>> + return ret;
>> +}
>> +EXPORT_SYMBOL_GPL(ivsc_release_camera);
>> +
>> +MODULE_AUTHOR("Wentong Wu <[email protected]>");
>> +MODULE_AUTHOR("Zhifeng Wang <[email protected]>");
>> +MODULE_SOFTDEP("pre: mei_csi mei_ace");
>> +MODULE_DESCRIPTION("IVSC interface");
>> +MODULE_LICENSE("GPL");
>> +MODULE_IMPORT_NS(IVSC);
>> diff --git a/include/linux/ivsc.h b/include/linux/ivsc.h
>> index 6572ca4f340c..bc9006cd6efc 100644
>> --- a/include/linux/ivsc.h
>> +++ b/include/linux/ivsc.h
>> @@ -16,4 +16,59 @@ enum ivsc_privacy_status {
>> IVSC_PRIVACY_MAX,
>> };
>>
>> +#if IS_ENABLED(CONFIG_INTEL_VSC)
>> +/*
>> + * @brief Acquire camera sensor ownership to host and setup
>> + * the CSI-2 link between host and IVSC
>> + *
>> + * IVSC provides a privacy mode. When the privacy mode is turned
>> + * on, camera sensor can't be used. This means that both IVSC and
>> + * host Image Processing Unit(IPU) can't get image data. And when
>> + * this mode is turned on, host Image Processing Unit(IPU) driver
>> + * is informed via the registered callback, so that user can be
>> + * notified.
>> + *
>> + * @param nr_of_lanes Number of data lanes used on the CSI-2 link
>> + * @param link_freq Frequency of the CSI-2 link
>> + * @param callback The pointer of privacy callback function
>> + * @param context Privacy callback function runtime context
>> + *
>> + * @retval 0 If success
>> + * @retval -EIO IO error
>> + * @retval -EINVAL Invalid argument
>> + * @retval -EAGAIN IVSC device not ready
>> + * @retval negative values for other errors
>> + */
>> +int ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
>> + void (*callback)(void *, enum ivsc_privacy_status),
>> + void *context);
>> +
>> +/*
>> + * @brief Release camera sensor ownership and stop the CSI-2
>> + * link between host and IVSC
>> + *
>> + * @retval 0 If success
>> + * @retval -EIO IO error
>> + * @retval -EINVAL Invalid argument
>> + * @retval -EAGAIN IVSC device not ready
>> + * @retval negative values for other errors
>> + */
>> +int ivsc_release_camera(void);
>> +
>> +#else
>> +static inline
>> +int ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
>> + void (*callback)(void *, enum ivsc_privacy_status),
>> + void *context)
>> +{
>> + return 0;
>> +}
>> +
>> +static inline int ivsc_release_camera(void)
>> +{
>> + return 0;
>> +}
>> +
>> +#endif
>> +
>> #endif
>
Hello Wentong,
On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
> Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a
> companion chip designed to provide secure and low power vision capability
> to IA platforms. IVSC is available in existing commercial platforms from
> multiple OEMs.
>
> The primary use case of IVSC is to bring in context awareness. IVSC
> interfaces directly with the platform main camera sensor via a CSI-2 link
> and processes the image data with the embedded AI engine. The detected
> events are sent over I2C to ISH (Intel Sensor Hub) for additional data
> fusion from multiple sensors. The fusion results are used to implement
> advanced use cases like:
> - Face detection to unlock screen
> - Detect user presence to manage backlight setting or waking up system
Do you have plan to support these features in the ivsc driver in the
future ?
> Since the Image Processing Unit(IPU) used on the host processor needs to
> configure the CSI-2 link in normal camera usages, the CSI-2 link and
> camera sensor can only be used in mutually-exclusive ways by host IPU and
> IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU
> driver can take ownership of the CSI-2 link and camera sensor using
> interfaces provided by this IVSC driver.
>
> Switching ownership requires an interface with two different hardware
> modules inside IVSC. The software interface to these modules is via Intel
> MEI (The Intel Management Engine) commands. These two hardware modules
> have two different MEI UUIDs to enumerate. These hardware modules are:
> - ACE (Algorithm Context Engine): This module is for algorithm computing
> when IVSC owns camera sensor. Also ACE module controls camera sensor's
> ownership. This hardware module is used to set ownership of camera sensor.
> - CSI (Camera Serial Interface): This module is used to route camera
> sensor data either to IVSC or to host for IPU driver and application.
>
> IVSC also provides a privacy mode. When privacy mode is turned on,
> camera sensor can't be used. This means that both ACE and host IPU can't
> get image data. And when this mode is turned on, host IPU driver is
> informed via a registered callback, so that user can be notified.
How does the privacy mode work, and how can the user trust that the
closed-source IVSC and IME firmwares will honour the privacy settings ?
> In summary, to acquire ownership of camera by IPU driver, first ACE
> module needs to be informed of ownership and then to setup MIPI CSI-2
> link for the camera sensor and IPU.
>
> Implementation:
> There are two different drivers to handle ACE and CSI hardware modules
> inside IVSC.
> - mei_csi: MEI client driver to send commands and receive notifications
> from CSI module.
> - mei_ace: MEI client driver to send commands and get status from ACE
> module.
> Interface is exposed via ivsc.h to acquire and release camera sensor and
> CSI-2 link.
Do I understand correctly, from your diagram below, that the
communication between the IME and IVSC goes through SPI ?
> Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
> -----------------------------------------------------------------------------
> | Host Processor |
> | |
> | ----------------- ----------------- --------------- |
> | | | | | | | I2C |
> | | IPU | | ISH | |camera driver|--| |
> | | | | | | | | |
> | ----------------- ----------------- --------------- | |
> | | | | | |
> | | | --------------- | |
> | | | | | | |
> | | | | IVSC driver | | |
> | | | | | | |
> | | | --------------- | |
> | | | | | |
> ----------------|-----------------------|----------------------|---------|---
> | CSI | I2C |SPI |
> | | | |
> ----------------|-----------------------|---------------- | |
> | IVSC | | | |
> | | | | |
> | ----------------- ----------------- | | |
> | | | | | | | |
> | | CSI | | ACE | |------| |
> | | | | | | |
> | ----------------- ----------------- | |
> | | | I2C | |
> ----------------|-----------------------|---------------- |
> | CSI | |
> | | |
> -------------------------------- |
> | | I2C |
> | camera sensor |-----------------------------|
> | |
> --------------------------------
>
> Wentong Wu (3):
> media: pci: intel: ivsc: Add CSI submodule
> media: pci: intel: ivsc: Add ACE submodule
> media: pci: intel: ivsc: Add acquire/release API for ivsc
>
> drivers/media/pci/Kconfig | 1 +
> drivers/media/pci/intel/Makefile | 2 +
> drivers/media/pci/intel/ivsc/Kconfig | 12 +
> drivers/media/pci/intel/ivsc/Makefile | 7 +
> drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++
> drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++
> drivers/media/pci/intel/ivsc/mei_ace.h | 36 ++
> drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++
> drivers/media/pci/intel/ivsc/mei_csi.h | 60 ++++
> include/linux/ivsc.h | 74 ++++
> 10 files changed, 1090 insertions(+)
> create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
> create mode 100644 drivers/media/pci/intel/ivsc/Makefile
> create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
> create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
> create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
> create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
> create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
> create mode 100644 include/linux/ivsc.h
--
Regards,
Laurent Pinchart
On Wed, Feb 15, 2023 at 10:03:29AM +0100, Hans de Goede wrote:
> Hi Sakari,
>
> On 2/14/23 17:06, Sakari Ailus wrote:
> > Hi Wentong,
> >
> > Thanks for the patchset.
> >
> > On Mon, Feb 13, 2023 at 10:23:47AM +0800, Wentong Wu wrote:
> >> IVSC directly connects to camera sensor on source side, and on
> >> output side it not only connects ISH via I2C, but also exposes
> >> MIPI CSI-2 interface to output camera sensor data. IVSC can use
> >> the camera sensor data to do AI algorithm, and send the results
> >> to ISH. On the other end, IVSC can share camera sensor to host
> >> by routing the raw camera sensor data to the exposed MIPI CSI-2
> >> interface. But they can not work at the same time, so software
> >> APIs are defined to sync the ownership.
> >>
> >> This commit defines the interfaces between IVSC and camera sensor
> >> driver in include/linux/ivsc.h. The camera driver controls
> >> ownership of the CSI-2 link and sensor with the acquire/release
> >> APIs. When acquiring camera, lane number and link freq are also
> >> required by IVSC frame router.
> >
> > The more I learn about this system, the more I'm inclined to think this
> > functionality should be exposed as a V4L2 sub-device. IVSC doesn't really
> > do anything to the data (as long as it directs it towards the CSI-2
> > receiver in the SoC), but it is definitely part of the image pipeline.
>
> Yes I happened to discuss this exact same thing with Laurent at FOSDEM
> and we also came to the conclusion that the IVSC chip should be modeled
> as a V4L2 sub-device.
Agreed.
> > I suppose the intended use cases assume a single instance of IVSC (as well
> > as MEI) but there can, and often are, be multiple camera sensors in the
> > system. The decision whether to request pass-through from IVCS can't be
> > done in the camera sensor driver, and should not be visible to the camera
> > sensor driver. Exposing IVSC as a V4L2 sub-device makes this trivial to
> > address, as the IVSC driver's V4L2 sub-device video s_stream() operation
> > gets called before streaming is started.
> >
> > The information whether IVSC is found between the camera sensor and the
> > host's CSI-2 receiver (IPU in this case) should come from system firmware
> > and accessed most probably by what is called cio2-bridge at the moment.
> >
> > The privacy status can be a V4L2 control.
> >
> > Also cc Hans.
> >
> >>
> >> Signed-off-by: Wentong Wu <[email protected]>
> >> ---
> >> drivers/media/pci/intel/ivsc/Makefile | 1 +
> >> drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++++++++++++++++++++++++
> >> include/linux/ivsc.h | 55 ++++++++++++++++++
> >> 3 files changed, 140 insertions(+)
> >> create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
> >>
> >> diff --git a/drivers/media/pci/intel/ivsc/Makefile b/drivers/media/pci/intel/ivsc/Makefile
> >> index de0a425c22c2..b8b6fc1083be 100644
> >> --- a/drivers/media/pci/intel/ivsc/Makefile
> >> +++ b/drivers/media/pci/intel/ivsc/Makefile
> >> @@ -4,3 +4,4 @@
> >>
> >> obj-$(CONFIG_INTEL_VSC) += mei_csi.o
> >> obj-$(CONFIG_INTEL_VSC) += mei_ace.o
> >> +obj-$(CONFIG_INTEL_VSC) += ivsc.o
> >> diff --git a/drivers/media/pci/intel/ivsc/ivsc.c b/drivers/media/pci/intel/ivsc/ivsc.c
> >> new file mode 100644
> >> index 000000000000..12996b587639
> >> --- /dev/null
> >> +++ b/drivers/media/pci/intel/ivsc/ivsc.c
> >> @@ -0,0 +1,84 @@
> >> +// SPDX-License-Identifier: GPL-2.0-only
> >> +/*
> >> + * Copyright (C) 2023 Intel Corporation. All rights reserved.
> >> + * Intel Visual Sensing Controller interface
> >> + */
> >> +
> >> +#include <linux/delay.h>
> >> +#include <linux/ivsc.h>
> >> +#include <linux/module.h>
> >> +#include <linux/mutex.h>
> >> +
> >> +#include "mei_ace.h"
> >> +#include "mei_csi.h"
> >> +
> >> +/* lock for ivsc APIs */
> >> +static DEFINE_MUTEX(ivsc_mutex);
> >> +
> >> +int ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
> >> + void (*callback)(void *, enum ivsc_privacy_status),
> >> + void *context)
> >> +{
> >> + int ret;
> >> +
> >> + mutex_lock(&ivsc_mutex);
> >> +
> >> + /* switch camera sensor ownership to host */
> >> + ret = ace_set_camera_owner(ACE_CAMERA_HOST);
> >> + if (ret)
> >> + goto error;
> >> +
> >> + /* switch CSI-2 link to host */
> >> + ret = csi_set_link_owner(CSI_LINK_HOST, callback, context);
> >> + if (ret)
> >> + goto release_camera;
> >> +
> >> + /* configure CSI-2 link */
> >> + ret = csi_set_link_cfg(nr_of_lanes, link_freq);
> >> + if (ret)
> >> + goto release_csi;
> >> +
> >> + mutex_unlock(&ivsc_mutex);
> >> +
> >> + return 0;
> >> +
> >> +release_csi:
> >> + csi_set_link_owner(CSI_LINK_IVSC, NULL, NULL);
> >> +
> >> +release_camera:
> >> + ace_set_camera_owner(ACE_CAMERA_IVSC);
> >> +
> >> +error:
> >> + mutex_unlock(&ivsc_mutex);
> >> +
> >> + return ret;
> >> +}
> >> +EXPORT_SYMBOL_GPL(ivsc_acquire_camera);
> >> +
> >> +int ivsc_release_camera(void)
> >> +{
> >> + int ret;
> >> +
> >> + mutex_lock(&ivsc_mutex);
> >> +
> >> + /* switch CSI-2 link to IVSC */
> >> + ret = csi_set_link_owner(CSI_LINK_IVSC, NULL, NULL);
> >> + if (ret)
> >> + goto error;
> >> +
> >> + /* switch camera sensor ownership to IVSC */
> >> + ret = ace_set_camera_owner(ACE_CAMERA_IVSC);
> >> +
> >> +error:
> >> + mutex_unlock(&ivsc_mutex);
> >> +
> >> + return ret;
> >> +}
> >> +EXPORT_SYMBOL_GPL(ivsc_release_camera);
> >> +
> >> +MODULE_AUTHOR("Wentong Wu <[email protected]>");
> >> +MODULE_AUTHOR("Zhifeng Wang <[email protected]>");
> >> +MODULE_SOFTDEP("pre: mei_csi mei_ace");
> >> +MODULE_DESCRIPTION("IVSC interface");
> >> +MODULE_LICENSE("GPL");
> >> +MODULE_IMPORT_NS(IVSC);
> >> diff --git a/include/linux/ivsc.h b/include/linux/ivsc.h
> >> index 6572ca4f340c..bc9006cd6efc 100644
> >> --- a/include/linux/ivsc.h
> >> +++ b/include/linux/ivsc.h
> >> @@ -16,4 +16,59 @@ enum ivsc_privacy_status {
> >> IVSC_PRIVACY_MAX,
> >> };
> >>
> >> +#if IS_ENABLED(CONFIG_INTEL_VSC)
> >> +/*
> >> + * @brief Acquire camera sensor ownership to host and setup
> >> + * the CSI-2 link between host and IVSC
> >> + *
> >> + * IVSC provides a privacy mode. When the privacy mode is turned
> >> + * on, camera sensor can't be used. This means that both IVSC and
> >> + * host Image Processing Unit(IPU) can't get image data. And when
> >> + * this mode is turned on, host Image Processing Unit(IPU) driver
> >> + * is informed via the registered callback, so that user can be
> >> + * notified.
> >> + *
> >> + * @param nr_of_lanes Number of data lanes used on the CSI-2 link
> >> + * @param link_freq Frequency of the CSI-2 link
> >> + * @param callback The pointer of privacy callback function
> >> + * @param context Privacy callback function runtime context
> >> + *
> >> + * @retval 0 If success
> >> + * @retval -EIO IO error
> >> + * @retval -EINVAL Invalid argument
> >> + * @retval -EAGAIN IVSC device not ready
> >> + * @retval negative values for other errors
> >> + */
> >> +int ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
> >> + void (*callback)(void *, enum ivsc_privacy_status),
> >> + void *context);
> >> +
> >> +/*
> >> + * @brief Release camera sensor ownership and stop the CSI-2
> >> + * link between host and IVSC
> >> + *
> >> + * @retval 0 If success
> >> + * @retval -EIO IO error
> >> + * @retval -EINVAL Invalid argument
> >> + * @retval -EAGAIN IVSC device not ready
> >> + * @retval negative values for other errors
> >> + */
> >> +int ivsc_release_camera(void);
> >> +
> >> +#else
> >> +static inline
> >> +int ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
> >> + void (*callback)(void *, enum ivsc_privacy_status),
> >> + void *context)
> >> +{
> >> + return 0;
> >> +}
> >> +
> >> +static inline int ivsc_release_camera(void)
> >> +{
> >> + return 0;
> >> +}
> >> +
> >> +#endif
> >> +
> >> #endif
--
Regards,
Laurent Pinchart
Hi, Wentong,
On 2/15/23 5:43 PM, Laurent Pinchart wrote:
> Hello Wentong,
>
> On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
>> Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a
>> companion chip designed to provide secure and low power vision capability
>> to IA platforms. IVSC is available in existing commercial platforms from
>> multiple OEMs.
>>
>> The primary use case of IVSC is to bring in context awareness. IVSC
>> interfaces directly with the platform main camera sensor via a CSI-2 link
>> and processes the image data with the embedded AI engine. The detected
>> events are sent over I2C to ISH (Intel Sensor Hub) for additional data
>> fusion from multiple sensors. The fusion results are used to implement
>> advanced use cases like:
>> - Face detection to unlock screen
>> - Detect user presence to manage backlight setting or waking up system
>
> Do you have plan to support these features in the ivsc driver in the
> future ?
>
>> Since the Image Processing Unit(IPU) used on the host processor needs to
>> configure the CSI-2 link in normal camera usages, the CSI-2 link and
>> camera sensor can only be used in mutually-exclusive ways by host IPU and
>> IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU
>> driver can take ownership of the CSI-2 link and camera sensor using
>> interfaces provided by this IVSC driver.
>>
>> Switching ownership requires an interface with two different hardware
>> modules inside IVSC. The software interface to these modules is via Intel
>> MEI (The Intel Management Engine) commands. These two hardware modules
>> have two different MEI UUIDs to enumerate. These hardware modules are:
>> - ACE (Algorithm Context Engine): This module is for algorithm computing
>> when IVSC owns camera sensor. Also ACE module controls camera sensor's
>> ownership. This hardware module is used to set ownership of camera sensor.
>> - CSI (Camera Serial Interface): This module is used to route camera
>> sensor data either to IVSC or to host for IPU driver and application.
>>
>> IVSC also provides a privacy mode. When privacy mode is turned on,
>> camera sensor can't be used. This means that both ACE and host IPU can't
>> get image data. And when this mode is turned on, host IPU driver is
>> informed via a registered callback, so that user can be notified.
>
> How does the privacy mode work, and how can the user trust that the
> closed-source IVSC and IME firmwares will honour the privacy settings ?
>
Continue with question from Laurent,
How IVSC handle the privacy request from user? Is there some notification
mechanism to user-space? I'd have concern if IVSC driver need private callback
to request back-end(e.g. ISP driver) to handle stream cutting.
>> In summary, to acquire ownership of camera by IPU driver, first ACE
>> module needs to be informed of ownership and then to setup MIPI CSI-2
>> link for the camera sensor and IPU.
>>
>> Implementation:
>> There are two different drivers to handle ACE and CSI hardware modules
>> inside IVSC.
>> - mei_csi: MEI client driver to send commands and receive notifications
>> from CSI module.
>> - mei_ace: MEI client driver to send commands and get status from ACE
>> module.
>> Interface is exposed via ivsc.h to acquire and release camera sensor and
>> CSI-2 link.
>
> Do I understand correctly, from your diagram below, that the
> communication between the IME and IVSC goes through SPI ?
>
>> Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
>> -----------------------------------------------------------------------------
>> | Host Processor |
>> | |
>> | ----------------- ----------------- --------------- |
>> | | | | | | | I2C |
>> | | IPU | | ISH | |camera driver|--| |
>> | | | | | | | | |
>> | ----------------- ----------------- --------------- | |
>> | | | | | |
>> | | | --------------- | |
>> | | | | | | |
>> | | | | IVSC driver | | |
>> | | | | | | |
>> | | | --------------- | |
>> | | | | | |
>> ----------------|-----------------------|----------------------|---------|---
>> | CSI | I2C |SPI |
>> | | | |
>> ----------------|-----------------------|---------------- | |
>> | IVSC | | | |
>> | | | | |
>> | ----------------- ----------------- | | |
>> | | | | | | | |
>> | | CSI | | ACE | |------| |
>> | | | | | | |
>> | ----------------- ----------------- | |
>> | | | I2C | |
>> ----------------|-----------------------|---------------- |
>> | CSI | |
>> | | |
>> -------------------------------- |
>> | | I2C |
>> | camera sensor |-----------------------------|
>> | |
>> --------------------------------
>>
>> Wentong Wu (3):
>> media: pci: intel: ivsc: Add CSI submodule
>> media: pci: intel: ivsc: Add ACE submodule
>> media: pci: intel: ivsc: Add acquire/release API for ivsc
>>
>> drivers/media/pci/Kconfig | 1 +
>> drivers/media/pci/intel/Makefile | 2 +
>> drivers/media/pci/intel/ivsc/Kconfig | 12 +
>> drivers/media/pci/intel/ivsc/Makefile | 7 +
>> drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++
>> drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++
>> drivers/media/pci/intel/ivsc/mei_ace.h | 36 ++
>> drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++
>> drivers/media/pci/intel/ivsc/mei_csi.h | 60 ++++
>> include/linux/ivsc.h | 74 ++++
>> 10 files changed, 1090 insertions(+)
>> create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
>> create mode 100644 drivers/media/pci/intel/ivsc/Makefile
>> create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
>> create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
>> create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
>> create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
>> create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
>> create mode 100644 include/linux/ivsc.h
>
--
Best regards,
Bingbu Cao
Hi Bingbu, Wentong,
On Wed, Feb 15, 2023 at 08:09:50PM +0800, Bingbu Cao wrote:
>
> Hi, Wentong,
>
> On 2/15/23 5:43 PM, Laurent Pinchart wrote:
> > Hello Wentong,
> >
> > On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
> >> Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a
> >> companion chip designed to provide secure and low power vision capability
> >> to IA platforms. IVSC is available in existing commercial platforms from
> >> multiple OEMs.
> >>
> >> The primary use case of IVSC is to bring in context awareness. IVSC
> >> interfaces directly with the platform main camera sensor via a CSI-2 link
> >> and processes the image data with the embedded AI engine. The detected
> >> events are sent over I2C to ISH (Intel Sensor Hub) for additional data
> >> fusion from multiple sensors. The fusion results are used to implement
> >> advanced use cases like:
> >> - Face detection to unlock screen
> >> - Detect user presence to manage backlight setting or waking up system
> >
> > Do you have plan to support these features in the ivsc driver in the
> > future ?
> >
> >> Since the Image Processing Unit(IPU) used on the host processor needs to
> >> configure the CSI-2 link in normal camera usages, the CSI-2 link and
> >> camera sensor can only be used in mutually-exclusive ways by host IPU and
> >> IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU
> >> driver can take ownership of the CSI-2 link and camera sensor using
> >> interfaces provided by this IVSC driver.
> >>
> >> Switching ownership requires an interface with two different hardware
> >> modules inside IVSC. The software interface to these modules is via Intel
> >> MEI (The Intel Management Engine) commands. These two hardware modules
> >> have two different MEI UUIDs to enumerate. These hardware modules are:
> >> - ACE (Algorithm Context Engine): This module is for algorithm computing
> >> when IVSC owns camera sensor. Also ACE module controls camera sensor's
> >> ownership. This hardware module is used to set ownership of camera sensor.
> >> - CSI (Camera Serial Interface): This module is used to route camera
> >> sensor data either to IVSC or to host for IPU driver and application.
> >>
> >> IVSC also provides a privacy mode. When privacy mode is turned on,
> >> camera sensor can't be used. This means that both ACE and host IPU can't
> >> get image data. And when this mode is turned on, host IPU driver is
> >> informed via a registered callback, so that user can be notified.
> >
> > How does the privacy mode work, and how can the user trust that the
> > closed-source IVSC and IME firmwares will honour the privacy settings ?
> >
>
> Continue with question from Laurent,
>
> How IVSC handle the privacy request from user? Is there some notification
> mechanism to user-space? I'd have concern if IVSC driver need private callback
> to request back-end(e.g. ISP driver) to handle stream cutting.
How does the privacy mode work, does it just pass zeroes (or other dummy
data) towards the host or nothing?
A V4L2 control can be used for the purpose of passing the information to
the user space at least.
--
Regards,
Sakari Ailus
Hi, Sakari,
On 2/16/23 9:12 PM, Sakari Ailus wrote:
> Hi Bingbu, Wentong,
>
> On Wed, Feb 15, 2023 at 08:09:50PM +0800, Bingbu Cao wrote:
>>
>> Hi, Wentong,
>>
>> On 2/15/23 5:43 PM, Laurent Pinchart wrote:
>>> Hello Wentong,
>>>
>>> On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
>>>> Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a
>>>> companion chip designed to provide secure and low power vision capability
>>>> to IA platforms. IVSC is available in existing commercial platforms from
>>>> multiple OEMs.
>>>>
>>>> The primary use case of IVSC is to bring in context awareness. IVSC
>>>> interfaces directly with the platform main camera sensor via a CSI-2 link
>>>> and processes the image data with the embedded AI engine. The detected
>>>> events are sent over I2C to ISH (Intel Sensor Hub) for additional data
>>>> fusion from multiple sensors. The fusion results are used to implement
>>>> advanced use cases like:
>>>> - Face detection to unlock screen
>>>> - Detect user presence to manage backlight setting or waking up system
>>>
>>> Do you have plan to support these features in the ivsc driver in the
>>> future ?
>>>
>>>> Since the Image Processing Unit(IPU) used on the host processor needs to
>>>> configure the CSI-2 link in normal camera usages, the CSI-2 link and
>>>> camera sensor can only be used in mutually-exclusive ways by host IPU and
>>>> IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU
>>>> driver can take ownership of the CSI-2 link and camera sensor using
>>>> interfaces provided by this IVSC driver.
>>>>
>>>> Switching ownership requires an interface with two different hardware
>>>> modules inside IVSC. The software interface to these modules is via Intel
>>>> MEI (The Intel Management Engine) commands. These two hardware modules
>>>> have two different MEI UUIDs to enumerate. These hardware modules are:
>>>> - ACE (Algorithm Context Engine): This module is for algorithm computing
>>>> when IVSC owns camera sensor. Also ACE module controls camera sensor's
>>>> ownership. This hardware module is used to set ownership of camera sensor.
>>>> - CSI (Camera Serial Interface): This module is used to route camera
>>>> sensor data either to IVSC or to host for IPU driver and application.
>>>>
>>>> IVSC also provides a privacy mode. When privacy mode is turned on,
>>>> camera sensor can't be used. This means that both ACE and host IPU can't
>>>> get image data. And when this mode is turned on, host IPU driver is
>>>> informed via a registered callback, so that user can be notified.
>>>
>>> How does the privacy mode work, and how can the user trust that the
>>> closed-source IVSC and IME firmwares will honour the privacy settings ?
As I know, without IVSC, once user enable the privacy mode, the Intel
Converged Security Engine will configure the IPU camera mask (security
register), which will mask the specific CSI2 port and produce dummy
imaging data. For the case with IVSC, there is no final solution on Linux
so far I think.
Wentong, is IVSC trying to cut off the stream and then notify user and IPU?
>>>
>>
>> Continue with question from Laurent,
>>
>> How IVSC handle the privacy request from user? Is there some notification
>> mechanism to user-space? I'd have concern if IVSC driver need private callback
>> to request back-end(e.g. ISP driver) to handle stream cutting.
>
> How does the privacy mode work, does it just pass zeroes (or other dummy
> data) towards the host or nothing?
>
> A V4L2 control can be used for the purpose of passing the information to
> the user space at least.
>
--
Best regards,
Bingbu Cao
> -----Original Message-----
> From: Hillf Danton <[email protected]>
> Sent: Wednesday, February 15, 2023 7:54 AM
>
> On Tue, 14 Feb 2023 12:40:12 +0000 Wentong Wu <[email protected]>
> > > -----Original Message-----
> > > From: Hillf Danton <[email protected]>
> > > Sent: Tuesday, February 14, 2023 4:36 PM
> > >=20
> > > On Tue, 14 Feb 2023 06:25:35 +0000 Wentong Wu <[email protected]>
> > > >
> > > > This allows new command(e.g. 'get status' command to check current
> > > > firmware=3D
> > > > status) to be downloaded to firmware for debugging purpose in
> > > > case no resp=3D onse for current command though I have never saw this
> happen.
> > > > =3D20
> > >=20
> > > Thanks for your specification.
> > >=20
> > > Another question, is it likely for the wakeup to come after reinit
> > >becaus=
> > e of for
> > > instance timeout on the waiter side?
> >
> > Thanks, first I want to clarify why we have timeout here, in case
> > firmware = stuck somehow or IVSC device is removed suddenly, calling
> > thread can't be w= akeup if wait forever here.
>
> Fair enough on the waiter side.
> >
> > And firmware gets high priority to handle command and response, also
> > there = is timeout mechanism inside firmware, I think there will be
> > some problem wi= th firmware if driver gets command response after 5s
> > which will be captured= by driver code(if (csi->cmd_response.cmd_id
> > !=3D cmd->cmd_id)) but I have = never saw this happen.
>
> The problem with firmware could be emulated by directly skipping the wait for
> completion and returning ETIMEDOUT, in order to see what will happen if
> wakeup comes later than timeout.
Thanks, but what do you mean "emulated"?
Hi,
Thanks for your review
> -----Original Message-----
> From: Laurent Pinchart <[email protected]>
> Sent: Wednesday, February 15, 2023 5:46 PM
>
> On Wed, Feb 15, 2023 at 10:03:29AM +0100, Hans de Goede wrote:
> > Hi Sakari,
> >
> > On 2/14/23 17:06, Sakari Ailus wrote:
> > > Hi Wentong,
> > >
> > > Thanks for the patchset.
> > >
> > > On Mon, Feb 13, 2023 at 10:23:47AM +0800, Wentong Wu wrote:
> > >> IVSC directly connects to camera sensor on source side, and on
> > >> output side it not only connects ISH via I2C, but also exposes MIPI
> > >> CSI-2 interface to output camera sensor data. IVSC can use the
> > >> camera sensor data to do AI algorithm, and send the results to ISH.
> > >> On the other end, IVSC can share camera sensor to host by routing
> > >> the raw camera sensor data to the exposed MIPI CSI-2 interface. But
> > >> they can not work at the same time, so software APIs are defined to
> > >> sync the ownership.
> > >>
> > >> This commit defines the interfaces between IVSC and camera sensor
> > >> driver in include/linux/ivsc.h. The camera driver controls
> > >> ownership of the CSI-2 link and sensor with the acquire/release
> > >> APIs. When acquiring camera, lane number and link freq are also
> > >> required by IVSC frame router.
> > >
> > > The more I learn about this system, the more I'm inclined to think
> > > this functionality should be exposed as a V4L2 sub-device. IVSC
> > > doesn't really do anything to the data (as long as it directs it
> > > towards the CSI-2 receiver in the SoC), but it is definitely part of the image
> pipeline.
> >
> > Yes I happened to discuss this exact same thing with Laurent at FOSDEM
> > and we also came to the conclusion that the IVSC chip should be
> > modeled as a V4L2 sub-device.
>
> Agreed.
Thanks for your quick review and conclusion, I'm fresh to media sub-system, is there any convention that I should follow to upstream this kind of v4l2 sub-device driver so that not too much back and forth?
>
> > > I suppose the intended use cases assume a single instance of IVSC
> > > (as well as MEI) but there can, and often are, be multiple camera
> > > sensors in the system. The decision whether to request pass-through
> > > from IVCS can't be done in the camera sensor driver, and should not
> > > be visible to the camera sensor driver. Exposing IVSC as a V4L2
> > > sub-device makes this trivial to address, as the IVSC driver's V4L2
> > > sub-device video s_stream() operation gets called before streaming is started.
> > >
> > > The information whether IVSC is found between the camera sensor and
> > > the host's CSI-2 receiver (IPU in this case) should come from system
> > > firmware and accessed most probably by what is called cio2-bridge at the
> moment.
> > >
> > > The privacy status can be a V4L2 control.
> > >
> > > Also cc Hans.
> > >
> > >>
> > >> Signed-off-by: Wentong Wu <[email protected]>
> > >> ---
> > >> drivers/media/pci/intel/ivsc/Makefile | 1 +
> > >> drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++++++++++++++++++++++++
> > >> include/linux/ivsc.h | 55 ++++++++++++++++++
> > >> 3 files changed, 140 insertions(+) create mode 100644
> > >> drivers/media/pci/intel/ivsc/ivsc.c
> > >>
> > >> diff --git a/drivers/media/pci/intel/ivsc/Makefile
> > >> b/drivers/media/pci/intel/ivsc/Makefile
> > >> index de0a425c22c2..b8b6fc1083be 100644
> > >> --- a/drivers/media/pci/intel/ivsc/Makefile
> > >> +++ b/drivers/media/pci/intel/ivsc/Makefile
> > >> @@ -4,3 +4,4 @@
> > >>
> > >> obj-$(CONFIG_INTEL_VSC) += mei_csi.o
> > >> obj-$(CONFIG_INTEL_VSC) += mei_ace.o
> > >> +obj-$(CONFIG_INTEL_VSC) += ivsc.o
> > >> diff --git a/drivers/media/pci/intel/ivsc/ivsc.c
> > >> b/drivers/media/pci/intel/ivsc/ivsc.c
> > >> new file mode 100644
> > >> index 000000000000..12996b587639
> > >> --- /dev/null
> > >> +++ b/drivers/media/pci/intel/ivsc/ivsc.c
> > >> @@ -0,0 +1,84 @@
> > >> +// SPDX-License-Identifier: GPL-2.0-only
> > >> +/*
> > >> + * Copyright (C) 2023 Intel Corporation. All rights reserved.
> > >> + * Intel Visual Sensing Controller interface */
> > >> +
> > >> +#include <linux/delay.h>
> > >> +#include <linux/ivsc.h>
> > >> +#include <linux/module.h>
> > >> +#include <linux/mutex.h>
> > >> +
> > >> +#include "mei_ace.h"
> > >> +#include "mei_csi.h"
> > >> +
> > >> +/* lock for ivsc APIs */
> > >> +static DEFINE_MUTEX(ivsc_mutex);
> > >> +
> > >> +int ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
> > >> + void (*callback)(void *, enum ivsc_privacy_status),
> > >> + void *context)
> > >> +{
> > >> + int ret;
> > >> +
> > >> + mutex_lock(&ivsc_mutex);
> > >> +
> > >> + /* switch camera sensor ownership to host */
> > >> + ret = ace_set_camera_owner(ACE_CAMERA_HOST);
> > >> + if (ret)
> > >> + goto error;
> > >> +
> > >> + /* switch CSI-2 link to host */
> > >> + ret = csi_set_link_owner(CSI_LINK_HOST, callback, context);
> > >> + if (ret)
> > >> + goto release_camera;
> > >> +
> > >> + /* configure CSI-2 link */
> > >> + ret = csi_set_link_cfg(nr_of_lanes, link_freq);
> > >> + if (ret)
> > >> + goto release_csi;
> > >> +
> > >> + mutex_unlock(&ivsc_mutex);
> > >> +
> > >> + return 0;
> > >> +
> > >> +release_csi:
> > >> + csi_set_link_owner(CSI_LINK_IVSC, NULL, NULL);
> > >> +
> > >> +release_camera:
> > >> + ace_set_camera_owner(ACE_CAMERA_IVSC);
> > >> +
> > >> +error:
> > >> + mutex_unlock(&ivsc_mutex);
> > >> +
> > >> + return ret;
> > >> +}
> > >> +EXPORT_SYMBOL_GPL(ivsc_acquire_camera);
> > >> +
> > >> +int ivsc_release_camera(void)
> > >> +{
> > >> + int ret;
> > >> +
> > >> + mutex_lock(&ivsc_mutex);
> > >> +
> > >> + /* switch CSI-2 link to IVSC */
> > >> + ret = csi_set_link_owner(CSI_LINK_IVSC, NULL, NULL);
> > >> + if (ret)
> > >> + goto error;
> > >> +
> > >> + /* switch camera sensor ownership to IVSC */
> > >> + ret = ace_set_camera_owner(ACE_CAMERA_IVSC);
> > >> +
> > >> +error:
> > >> + mutex_unlock(&ivsc_mutex);
> > >> +
> > >> + return ret;
> > >> +}
> > >> +EXPORT_SYMBOL_GPL(ivsc_release_camera);
> > >> +
> > >> +MODULE_AUTHOR("Wentong Wu <[email protected]>");
> > >> +MODULE_AUTHOR("Zhifeng Wang <[email protected]>");
> > >> +MODULE_SOFTDEP("pre: mei_csi mei_ace");
> MODULE_DESCRIPTION("IVSC
> > >> +interface"); MODULE_LICENSE("GPL"); MODULE_IMPORT_NS(IVSC);
> > >> diff --git a/include/linux/ivsc.h b/include/linux/ivsc.h index
> > >> 6572ca4f340c..bc9006cd6efc 100644
> > >> --- a/include/linux/ivsc.h
> > >> +++ b/include/linux/ivsc.h
> > >> @@ -16,4 +16,59 @@ enum ivsc_privacy_status {
> > >> IVSC_PRIVACY_MAX,
> > >> };
> > >>
> > >> +#if IS_ENABLED(CONFIG_INTEL_VSC)
> > >> +/*
> > >> + * @brief Acquire camera sensor ownership to host and setup
> > >> + * the CSI-2 link between host and IVSC
> > >> + *
> > >> + * IVSC provides a privacy mode. When the privacy mode is turned
> > >> + * on, camera sensor can't be used. This means that both IVSC and
> > >> + * host Image Processing Unit(IPU) can't get image data. And when
> > >> + * this mode is turned on, host Image Processing Unit(IPU) driver
> > >> + * is informed via the registered callback, so that user can be
> > >> + * notified.
> > >> + *
> > >> + * @param nr_of_lanes Number of data lanes used on the CSI-2 link
> > >> + * @param link_freq Frequency of the CSI-2 link
> > >> + * @param callback The pointer of privacy callback function
> > >> + * @param context Privacy callback function runtime context
> > >> + *
> > >> + * @retval 0 If success
> > >> + * @retval -EIO IO error
> > >> + * @retval -EINVAL Invalid argument
> > >> + * @retval -EAGAIN IVSC device not ready
> > >> + * @retval negative values for other errors */ int
> > >> +ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
> > >> + void (*callback)(void *, enum ivsc_privacy_status),
> > >> + void *context);
> > >> +
> > >> +/*
> > >> + * @brief Release camera sensor ownership and stop the CSI-2
> > >> + * link between host and IVSC
> > >> + *
> > >> + * @retval 0 If success
> > >> + * @retval -EIO IO error
> > >> + * @retval -EINVAL Invalid argument
> > >> + * @retval -EAGAIN IVSC device not ready
> > >> + * @retval negative values for other errors */ int
> > >> +ivsc_release_camera(void);
> > >> +
> > >> +#else
> > >> +static inline
> > >> +int ivsc_acquire_camera(u32 nr_of_lanes, u64 link_freq,
> > >> + void (*callback)(void *, enum ivsc_privacy_status),
> > >> + void *context)
> > >> +{
> > >> + return 0;
> > >> +}
> > >> +
> > >> +static inline int ivsc_release_camera(void) {
> > >> + return 0;
> > >> +}
> > >> +
> > >> +#endif
> > >> +
> > >> #endif
>
> --
> Regards,
>
> Laurent Pinchart
> -----Original Message-----
> From: Laurent Pinchart <[email protected]>
> Sent: Wednesday, February 15, 2023 5:43 PM
>
> Hello Wentong,
>
> On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
> > Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a
> > companion chip designed to provide secure and low power vision
> > capability to IA platforms. IVSC is available in existing commercial
> > platforms from multiple OEMs.
> >
> > The primary use case of IVSC is to bring in context awareness. IVSC
> > interfaces directly with the platform main camera sensor via a CSI-2
> > link and processes the image data with the embedded AI engine. The
> > detected events are sent over I2C to ISH (Intel Sensor Hub) for
> > additional data fusion from multiple sensors. The fusion results are
> > used to implement advanced use cases like:
> > - Face detection to unlock screen
> > - Detect user presence to manage backlight setting or waking up
> > system
>
> Do you have plan to support these features in the ivsc driver in the future ?
Not sure, but the first step is to upstream this driver.
>
> > Since the Image Processing Unit(IPU) used on the host processor needs
> > to configure the CSI-2 link in normal camera usages, the CSI-2 link
> > and camera sensor can only be used in mutually-exclusive ways by host
> > IPU and IVSC. By default the IVSC owns the CSI-2 link and camera
> > sensor. The IPU driver can take ownership of the CSI-2 link and camera
> > sensor using interfaces provided by this IVSC driver.
> >
> > Switching ownership requires an interface with two different hardware
> > modules inside IVSC. The software interface to these modules is via
> > Intel MEI (The Intel Management Engine) commands. These two hardware
> > modules have two different MEI UUIDs to enumerate. These hardware
> modules are:
> > - ACE (Algorithm Context Engine): This module is for algorithm
> > computing when IVSC owns camera sensor. Also ACE module controls
> > camera sensor's ownership. This hardware module is used to set ownership of
> camera sensor.
> > - CSI (Camera Serial Interface): This module is used to route camera
> > sensor data either to IVSC or to host for IPU driver and application.
> >
> > IVSC also provides a privacy mode. When privacy mode is turned on,
> > camera sensor can't be used. This means that both ACE and host IPU
> > can't get image data. And when this mode is turned on, host IPU driver
> > is informed via a registered callback, so that user can be notified.
>
> How does the privacy mode work, and how can the user trust that the closed-
> source IVSC and IME firmwares will honour the privacy settings ?
No camera data will be allowed to go through IVSC, and then there will be no data on IVSC CSI transmitter side.
>
> > In summary, to acquire ownership of camera by IPU driver, first ACE
> > module needs to be informed of ownership and then to setup MIPI CSI-2
> > link for the camera sensor and IPU.
> >
> > Implementation:
> > There are two different drivers to handle ACE and CSI hardware modules
> > inside IVSC.
> > - mei_csi: MEI client driver to send commands and receive
> > notifications from CSI module.
> > - mei_ace: MEI client driver to send commands and get status from ACE
> > module.
> > Interface is exposed via ivsc.h to acquire and release camera sensor
> > and
> > CSI-2 link.
>
> Do I understand correctly, from your diagram below, that the communication
> between the IME and IVSC goes through SPI ?
>
> > Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
> > ----------------------------------------------------------------------
> > -------
> > | Host Processor |
> > | |
> > | ----------------- ----------------- --------------- |
> > | | | | | | | I2C |
> > | | IPU | | ISH | |camera driver|--| |
> > | | | | | | | | |
> > | ----------------- ----------------- --------------- | |
> > | | | | | |
> > | | | --------------- | |
> > | | | | | | |
> > | | | | IVSC driver | | |
> > | | | | | | |
> > | | | --------------- | |
> > | | | | | |
> > ----------------|-----------------------|----------------------|---------|---
> > | CSI | I2C |SPI |
> > | | | |
> > ----------------|-----------------------|---------------- | |
> > | IVSC | | | |
> > | | | | |
> > | ----------------- ----------------- | | |
> > | | | | | | | |
> > | | CSI | | ACE | |------| |
> > | | | | | | |
> > | ----------------- ----------------- | |
> > | | | I2C | |
> > ----------------|-----------------------|---------------- |
> > | CSI | |
> > | | |
> > -------------------------------- |
> > | | I2C |
> > | camera sensor |-----------------------------|
> > | |
> > --------------------------------
> >
> > Wentong Wu (3):
> > media: pci: intel: ivsc: Add CSI submodule
> > media: pci: intel: ivsc: Add ACE submodule
> > media: pci: intel: ivsc: Add acquire/release API for ivsc
> >
> > drivers/media/pci/Kconfig | 1 +
> > drivers/media/pci/intel/Makefile | 2 +
> > drivers/media/pci/intel/ivsc/Kconfig | 12 +
> > drivers/media/pci/intel/ivsc/Makefile | 7 +
> > drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++
> > drivers/media/pci/intel/ivsc/mei_ace.c | 472
> > +++++++++++++++++++++++++ drivers/media/pci/intel/ivsc/mei_ace.h |
> > 36 ++ drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++
> > drivers/media/pci/intel/ivsc/mei_csi.h | 60 ++++
> > include/linux/ivsc.h | 74 ++++
> > 10 files changed, 1090 insertions(+)
> > create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
> > create mode 100644 drivers/media/pci/intel/ivsc/Makefile
> > create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
> > create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
> > create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
> > create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
> > create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
> > create mode 100644 include/linux/ivsc.h
>
> --
> Regards,
>
> Laurent Pinchart
> -----Original Message-----
> From: Bingbu Cao <[email protected]>
> Sent: Friday, February 17, 2023 9:44 AM
>
> Hi, Sakari,
>
> On 2/16/23 9:12 PM, Sakari Ailus wrote:
> > Hi Bingbu, Wentong,
> >
> > On Wed, Feb 15, 2023 at 08:09:50PM +0800, Bingbu Cao wrote:
> >>
> >> Hi, Wentong,
> >>
> >> On 2/15/23 5:43 PM, Laurent Pinchart wrote:
> >>> Hello Wentong,
> >>>
> >>> On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
> >>>> Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls",
> >>>> is a companion chip designed to provide secure and low power vision
> >>>> capability to IA platforms. IVSC is available in existing
> >>>> commercial platforms from multiple OEMs.
> >>>>
> >>>> The primary use case of IVSC is to bring in context awareness. IVSC
> >>>> interfaces directly with the platform main camera sensor via a
> >>>> CSI-2 link and processes the image data with the embedded AI
> >>>> engine. The detected events are sent over I2C to ISH (Intel Sensor
> >>>> Hub) for additional data fusion from multiple sensors. The fusion
> >>>> results are used to implement advanced use cases like:
> >>>> - Face detection to unlock screen
> >>>> - Detect user presence to manage backlight setting or waking up
> >>>> system
> >>>
> >>> Do you have plan to support these features in the ivsc driver in the
> >>> future ?
> >>>
> >>>> Since the Image Processing Unit(IPU) used on the host processor
> >>>> needs to configure the CSI-2 link in normal camera usages, the
> >>>> CSI-2 link and camera sensor can only be used in mutually-exclusive
> >>>> ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link
> >>>> and camera sensor. The IPU driver can take ownership of the CSI-2
> >>>> link and camera sensor using interfaces provided by this IVSC driver.
> >>>>
> >>>> Switching ownership requires an interface with two different
> >>>> hardware modules inside IVSC. The software interface to these
> >>>> modules is via Intel MEI (The Intel Management Engine) commands.
> >>>> These two hardware modules have two different MEI UUIDs to enumerate.
> These hardware modules are:
> >>>> - ACE (Algorithm Context Engine): This module is for algorithm
> >>>> computing when IVSC owns camera sensor. Also ACE module controls
> >>>> camera sensor's ownership. This hardware module is used to set ownership
> of camera sensor.
> >>>> - CSI (Camera Serial Interface): This module is used to route
> >>>> camera sensor data either to IVSC or to host for IPU driver and application.
> >>>>
> >>>> IVSC also provides a privacy mode. When privacy mode is turned on,
> >>>> camera sensor can't be used. This means that both ACE and host IPU
> >>>> can't get image data. And when this mode is turned on, host IPU
> >>>> driver is informed via a registered callback, so that user can be notified.
> >>>
> >>> How does the privacy mode work, and how can the user trust that the
> >>> closed-source IVSC and IME firmwares will honour the privacy settings ?
>
> As I know, without IVSC, once user enable the privacy mode, the Intel
> Converged Security Engine will configure the IPU camera mask (security register),
> which will mask the specific CSI2 port and produce dummy imaging data. For the
> case with IVSC, there is no final solution on Linux so far I think.
>
> Wentong, is IVSC trying to cut off the stream and then notify user and IPU?
yes
>
> >>>
> >>
> >> Continue with question from Laurent,
> >>
> >> How IVSC handle the privacy request from user? Is there some
> >> notification mechanism to user-space?
IVSC has already defined privacy callback for host IPU/camera driver.
> > I'd have concern if IVSC driver
> >> need private callback to request back-end(e.g. ISP driver) to handle stream
> cutting.
> >
> > How does the privacy mode work, does it just pass zeroes (or other
> > dummy
> > data) towards the host or nothing?
No data on CSI transmitter side
> >
> > A V4L2 control can be used for the purpose of passing the information
> > to the user space at least.
I will take some time to review V4L2 sub-device and control mechanism, and then update the driver.
BR,
Wentong
> >
>
> --
> Best regards,
> Bingbu Cao
> -----Original Message-----
> From: Hillf Danton <[email protected]>
> Sent: Friday, February 17, 2023 2:28 PM
>
> On Fri, 17 Feb 2023 05:52:38 +0000 Wentong Wu <[email protected]>
> >
> > Thanks, but what do you mean "emulated"?=20
> >
> Construct a scenario that ensures wakeup comes late because of no wait for
> completion. And we can see what will happen with the current test cases.
Thanks, I will remove reinit_completion
>
> > static int mei_csi_send(u8 *buf, size_t len)
> > +{
> > + struct csi_cmd *cmd = (struct csi_cmd *)buf;
> > + int ret;
> > +
> > + reinit_completion(&csi->cmd_completion);
> > +
> > + ret = mei_cldev_send(csi->cldev, buf, len);
> > + if (ret < 0)
> > + goto out;
>
> return -ETIMEDOUT;
>
> > +
> > + ret = wait_for_completion_killable_timeout(&csi->cmd_completion,
> > + CSI_CMD_TIMEOUT);
> > + if (ret < 0) {
> > + goto out;
> > + } else if (!ret) {
> > + ret = -ETIMEDOUT;
> > + goto out;
> > + }
> > +
> > + /* command response status */
> > + ret = csi->cmd_response.status;
> > + if (ret) {
> > + ret = -EINVAL;
> > + goto out;
> > + }
> > +
> > + if (csi->cmd_response.cmd_id != cmd->cmd_id)
> > + ret = -EINVAL;
> > +
> > +out:
> > + return ret;
> > +}
Hi Wentong,
On Fri, Feb 17, 2023 at 06:28:32AM +0000, Wu, Wentong wrote:
>
>
> > -----Original Message-----
> > From: Bingbu Cao <[email protected]>
> > Sent: Friday, February 17, 2023 9:44 AM
> >
> > Hi, Sakari,
> >
> > On 2/16/23 9:12 PM, Sakari Ailus wrote:
> > > Hi Bingbu, Wentong,
> > >
> > > On Wed, Feb 15, 2023 at 08:09:50PM +0800, Bingbu Cao wrote:
> > >>
> > >> Hi, Wentong,
> > >>
> > >> On 2/15/23 5:43 PM, Laurent Pinchart wrote:
> > >>> Hello Wentong,
> > >>>
> > >>> On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
> > >>>> Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls",
> > >>>> is a companion chip designed to provide secure and low power vision
> > >>>> capability to IA platforms. IVSC is available in existing
> > >>>> commercial platforms from multiple OEMs.
> > >>>>
> > >>>> The primary use case of IVSC is to bring in context awareness. IVSC
> > >>>> interfaces directly with the platform main camera sensor via a
> > >>>> CSI-2 link and processes the image data with the embedded AI
> > >>>> engine. The detected events are sent over I2C to ISH (Intel Sensor
> > >>>> Hub) for additional data fusion from multiple sensors. The fusion
> > >>>> results are used to implement advanced use cases like:
> > >>>> - Face detection to unlock screen
> > >>>> - Detect user presence to manage backlight setting or waking up
> > >>>> system
> > >>>
> > >>> Do you have plan to support these features in the ivsc driver in the
> > >>> future ?
> > >>>
> > >>>> Since the Image Processing Unit(IPU) used on the host processor
> > >>>> needs to configure the CSI-2 link in normal camera usages, the
> > >>>> CSI-2 link and camera sensor can only be used in mutually-exclusive
> > >>>> ways by host IPU and IVSC. By default the IVSC owns the CSI-2 link
> > >>>> and camera sensor. The IPU driver can take ownership of the CSI-2
> > >>>> link and camera sensor using interfaces provided by this IVSC driver.
> > >>>>
> > >>>> Switching ownership requires an interface with two different
> > >>>> hardware modules inside IVSC. The software interface to these
> > >>>> modules is via Intel MEI (The Intel Management Engine) commands.
> > >>>> These two hardware modules have two different MEI UUIDs to enumerate.
> > These hardware modules are:
> > >>>> - ACE (Algorithm Context Engine): This module is for algorithm
> > >>>> computing when IVSC owns camera sensor. Also ACE module controls
> > >>>> camera sensor's ownership. This hardware module is used to set ownership
> > of camera sensor.
> > >>>> - CSI (Camera Serial Interface): This module is used to route
> > >>>> camera sensor data either to IVSC or to host for IPU driver and application.
> > >>>>
> > >>>> IVSC also provides a privacy mode. When privacy mode is turned on,
> > >>>> camera sensor can't be used. This means that both ACE and host IPU
> > >>>> can't get image data. And when this mode is turned on, host IPU
> > >>>> driver is informed via a registered callback, so that user can be notified.
> > >>>
> > >>> How does the privacy mode work, and how can the user trust that the
> > >>> closed-source IVSC and IME firmwares will honour the privacy settings ?
> >
> > As I know, without IVSC, once user enable the privacy mode, the Intel
> > Converged Security Engine will configure the IPU camera mask (security register),
> > which will mask the specific CSI2 port and produce dummy imaging data. For the
> > case with IVSC, there is no final solution on Linux so far I think.
> >
> > Wentong, is IVSC trying to cut off the stream and then notify user and IPU?
>
> yes
Does the CSI-2 transmitter on IVCS go to some LP mode during this time, or
does the receiver need to initialise the bus again when the stream resuems?
>
> >
> > >>>
> > >>
> > >> Continue with question from Laurent,
> > >>
> > >> How IVSC handle the privacy request from user? Is there some
> > >> notification mechanism to user-space?
>
> IVSC has already defined privacy callback for host IPU/camera driver.
>
> > > I'd have concern if IVSC driver
> > >> need private callback to request back-end(e.g. ISP driver) to handle stream
> > cutting.
> > >
> > > How does the privacy mode work, does it just pass zeroes (or other
> > > dummy
> > > data) towards the host or nothing?
>
> No data on CSI transmitter side
Can it stop in the middle of the frame? Or is it guaranteed to produce full
frames (assuming the sensor does)?
>
> > >
> > > A V4L2 control can be used for the purpose of passing the information
> > > to the user space at least.
>
> I will take some time to review V4L2 sub-device and control mechanism,
> and then update the driver.
--
Kind regards,
Sakari Ailus
Hello Wentong,
On Fri, Feb 17, 2023 at 06:20:10AM +0000, Wu, Wentong wrote:
> On Wednesday, February 15, 2023 5:43 PM, Laurent Pinchart wrote:
> > On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
> > > Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a
> > > companion chip designed to provide secure and low power vision
> > > capability to IA platforms. IVSC is available in existing commercial
> > > platforms from multiple OEMs.
> > >
> > > The primary use case of IVSC is to bring in context awareness. IVSC
> > > interfaces directly with the platform main camera sensor via a CSI-2
> > > link and processes the image data with the embedded AI engine. The
> > > detected events are sent over I2C to ISH (Intel Sensor Hub) for
> > > additional data fusion from multiple sensors. The fusion results are
> > > used to implement advanced use cases like:
> > > - Face detection to unlock screen
> > > - Detect user presence to manage backlight setting or waking up
> > > system
> >
> > Do you have plan to support these features in the ivsc driver in the future ?
>
> Not sure, but the first step is to upstream this driver.
Sure, no problem.
> > > Since the Image Processing Unit(IPU) used on the host processor needs
> > > to configure the CSI-2 link in normal camera usages, the CSI-2 link
> > > and camera sensor can only be used in mutually-exclusive ways by host
> > > IPU and IVSC. By default the IVSC owns the CSI-2 link and camera
> > > sensor. The IPU driver can take ownership of the CSI-2 link and camera
> > > sensor using interfaces provided by this IVSC driver.
> > >
> > > Switching ownership requires an interface with two different hardware
> > > modules inside IVSC. The software interface to these modules is via
> > > Intel MEI (The Intel Management Engine) commands. These two hardware
> > > modules have two different MEI UUIDs to enumerate. These hardware
> > > modules are:
> > > - ACE (Algorithm Context Engine): This module is for algorithm
> > > computing when IVSC owns camera sensor. Also ACE module controls
> > > camera sensor's ownership. This hardware module is used to set ownership of
> > > camera sensor.
> > > - CSI (Camera Serial Interface): This module is used to route camera
> > > sensor data either to IVSC or to host for IPU driver and application.
> > >
> > > IVSC also provides a privacy mode. When privacy mode is turned on,
> > > camera sensor can't be used. This means that both ACE and host IPU
> > > can't get image data. And when this mode is turned on, host IPU driver
> > > is informed via a registered callback, so that user can be notified.
> >
> > How does the privacy mode work, and how can the user trust that the closed-
> > source IVSC and IME firmwares will honour the privacy settings ?
>
> No camera data will be allowed to go through IVSC, and then there will
> be no data on IVSC CSI transmitter side.
But how can I be sure that the IVSC will not use the camera behind my
back, if it's all controlled through a closed-source firmware ?
> > > In summary, to acquire ownership of camera by IPU driver, first ACE
> > > module needs to be informed of ownership and then to setup MIPI CSI-2
> > > link for the camera sensor and IPU.
> > >
> > > Implementation:
> > > There are two different drivers to handle ACE and CSI hardware modules
> > > inside IVSC.
> > > - mei_csi: MEI client driver to send commands and receive notifications from CSI module.
> > > - mei_ace: MEI client driver to send commands and get status from ACE module.
> > > Interface is exposed via ivsc.h to acquire and release camera sensor and
> > > CSI-2 link.
> >
> > Do I understand correctly, from your diagram below, that the communication
> > between the IME and IVSC goes through SPI ?
> >
> > > Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
> > > -----------------------------------------------------------------------------
> > > | Host Processor |
> > > | |
> > > | ----------------- ----------------- --------------- |
> > > | | | | | | | I2C |
> > > | | IPU | | ISH | |camera driver|--| |
> > > | | | | | | | | |
> > > | ----------------- ----------------- --------------- | |
> > > | | | | | |
> > > | | | --------------- | |
> > > | | | | | | |
> > > | | | | IVSC driver | | |
> > > | | | | | | |
> > > | | | --------------- | |
> > > | | | | | |
> > > ----------------|-----------------------|----------------------|---------|---
> > > | CSI | I2C |SPI |
> > > | | | |
> > > ----------------|-----------------------|---------------- | |
> > > | IVSC | | | |
> > > | | | | |
> > > | ----------------- ----------------- | | |
> > > | | | | | | | |
> > > | | CSI | | ACE | |------| |
> > > | | | | | | |
> > > | ----------------- ----------------- | |
> > > | | | I2C | |
> > > ----------------|-----------------------|---------------- |
> > > | CSI | |
> > > | | |
> > > -------------------------------- |
> > > | | I2C |
> > > | camera sensor |-----------------------------|
> > > | |
> > > --------------------------------
> > >
> > > Wentong Wu (3):
> > > media: pci: intel: ivsc: Add CSI submodule
> > > media: pci: intel: ivsc: Add ACE submodule
> > > media: pci: intel: ivsc: Add acquire/release API for ivsc
> > >
> > > drivers/media/pci/Kconfig | 1 +
> > > drivers/media/pci/intel/Makefile | 2 +
> > > drivers/media/pci/intel/ivsc/Kconfig | 12 +
> > > drivers/media/pci/intel/ivsc/Makefile | 7 +
> > > drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++
> > > drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++
> > > drivers/media/pci/intel/ivsc/mei_ace.h |> 36 ++
> > > drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++
> > > drivers/media/pci/intel/ivsc/mei_csi.h | 60 ++++
> > > include/linux/ivsc.h | 74 ++++
> > > 10 files changed, 1090 insertions(+)
> > > create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
> > > create mode 100644 drivers/media/pci/intel/ivsc/Makefile
> > > create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
> > > create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
> > > create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
> > > create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
> > > create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
> > > create mode 100644 include/linux/ivsc.h
--
Regards,
Laurent Pinchart
Hi Wentong,
On Fri, Feb 17, 2023 at 06:10:22AM +0000, Wu, Wentong wrote:
> On Sent: Wednesday, February 15, 2023 5:46 PM, Laurent Pinchart wrote:
> > On Wed, Feb 15, 2023 at 10:03:29AM +0100, Hans de Goede wrote:
> > > On 2/14/23 17:06, Sakari Ailus wrote:
> > > > On Mon, Feb 13, 2023 at 10:23:47AM +0800, Wentong Wu wrote:
> > > >> IVSC directly connects to camera sensor on source side, and on
> > > >> output side it not only connects ISH via I2C, but also exposes MIPI
> > > >> CSI-2 interface to output camera sensor data. IVSC can use the
> > > >> camera sensor data to do AI algorithm, and send the results to ISH.
> > > >> On the other end, IVSC can share camera sensor to host by routing
> > > >> the raw camera sensor data to the exposed MIPI CSI-2 interface. But
> > > >> they can not work at the same time, so software APIs are defined to
> > > >> sync the ownership.
> > > >>
> > > >> This commit defines the interfaces between IVSC and camera sensor
> > > >> driver in include/linux/ivsc.h. The camera driver controls
> > > >> ownership of the CSI-2 link and sensor with the acquire/release
> > > >> APIs. When acquiring camera, lane number and link freq are also
> > > >> required by IVSC frame router.
> > > >
> > > > The more I learn about this system, the more I'm inclined to think
> > > > this functionality should be exposed as a V4L2 sub-device. IVSC
> > > > doesn't really do anything to the data (as long as it directs it
> > > > towards the CSI-2 receiver in the SoC), but it is definitely
> > > > part of the image pipeline.
> > >
> > > Yes I happened to discuss this exact same thing with Laurent at FOSDEM
> > > and we also came to the conclusion that the IVSC chip should be
> > > modeled as a V4L2 sub-device.
> >
> > Agreed.
>
> Thanks for your quick review and conclusion, I'm fresh to media
> sub-system, is there any convention that I should follow to upstream
> this kind of v4l2 sub-device driver so that not too much back and
> forth?
This is a tentative proposal, as I'm not very familiar with the hardware
architecture:
- The subdev should have two pads, a sink pad connected to the camera
sensor, and a source pad connected to the CSI-2 receiver in the IPU.
- As for any new driver, the subdev driver should use the active state
managed by the v4l2-subdev core. This requires calling
v4l2_subdev_init_finalize() at probe time. See
https://git.linuxtv.org/media_tree.git/commit/?id=a2514b9a634a for an
example of a subdev driver converted to this mechanism.
- As we're talking about CSI-2, the subdev driver should use the streams
API that was recently merged, and in particular support the
.get_frame_desc(), .set_routing(), .enable_streams() and
.disable_streams() operations.
- I don't see a need to support V4L2 controls in the subdev driver, but
I may be missing something.
- The driver should be validated with v4l2-compliance, part of
v4l-utils.
> > > > I suppose the intended use cases assume a single instance of IVSC
> > > > (as well as MEI) but there can, and often are, be multiple camera
> > > > sensors in the system. The decision whether to request pass-through
> > > > from IVCS can't be done in the camera sensor driver, and should not
> > > > be visible to the camera sensor driver. Exposing IVSC as a V4L2
> > > > sub-device makes this trivial to address, as the IVSC driver's V4L2
> > > > sub-device video s_stream() operation gets called before streaming is started.
> > > >
> > > > The information whether IVSC is found between the camera sensor and
> > > > the host's CSI-2 receiver (IPU in this case) should come from system
> > > > firmware and accessed most probably by what is called cio2-bridge at
> > > > the moment.
> > > >
> > > > The privacy status can be a V4L2 control.
> > > >
> > > > Also cc Hans.
> > > >
> > > >> Signed-off-by: Wentong Wu <[email protected]>
> > > >> ---
> > > >> drivers/media/pci/intel/ivsc/Makefile | 1 +
> > > >> drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++++++++++++++++++++++++
> > > >> include/linux/ivsc.h | 55 ++++++++++++++++++
> > > >> 3 files changed, 140 insertions(+) create mode 100644
> > > >> drivers/media/pci/intel/ivsc/ivsc.c
[snip]
--
Regards,
Laurent Pinchart
Hi Sakari,
> -----Original Message-----
> From: Sakari Ailus <[email protected]>
> Sent: Friday, February 17, 2023 6:50 PM
>
> Hi Wentong,
>
> On Fri, Feb 17, 2023 at 06:28:32AM +0000, Wu, Wentong wrote:
> >
> >
> > > -----Original Message-----
> > > From: Bingbu Cao <[email protected]>
> > > Sent: Friday, February 17, 2023 9:44 AM
> > >
> > > Hi, Sakari,
> > >
> > > On 2/16/23 9:12 PM, Sakari Ailus wrote:
> > > > Hi Bingbu, Wentong,
> > > >
> > > > On Wed, Feb 15, 2023 at 08:09:50PM +0800, Bingbu Cao wrote:
> > > >>
> > > >> Hi, Wentong,
> > > >>
> > > >> On 2/15/23 5:43 PM, Laurent Pinchart wrote:
> > > >>> Hello Wentong,
> > > >>>
> > > >>> On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
> > > >>>> Intel Visual Sensing Controller (IVSC), codenamed "Clover
> > > >>>> Falls", is a companion chip designed to provide secure and low
> > > >>>> power vision capability to IA platforms. IVSC is available in
> > > >>>> existing commercial platforms from multiple OEMs.
> > > >>>>
> > > >>>> The primary use case of IVSC is to bring in context awareness.
> > > >>>> IVSC interfaces directly with the platform main camera sensor
> > > >>>> via a
> > > >>>> CSI-2 link and processes the image data with the embedded AI
> > > >>>> engine. The detected events are sent over I2C to ISH (Intel
> > > >>>> Sensor
> > > >>>> Hub) for additional data fusion from multiple sensors. The
> > > >>>> fusion results are used to implement advanced use cases like:
> > > >>>> - Face detection to unlock screen
> > > >>>> - Detect user presence to manage backlight setting or waking
> > > >>>> up system
> > > >>>
> > > >>> Do you have plan to support these features in the ivsc driver in
> > > >>> the future ?
> > > >>>
> > > >>>> Since the Image Processing Unit(IPU) used on the host processor
> > > >>>> needs to configure the CSI-2 link in normal camera usages, the
> > > >>>> CSI-2 link and camera sensor can only be used in
> > > >>>> mutually-exclusive ways by host IPU and IVSC. By default the
> > > >>>> IVSC owns the CSI-2 link and camera sensor. The IPU driver can
> > > >>>> take ownership of the CSI-2 link and camera sensor using interfaces
> provided by this IVSC driver.
> > > >>>>
> > > >>>> Switching ownership requires an interface with two different
> > > >>>> hardware modules inside IVSC. The software interface to these
> > > >>>> modules is via Intel MEI (The Intel Management Engine) commands.
> > > >>>> These two hardware modules have two different MEI UUIDs to
> enumerate.
> > > These hardware modules are:
> > > >>>> - ACE (Algorithm Context Engine): This module is for algorithm
> > > >>>> computing when IVSC owns camera sensor. Also ACE module
> > > >>>> controls camera sensor's ownership. This hardware module is
> > > >>>> used to set ownership
> > > of camera sensor.
> > > >>>> - CSI (Camera Serial Interface): This module is used to route
> > > >>>> camera sensor data either to IVSC or to host for IPU driver and
> application.
> > > >>>>
> > > >>>> IVSC also provides a privacy mode. When privacy mode is turned
> > > >>>> on, camera sensor can't be used. This means that both ACE and
> > > >>>> host IPU can't get image data. And when this mode is turned on,
> > > >>>> host IPU driver is informed via a registered callback, so that user can be
> notified.
> > > >>>
> > > >>> How does the privacy mode work, and how can the user trust that
> > > >>> the closed-source IVSC and IME firmwares will honour the privacy
> settings ?
> > >
> > > As I know, without IVSC, once user enable the privacy mode, the
> > > Intel Converged Security Engine will configure the IPU camera mask
> > > (security register), which will mask the specific CSI2 port and
> > > produce dummy imaging data. For the case with IVSC, there is no final
> solution on Linux so far I think.
> > >
> > > Wentong, is IVSC trying to cut off the stream and then notify user and IPU?
> >
> > yes
>
> Does the CSI-2 transmitter on IVCS go to some LP mode during this time,
Yes, The low power mode is following the D-Phy spec.
> or does
> the receiver need to initialise the bus again when the stream resuems?
No, it's the same link.
>
> >
> > >
> > > >>>
> > > >>
> > > >> Continue with question from Laurent,
> > > >>
> > > >> How IVSC handle the privacy request from user? Is there some
> > > >> notification mechanism to user-space?
> >
> > IVSC has already defined privacy callback for host IPU/camera driver.
> >
> > > > I'd have concern if IVSC driver
> > > >> need private callback to request back-end(e.g. ISP driver) to
> > > >> handle stream
> > > cutting.
> > > >
> > > > How does the privacy mode work, does it just pass zeroes (or other
> > > > dummy
> > > > data) towards the host or nothing?
> >
> > No data on CSI transmitter side
>
> Can it stop in the middle of the frame?
No,
Or is it guaranteed to produce full frames
> (assuming the sensor does)?
Yes, full frame will be guaranteed as camera sensor does.
>
> >
> > > >
> > > > A V4L2 control can be used for the purpose of passing the
> > > > information to the user space at least.
> >
> > I will take some time to review V4L2 sub-device and control mechanism,
> > and then update the driver.
>
> --
> Kind regards,
>
> Sakari Ailus
> -----Original Message-----
> From: Laurent Pinchart <[email protected]>
> Sent: Friday, February 17, 2023 7:19 PM
>
> Hi Wentong,
>
> On Fri, Feb 17, 2023 at 06:10:22AM +0000, Wu, Wentong wrote:
> > On Sent: Wednesday, February 15, 2023 5:46 PM, Laurent Pinchart wrote:
> > > On Wed, Feb 15, 2023 at 10:03:29AM +0100, Hans de Goede wrote:
> > > > On 2/14/23 17:06, Sakari Ailus wrote:
> > > > > On Mon, Feb 13, 2023 at 10:23:47AM +0800, Wentong Wu wrote:
> > > > >> IVSC directly connects to camera sensor on source side, and on
> > > > >> output side it not only connects ISH via I2C, but also exposes
> > > > >> MIPI
> > > > >> CSI-2 interface to output camera sensor data. IVSC can use the
> > > > >> camera sensor data to do AI algorithm, and send the results to ISH.
> > > > >> On the other end, IVSC can share camera sensor to host by
> > > > >> routing the raw camera sensor data to the exposed MIPI CSI-2
> > > > >> interface. But they can not work at the same time, so software
> > > > >> APIs are defined to sync the ownership.
> > > > >>
> > > > >> This commit defines the interfaces between IVSC and camera
> > > > >> sensor driver in include/linux/ivsc.h. The camera driver
> > > > >> controls ownership of the CSI-2 link and sensor with the
> > > > >> acquire/release APIs. When acquiring camera, lane number and
> > > > >> link freq are also required by IVSC frame router.
> > > > >
> > > > > The more I learn about this system, the more I'm inclined to
> > > > > think this functionality should be exposed as a V4L2 sub-device.
> > > > > IVSC doesn't really do anything to the data (as long as it
> > > > > directs it towards the CSI-2 receiver in the SoC), but it is
> > > > > definitely part of the image pipeline.
> > > >
> > > > Yes I happened to discuss this exact same thing with Laurent at
> > > > FOSDEM and we also came to the conclusion that the IVSC chip
> > > > should be modeled as a V4L2 sub-device.
> > >
> > > Agreed.
> >
> > Thanks for your quick review and conclusion, I'm fresh to media
> > sub-system, is there any convention that I should follow to upstream
> > this kind of v4l2 sub-device driver so that not too much back and
> > forth?
>
> This is a tentative proposal, as I'm not very familiar with the hardware
> architecture:
>
> - The subdev should have two pads, a sink pad connected to the camera
> sensor, and a source pad connected to the CSI-2 receiver in the IPU.
>
> - As for any new driver, the subdev driver should use the active state
> managed by the v4l2-subdev core. This requires calling
> v4l2_subdev_init_finalize() at probe time. See
> https://git.linuxtv.org/media_tree.git/commit/?id=a2514b9a634a for an
> example of a subdev driver converted to this mechanism.
>
> - As we're talking about CSI-2, the subdev driver should use the streams
> API that was recently merged, and in particular support the
> .get_frame_desc(), .set_routing(), .enable_streams() and
> .disable_streams() operations.
>
> - I don't see a need to support V4L2 controls in the subdev driver, but
> I may be missing something.
>
> - The driver should be validated with v4l2-compliance, part of
> v4l-utils.
>
Thanks for the detail, but I have one more question, during probe of sensor(v4l2-sudev) driver, it will configure camera sensor connected to IVSC via I2C, but before that it should acquire camera sensor's ownership from IVSC, how v4l2 framework guarantee this?
Thanks,
Wentong
> > > > > I suppose the intended use cases assume a single instance of
> > > > > IVSC (as well as MEI) but there can, and often are, be multiple
> > > > > camera sensors in the system. The decision whether to request
> > > > > pass-through from IVCS can't be done in the camera sensor
> > > > > driver, and should not be visible to the camera sensor driver.
> > > > > Exposing IVSC as a V4L2 sub-device makes this trivial to
> > > > > address, as the IVSC driver's V4L2 sub-device video s_stream() operation
> gets called before streaming is started.
> > > > >
> > > > > The information whether IVSC is found between the camera sensor
> > > > > and the host's CSI-2 receiver (IPU in this case) should come
> > > > > from system firmware and accessed most probably by what is
> > > > > called cio2-bridge at the moment.
> > > > >
> > > > > The privacy status can be a V4L2 control.
> > > > >
> > > > > Also cc Hans.
> > > > >
> > > > >> Signed-off-by: Wentong Wu <[email protected]>
> > > > >> ---
> > > > >> drivers/media/pci/intel/ivsc/Makefile | 1 +
> > > > >> drivers/media/pci/intel/ivsc/ivsc.c | 84
> +++++++++++++++++++++++++++
> > > > >> include/linux/ivsc.h | 55 ++++++++++++++++++
> > > > >> 3 files changed, 140 insertions(+) create mode 100644
> > > > >> drivers/media/pci/intel/ivsc/ivsc.c
>
> [snip]
>
> --
> Regards,
>
> Laurent Pinchart
> -----Original Message-----
> From: Laurent Pinchart <[email protected]>
> Sent: Friday, February 17, 2023 7:12 PM
>
> Hello Wentong,
>
> On Fri, Feb 17, 2023 at 06:20:10AM +0000, Wu, Wentong wrote:
> > On Wednesday, February 15, 2023 5:43 PM, Laurent Pinchart wrote:
> > > On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
> > > > Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls",
> > > > is a companion chip designed to provide secure and low power
> > > > vision capability to IA platforms. IVSC is available in existing
> > > > commercial platforms from multiple OEMs.
> > > >
> > > > The primary use case of IVSC is to bring in context awareness.
> > > > IVSC interfaces directly with the platform main camera sensor via
> > > > a CSI-2 link and processes the image data with the embedded AI
> > > > engine. The detected events are sent over I2C to ISH (Intel Sensor
> > > > Hub) for additional data fusion from multiple sensors. The fusion
> > > > results are used to implement advanced use cases like:
> > > > - Face detection to unlock screen
> > > > - Detect user presence to manage backlight setting or waking up
> > > > system
> > >
> > > Do you have plan to support these features in the ivsc driver in the future ?
> >
> > Not sure, but the first step is to upstream this driver.
>
> Sure, no problem.
>
> > > > Since the Image Processing Unit(IPU) used on the host processor
> > > > needs to configure the CSI-2 link in normal camera usages, the
> > > > CSI-2 link and camera sensor can only be used in
> > > > mutually-exclusive ways by host IPU and IVSC. By default the IVSC
> > > > owns the CSI-2 link and camera sensor. The IPU driver can take
> > > > ownership of the CSI-2 link and camera sensor using interfaces provided by
> this IVSC driver.
> > > >
> > > > Switching ownership requires an interface with two different
> > > > hardware modules inside IVSC. The software interface to these
> > > > modules is via Intel MEI (The Intel Management Engine) commands.
> > > > These two hardware modules have two different MEI UUIDs to
> > > > enumerate. These hardware modules are:
> > > > - ACE (Algorithm Context Engine): This module is for algorithm
> > > > computing when IVSC owns camera sensor. Also ACE module controls
> > > > camera sensor's ownership. This hardware module is used to set
> > > > ownership of camera sensor.
> > > > - CSI (Camera Serial Interface): This module is used to route
> > > > camera sensor data either to IVSC or to host for IPU driver and application.
> > > >
> > > > IVSC also provides a privacy mode. When privacy mode is turned on,
> > > > camera sensor can't be used. This means that both ACE and host IPU
> > > > can't get image data. And when this mode is turned on, host IPU
> > > > driver is informed via a registered callback, so that user can be notified.
> > >
> > > How does the privacy mode work, and how can the user trust that the
> > > closed- source IVSC and IME firmwares will honour the privacy settings ?
> >
> > No camera data will be allowed to go through IVSC, and then there will
> > be no data on IVSC CSI transmitter side.
>
> But how can I be sure that the IVSC will not use the camera behind my back, if
> it's all controlled through a closed-source firmware ?
Actually I don't know how to answer your question, but this is guaranteed though we have no plan to open-source the firmware.
>
> > > > In summary, to acquire ownership of camera by IPU driver, first
> > > > ACE module needs to be informed of ownership and then to setup
> > > > MIPI CSI-2 link for the camera sensor and IPU.
> > > >
> > > > Implementation:
> > > > There are two different drivers to handle ACE and CSI hardware
> > > > modules inside IVSC.
> > > > - mei_csi: MEI client driver to send commands and receive notifications
> from CSI module.
> > > > - mei_ace: MEI client driver to send commands and get status from ACE
> module.
> > > > Interface is exposed via ivsc.h to acquire and release camera
> > > > sensor and
> > > > CSI-2 link.
> > >
> > > Do I understand correctly, from your diagram below, that the
> > > communication between the IME and IVSC goes through SPI ?
> > >
> > > > Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
> > > > ------------------------------------------------------------------
> > > > -----------
> > > > | Host Processor |
> > > > | |
> > > > | ----------------- ----------------- --------------- |
> > > > | | | | | | | I2C |
> > > > | | IPU | | ISH | |camera driver|--| |
> > > > | | | | | | | | |
> > > > | ----------------- ----------------- --------------- | |
> > > > | | | | | |
> > > > | | | --------------- | |
> > > > | | | | | | |
> > > > | | | | IVSC driver | | |
> > > > | | | | | | |
> > > > | | | --------------- | |
> > > > | | | | | |
> > > > ----------------|-----------------------|----------------------|---------|---
> > > > | CSI | I2C |SPI |
> > > > | | | |
> > > > ----------------|-----------------------|---------------- | |
> > > > | IVSC | | | |
> > > > | | | | |
> > > > | ----------------- ----------------- | | |
> > > > | | | | | | | |
> > > > | | CSI | | ACE | |------| |
> > > > | | | | | | |
> > > > | ----------------- ----------------- | |
> > > > | | | I2C | |
> > > > ----------------|-----------------------|---------------- |
> > > > | CSI | |
> > > > | | |
> > > > -------------------------------- |
> > > > | | I2C |
> > > > | camera sensor |-----------------------------|
> > > > | |
> > > > --------------------------------
> > > >
> > > > Wentong Wu (3):
> > > > media: pci: intel: ivsc: Add CSI submodule
> > > > media: pci: intel: ivsc: Add ACE submodule
> > > > media: pci: intel: ivsc: Add acquire/release API for ivsc
> > > >
> > > > drivers/media/pci/Kconfig | 1 +
> > > > drivers/media/pci/intel/Makefile | 2 +
> > > > drivers/media/pci/intel/ivsc/Kconfig | 12 +
> > > > drivers/media/pci/intel/ivsc/Makefile | 7 +
> > > > drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++
> > > > drivers/media/pci/intel/ivsc/mei_ace.c | 472
> > > > +++++++++++++++++++++++++ drivers/media/pci/intel/ivsc/mei_ace.h
> > > > |> 36 ++ drivers/media/pci/intel/ivsc/mei_csi.c | 342
> > > > ++++++++++++++++++ drivers/media/pci/intel/ivsc/mei_csi.h | 60 ++++
> > > > include/linux/ivsc.h | 74 ++++
> > > > 10 files changed, 1090 insertions(+) create mode 100644
> > > > drivers/media/pci/intel/ivsc/Kconfig
> > > > create mode 100644 drivers/media/pci/intel/ivsc/Makefile
> > > > create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
> > > > create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
> > > > create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
> > > > create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
> > > > create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
> > > > create mode 100644 include/linux/ivsc.h
>
> --
> Regards,
>
> Laurent Pinchart
Hi Wentong,
On Mon, Feb 20, 2023 at 03:50:55AM +0000, Wu, Wentong wrote:
>
>
> > -----Original Message-----
> > From: Laurent Pinchart <[email protected]>
> > Sent: Friday, February 17, 2023 7:19 PM
> >
> > Hi Wentong,
> >
> > On Fri, Feb 17, 2023 at 06:10:22AM +0000, Wu, Wentong wrote:
> > > On Sent: Wednesday, February 15, 2023 5:46 PM, Laurent Pinchart wrote:
> > > > On Wed, Feb 15, 2023 at 10:03:29AM +0100, Hans de Goede wrote:
> > > > > On 2/14/23 17:06, Sakari Ailus wrote:
> > > > > > On Mon, Feb 13, 2023 at 10:23:47AM +0800, Wentong Wu wrote:
> > > > > >> IVSC directly connects to camera sensor on source side, and on
> > > > > >> output side it not only connects ISH via I2C, but also exposes
> > > > > >> MIPI
> > > > > >> CSI-2 interface to output camera sensor data. IVSC can use the
> > > > > >> camera sensor data to do AI algorithm, and send the results to ISH.
> > > > > >> On the other end, IVSC can share camera sensor to host by
> > > > > >> routing the raw camera sensor data to the exposed MIPI CSI-2
> > > > > >> interface. But they can not work at the same time, so software
> > > > > >> APIs are defined to sync the ownership.
> > > > > >>
> > > > > >> This commit defines the interfaces between IVSC and camera
> > > > > >> sensor driver in include/linux/ivsc.h. The camera driver
> > > > > >> controls ownership of the CSI-2 link and sensor with the
> > > > > >> acquire/release APIs. When acquiring camera, lane number and
> > > > > >> link freq are also required by IVSC frame router.
> > > > > >
> > > > > > The more I learn about this system, the more I'm inclined to
> > > > > > think this functionality should be exposed as a V4L2 sub-device.
> > > > > > IVSC doesn't really do anything to the data (as long as it
> > > > > > directs it towards the CSI-2 receiver in the SoC), but it is
> > > > > > definitely part of the image pipeline.
> > > > >
> > > > > Yes I happened to discuss this exact same thing with Laurent at
> > > > > FOSDEM and we also came to the conclusion that the IVSC chip
> > > > > should be modeled as a V4L2 sub-device.
> > > >
> > > > Agreed.
> > >
> > > Thanks for your quick review and conclusion, I'm fresh to media
> > > sub-system, is there any convention that I should follow to upstream
> > > this kind of v4l2 sub-device driver so that not too much back and
> > > forth?
> >
> > This is a tentative proposal, as I'm not very familiar with the hardware
> > architecture:
> >
> > - The subdev should have two pads, a sink pad connected to the camera
> > sensor, and a source pad connected to the CSI-2 receiver in the IPU.
> >
> > - As for any new driver, the subdev driver should use the active state
> > managed by the v4l2-subdev core. This requires calling
> > v4l2_subdev_init_finalize() at probe time. See
> > https://git.linuxtv.org/media_tree.git/commit/?id=a2514b9a634a for an
> > example of a subdev driver converted to this mechanism.
> >
> > - As we're talking about CSI-2, the subdev driver should use the streams
> > API that was recently merged, and in particular support the
> > .get_frame_desc(), .set_routing(), .enable_streams() and
> > .disable_streams() operations.
> >
> > - I don't see a need to support V4L2 controls in the subdev driver, but
> > I may be missing something.
> >
> > - The driver should be validated with v4l2-compliance, part of
> > v4l-utils.
> >
>
> Thanks for the detail, but I have one more question, during probe of
> sensor(v4l2-sudev) driver, it will configure camera sensor connected to
> IVSC via I2C, but before that it should acquire camera sensor's ownership
> from IVSC, how v4l2 framework guarantee this?
Please wrap the lines at 74 characters or so when replying.
Do you mean accessing the sensor via I?C also requires acquiring the sensor
from IVSC?
--
Regards,
Sakari Ailus
> -----Original Message-----
> From: Sakari Ailus <[email protected]>
> Sent: Monday, February 20, 2023 4:37 PM
>
> Hi Wentong,
>
> On Mon, Feb 20, 2023 at 03:50:55AM +0000, Wu, Wentong wrote:
> >
> >
> > > -----Original Message-----
> > > From: Laurent Pinchart <[email protected]>
> > > Sent: Friday, February 17, 2023 7:19 PM
> > >
> > > Hi Wentong,
> > >
> > > On Fri, Feb 17, 2023 at 06:10:22AM +0000, Wu, Wentong wrote:
> > > > On Sent: Wednesday, February 15, 2023 5:46 PM, Laurent Pinchart wrote:
> > > > > On Wed, Feb 15, 2023 at 10:03:29AM +0100, Hans de Goede wrote:
> > > > > > On 2/14/23 17:06, Sakari Ailus wrote:
> > > > > > > On Mon, Feb 13, 2023 at 10:23:47AM +0800, Wentong Wu wrote:
> > > > > > >> IVSC directly connects to camera sensor on source side, and
> > > > > > >> on output side it not only connects ISH via I2C, but also
> > > > > > >> exposes MIPI
> > > > > > >> CSI-2 interface to output camera sensor data. IVSC can use
> > > > > > >> the camera sensor data to do AI algorithm, and send the results to
> ISH.
> > > > > > >> On the other end, IVSC can share camera sensor to host by
> > > > > > >> routing the raw camera sensor data to the exposed MIPI
> > > > > > >> CSI-2 interface. But they can not work at the same time, so
> > > > > > >> software APIs are defined to sync the ownership.
> > > > > > >>
> > > > > > >> This commit defines the interfaces between IVSC and camera
> > > > > > >> sensor driver in include/linux/ivsc.h. The camera driver
> > > > > > >> controls ownership of the CSI-2 link and sensor with the
> > > > > > >> acquire/release APIs. When acquiring camera, lane number
> > > > > > >> and link freq are also required by IVSC frame router.
> > > > > > >
> > > > > > > The more I learn about this system, the more I'm inclined to
> > > > > > > think this functionality should be exposed as a V4L2 sub-device.
> > > > > > > IVSC doesn't really do anything to the data (as long as it
> > > > > > > directs it towards the CSI-2 receiver in the SoC), but it is
> > > > > > > definitely part of the image pipeline.
> > > > > >
> > > > > > Yes I happened to discuss this exact same thing with Laurent
> > > > > > at FOSDEM and we also came to the conclusion that the IVSC
> > > > > > chip should be modeled as a V4L2 sub-device.
> > > > >
> > > > > Agreed.
> > > >
> > > > Thanks for your quick review and conclusion, I'm fresh to media
> > > > sub-system, is there any convention that I should follow to
> > > > upstream this kind of v4l2 sub-device driver so that not too much
> > > > back and forth?
> > >
> > > This is a tentative proposal, as I'm not very familiar with the
> > > hardware
> > > architecture:
> > >
> > > - The subdev should have two pads, a sink pad connected to the camera
> > > sensor, and a source pad connected to the CSI-2 receiver in the IPU.
> > >
> > > - As for any new driver, the subdev driver should use the active state
> > > managed by the v4l2-subdev core. This requires calling
> > > v4l2_subdev_init_finalize() at probe time. See
> > > https://git.linuxtv.org/media_tree.git/commit/?id=a2514b9a634a for an
> > > example of a subdev driver converted to this mechanism.
> > >
> > > - As we're talking about CSI-2, the subdev driver should use the streams
> > > API that was recently merged, and in particular support the
> > > .get_frame_desc(), .set_routing(), .enable_streams() and
> > > .disable_streams() operations.
> > >
> > > - I don't see a need to support V4L2 controls in the subdev driver, but
> > > I may be missing something.
> > >
> > > - The driver should be validated with v4l2-compliance, part of
> > > v4l-utils.
> > >
> >
> > Thanks for the detail, but I have one more question, during probe of
> > sensor(v4l2-sudev) driver, it will configure camera sensor connected
> > to IVSC via I2C, but before that it should acquire camera sensor's
> > ownership from IVSC, how v4l2 framework guarantee this?
>
> Please wrap the lines at 74 characters or so when replying.
Thanks, I will.
>
> Do you mean accessing the sensor via I?C also requires acquiring the sensor
> from IVSC?
Yes
>
> --
> Regards,
>
> Sakari Ailus
Hi,
On 2/20/23 09:57, Wu, Wentong wrote:
>
>
>> -----Original Message-----
>> From: Sakari Ailus <[email protected]>
>> Sent: Monday, February 20, 2023 4:37 PM
>>
>> Hi Wentong,
>>
>> On Mon, Feb 20, 2023 at 03:50:55AM +0000, Wu, Wentong wrote:
>>>
>>>
>>>> -----Original Message-----
>>>> From: Laurent Pinchart <[email protected]>
>>>> Sent: Friday, February 17, 2023 7:19 PM
>>>>
>>>> Hi Wentong,
>>>>
>>>> On Fri, Feb 17, 2023 at 06:10:22AM +0000, Wu, Wentong wrote:
>>>>> On Sent: Wednesday, February 15, 2023 5:46 PM, Laurent Pinchart wrote:
>>>>>> On Wed, Feb 15, 2023 at 10:03:29AM +0100, Hans de Goede wrote:
>>>>>>> On 2/14/23 17:06, Sakari Ailus wrote:
>>>>>>>> On Mon, Feb 13, 2023 at 10:23:47AM +0800, Wentong Wu wrote:
>>>>>>>>> IVSC directly connects to camera sensor on source side, and
>>>>>>>>> on output side it not only connects ISH via I2C, but also
>>>>>>>>> exposes MIPI
>>>>>>>>> CSI-2 interface to output camera sensor data. IVSC can use
>>>>>>>>> the camera sensor data to do AI algorithm, and send the results to
>> ISH.
>>>>>>>>> On the other end, IVSC can share camera sensor to host by
>>>>>>>>> routing the raw camera sensor data to the exposed MIPI
>>>>>>>>> CSI-2 interface. But they can not work at the same time, so
>>>>>>>>> software APIs are defined to sync the ownership.
>>>>>>>>>
>>>>>>>>> This commit defines the interfaces between IVSC and camera
>>>>>>>>> sensor driver in include/linux/ivsc.h. The camera driver
>>>>>>>>> controls ownership of the CSI-2 link and sensor with the
>>>>>>>>> acquire/release APIs. When acquiring camera, lane number
>>>>>>>>> and link freq are also required by IVSC frame router.
>>>>>>>>
>>>>>>>> The more I learn about this system, the more I'm inclined to
>>>>>>>> think this functionality should be exposed as a V4L2 sub-device.
>>>>>>>> IVSC doesn't really do anything to the data (as long as it
>>>>>>>> directs it towards the CSI-2 receiver in the SoC), but it is
>>>>>>>> definitely part of the image pipeline.
>>>>>>>
>>>>>>> Yes I happened to discuss this exact same thing with Laurent
>>>>>>> at FOSDEM and we also came to the conclusion that the IVSC
>>>>>>> chip should be modeled as a V4L2 sub-device.
>>>>>>
>>>>>> Agreed.
>>>>>
>>>>> Thanks for your quick review and conclusion, I'm fresh to media
>>>>> sub-system, is there any convention that I should follow to
>>>>> upstream this kind of v4l2 sub-device driver so that not too much
>>>>> back and forth?
>>>>
>>>> This is a tentative proposal, as I'm not very familiar with the
>>>> hardware
>>>> architecture:
>>>>
>>>> - The subdev should have two pads, a sink pad connected to the camera
>>>> sensor, and a source pad connected to the CSI-2 receiver in the IPU.
>>>>
>>>> - As for any new driver, the subdev driver should use the active state
>>>> managed by the v4l2-subdev core. This requires calling
>>>> v4l2_subdev_init_finalize() at probe time. See
>>>> https://git.linuxtv.org/media_tree.git/commit/?id=a2514b9a634a for an
>>>> example of a subdev driver converted to this mechanism.
>>>>
>>>> - As we're talking about CSI-2, the subdev driver should use the streams
>>>> API that was recently merged, and in particular support the
>>>> .get_frame_desc(), .set_routing(), .enable_streams() and
>>>> .disable_streams() operations.
>>>>
>>>> - I don't see a need to support V4L2 controls in the subdev driver, but
>>>> I may be missing something.
>>>>
>>>> - The driver should be validated with v4l2-compliance, part of
>>>> v4l-utils.
>>>>
>>>
>>> Thanks for the detail, but I have one more question, during probe of
>>> sensor(v4l2-sudev) driver, it will configure camera sensor connected
>>> to IVSC via I2C, but before that it should acquire camera sensor's
>>> ownership from IVSC, how v4l2 framework guarantee this?
>>
>> Please wrap the lines at 74 characters or so when replying.
>
> Thanks, I will.
>
>>
>> Do you mean accessing the sensor via I²C also requires acquiring the sensor
>> from IVSC?
>
> Yes
Hmm, that is going to be a problem since we really want to have
independent driver for the 2 which are not aware of each other.
I think that maybe we can model this part of the ivsc functionality
as an i2c-mux. But then we will need to somehow change the parent
of the i2c device for the sensor to the output of this mux ...
I guess this means adding some code (some hack likely) to
drivers/platform/x86/intel/int3472/discrete.c which' proe function
is already guaranteed to run before the sensor's i2c-client gets
instantiated, because of the ACPI _DEP on the INT3472 ACPI device
in the DSDT.
This is going to be a tricky problem to tackle.
Regards,
Hans
Hi Sakari,
few questions as switching to v4l2 sub-dev framework.
> -----Original Message-----
> From: Sakari Ailus <[email protected]>
> Sent: Wednesday, February 15, 2023 12:06 AM
>
> Hi Wentong,
>
> Thanks for the patchset.
>
> On Mon, Feb 13, 2023 at 10:23:47AM +0800, Wentong Wu wrote:
> > IVSC directly connects to camera sensor on source side, and on output
> > side it not only connects ISH via I2C, but also exposes MIPI CSI-2
> > interface to output camera sensor data. IVSC can use the camera sensor
> > data to do AI algorithm, and send the results to ISH. On the other
> > end, IVSC can share camera sensor to host by routing the raw camera
> > sensor data to the exposed MIPI CSI-2 interface. But they can not work
> > at the same time, so software APIs are defined to sync the ownership.
> >
> > This commit defines the interfaces between IVSC and camera sensor
> > driver in include/linux/ivsc.h. The camera driver controls ownership
> > of the CSI-2 link and sensor with the acquire/release APIs. When
> > acquiring camera, lane number and link freq are also required by IVSC
> > frame router.
>
> The more I learn about this system, the more I'm inclined to think this
> functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do
> anything to the data (as long as it directs it towards the CSI-2 receiver in the
> SoC), but it is definitely part of the image pipeline.
>
> I suppose the intended use cases assume a single instance of IVSC (as well as
> MEI) but there can, and often are, be multiple camera sensors in the system. The
> decision whether to request pass-through from IVCS can't be done in the camera
> sensor driver, and should not be visible to the camera sensor driver. Exposing
> IVSC as a V4L2 sub-device makes this trivial to address, as the IVSC driver's V4L2
> sub-device video s_stream() operation gets called before streaming is started.
>
> The information whether IVSC is found between the camera sensor and the
> host's CSI-2 receiver (IPU in this case) should come from system firmware and
> accessed most probably by what is called cio2-bridge at the moment.
>
> The privacy status can be a V4L2 control.
This should be a control or event? If control, how user-space handle privacy stuff?
For the required link freq and lane number, is v4l2 control the correct way to configure
them? If yes, seems there is no CID value for them so that we should custom some
CID value(link freqm, lane number, and privacy) for ivsc in linux/v4l2-controls.h, is this
acceptable?
Thanks,
Wentong
>
> Also cc Hans.
>
> >
Hi Laurent,
> -----Original Message-----
> From: Laurent Pinchart <[email protected]>
> Sent: Friday, February 17, 2023 7:19 PM
>
> Hi Wentong,
>
> On Fri, Feb 17, 2023 at 06:10:22AM +0000, Wu, Wentong wrote:
> > On Sent: Wednesday, February 15, 2023 5:46 PM, Laurent Pinchart wrote:
> > > On Wed, Feb 15, 2023 at 10:03:29AM +0100, Hans de Goede wrote:
> > > > On 2/14/23 17:06, Sakari Ailus wrote:
> > > > > On Mon, Feb 13, 2023 at 10:23:47AM +0800, Wentong Wu wrote:
> > > > >> IVSC directly connects to camera sensor on source side, and on
> > > > >> output side it not only connects ISH via I2C, but also exposes
> > > > >> MIPI
> > > > >> CSI-2 interface to output camera sensor data. IVSC can use the
> > > > >> camera sensor data to do AI algorithm, and send the results to ISH.
> > > > >> On the other end, IVSC can share camera sensor to host by
> > > > >> routing the raw camera sensor data to the exposed MIPI CSI-2
> > > > >> interface. But they can not work at the same time, so software
> > > > >> APIs are defined to sync the ownership.
> > > > >>
> > > > >> This commit defines the interfaces between IVSC and camera
> > > > >> sensor driver in include/linux/ivsc.h. The camera driver
> > > > >> controls ownership of the CSI-2 link and sensor with the
> > > > >> acquire/release APIs. When acquiring camera, lane number and
> > > > >> link freq are also required by IVSC frame router.
> > > > >
> > > > > The more I learn about this system, the more I'm inclined to
> > > > > think this functionality should be exposed as a V4L2 sub-device.
> > > > > IVSC doesn't really do anything to the data (as long as it
> > > > > directs it towards the CSI-2 receiver in the SoC), but it is
> > > > > definitely part of the image pipeline.
> > > >
> > > > Yes I happened to discuss this exact same thing with Laurent at
> > > > FOSDEM and we also came to the conclusion that the IVSC chip
> > > > should be modeled as a V4L2 sub-device.
> > >
> > > Agreed.
> >
> > Thanks for your quick review and conclusion, I'm fresh to media
> > sub-system, is there any convention that I should follow to upstream
> > this kind of v4l2 sub-device driver so that not too much back and
> > forth?
>
> This is a tentative proposal, as I'm not very familiar with the hardware
> architecture:
>
> - The subdev should have two pads, a sink pad connected to the camera
> sensor, and a source pad connected to the CSI-2 receiver in the IPU.
>
> - As for any new driver, the subdev driver should use the active state
> managed by the v4l2-subdev core. This requires calling
> v4l2_subdev_init_finalize() at probe time. See
> https://git.linuxtv.org/media_tree.git/commit/?id=a2514b9a634a for an
> example of a subdev driver converted to this mechanism.
>
> - As we're talking about CSI-2, the subdev driver should use the streams
> API that was recently merged, and in particular support the
> .get_frame_desc(), .set_routing(), .enable_streams() and
> .disable_streams() operations.
>
> - I don't see a need to support V4L2 controls in the subdev driver, but
> I may be missing something.
>
> - The driver should be validated with v4l2-compliance, part of
> v4l-utils.
Thanks, one more question, because ivsc just routes the received data from
sensor and no format convert, how we handle the get_fmt and set_fmt callback
for ivsc?
Thanks
Wentong
>
> > > > > I suppose the intended use cases assume a single instance of
> > > > > IVSC (as well as MEI) but there can, and often are, be multiple
> > > > > camera sensors in the system. The decision whether to request
> > > > > pass-through from IVCS can't be done in the camera sensor
> > > > > driver, and should not be visible to the camera sensor driver.
> > > > > Exposing IVSC as a V4L2 sub-device makes this trivial to
> > > > > address, as the IVSC driver's V4L2 sub-device video s_stream() operation
> gets called before streaming is started.
> > > > >
> > > > > The information whether IVSC is found between the camera sensor
> > > > > and the host's CSI-2 receiver (IPU in this case) should come
> > > > > from system firmware and accessed most probably by what is
> > > > > called cio2-bridge at the moment.
> > > > >
> > > > > The privacy status can be a V4L2 control.
> > > > >
> > > > > Also cc Hans.
> > > > >
> > > > >> Signed-off-by: Wentong Wu <[email protected]>
> > > > >> ---
> > > > >> drivers/media/pci/intel/ivsc/Makefile | 1 +
> > > > >> drivers/media/pci/intel/ivsc/ivsc.c | 84
> +++++++++++++++++++++++++++
> > > > >> include/linux/ivsc.h | 55 ++++++++++++++++++
> > > > >> 3 files changed, 140 insertions(+) create mode 100644
> > > > >> drivers/media/pci/intel/ivsc/ivsc.c
>
> [snip]
>
> --
> Regards,
>
> Laurent Pinchart
Hi Wentong,
On Tue, Feb 28, 2023 at 06:35:41AM +0000, Wu, Wentong wrote:
> Hi Sakari,
>
> few questions as switching to v4l2 sub-dev framework.
>
> > -----Original Message-----
> > From: Sakari Ailus <[email protected]>
> > Sent: Wednesday, February 15, 2023 12:06 AM
> >
> > Hi Wentong,
> >
> > Thanks for the patchset.
> >
> > On Mon, Feb 13, 2023 at 10:23:47AM +0800, Wentong Wu wrote:
> > > IVSC directly connects to camera sensor on source side, and on output
> > > side it not only connects ISH via I2C, but also exposes MIPI CSI-2
> > > interface to output camera sensor data. IVSC can use the camera sensor
> > > data to do AI algorithm, and send the results to ISH. On the other
> > > end, IVSC can share camera sensor to host by routing the raw camera
> > > sensor data to the exposed MIPI CSI-2 interface. But they can not work
> > > at the same time, so software APIs are defined to sync the ownership.
> > >
> > > This commit defines the interfaces between IVSC and camera sensor
> > > driver in include/linux/ivsc.h. The camera driver controls ownership
> > > of the CSI-2 link and sensor with the acquire/release APIs. When
> > > acquiring camera, lane number and link freq are also required by IVSC
> > > frame router.
> >
> > The more I learn about this system, the more I'm inclined to think this
> > functionality should be exposed as a V4L2 sub-device. IVSC doesn't really do
> > anything to the data (as long as it directs it towards the CSI-2 receiver in the
> > SoC), but it is definitely part of the image pipeline.
> >
> > I suppose the intended use cases assume a single instance of IVSC (as well as
> > MEI) but there can, and often are, be multiple camera sensors in the system. The
> > decision whether to request pass-through from IVCS can't be done in the camera
> > sensor driver, and should not be visible to the camera sensor driver. Exposing
> > IVSC as a V4L2 sub-device makes this trivial to address, as the IVSC driver's V4L2
> > sub-device video s_stream() operation gets called before streaming is started.
> >
> > The information whether IVSC is found between the camera sensor and the
> > host's CSI-2 receiver (IPU in this case) should come from system firmware and
> > accessed most probably by what is called cio2-bridge at the moment.
> >
> > The privacy status can be a V4L2 control.
>
> This should be a control or event? If control, how user-space handle
> privacy stuff?
Changing control events generates events for the user space.
<URL:https://hverkuil.home.xs4all.nl/spec/userspace-api/v4l/dev-event.html>
>
> For the required link freq and lane number, is v4l2 control the correct
> way to configure them? If yes, seems there is no CID value for them so
> that we should custom some CID value(link freqm, lane number, and
> privacy) for ivsc in linux/v4l2-controls.h, is this acceptable?
You should obtain these using the V4L2 fwnode interface. Please see e.g.
drivers/media/platform/renesas/rzg2l-cru/rzg2l-csi2.c how that driver
registers a V4L2 async sub-device and a V4L2 async notifier.
--
Kind regards,
Sakari Ailus
On Mon, Feb 20, 2023 at 10:03:08AM +0100, Hans de Goede wrote:
> On 2/20/23 09:57, Wu, Wentong wrote:
> >> -----Original Message-----
> >> From: Sakari Ailus <[email protected]>
> >> Sent: Monday, February 20, 2023 4:37 PM
> >>
> >> Hi Wentong,
> >>
> >> On Mon, Feb 20, 2023 at 03:50:55AM +0000, Wu, Wentong wrote:
> >>>> -----Original Message-----
> >>>> From: Laurent Pinchart <[email protected]>
> >>>> Sent: Friday, February 17, 2023 7:19 PM
> >>>>
> >>>> Hi Wentong,
> >>>>
> >>>> On Fri, Feb 17, 2023 at 06:10:22AM +0000, Wu, Wentong wrote:
> >>>>> On Sent: Wednesday, February 15, 2023 5:46 PM, Laurent Pinchart wrote:
> >>>>>> On Wed, Feb 15, 2023 at 10:03:29AM +0100, Hans de Goede wrote:
> >>>>>>> On 2/14/23 17:06, Sakari Ailus wrote:
> >>>>>>>> On Mon, Feb 13, 2023 at 10:23:47AM +0800, Wentong Wu wrote:
> >>>>>>>>> IVSC directly connects to camera sensor on source side, and
> >>>>>>>>> on output side it not only connects ISH via I2C, but also
> >>>>>>>>> exposes MIPI
> >>>>>>>>> CSI-2 interface to output camera sensor data. IVSC can use
> >>>>>>>>> the camera sensor data to do AI algorithm, and send the results to ISH.
> >>>>>>>>> On the other end, IVSC can share camera sensor to host by
> >>>>>>>>> routing the raw camera sensor data to the exposed MIPI
> >>>>>>>>> CSI-2 interface. But they can not work at the same time, so
> >>>>>>>>> software APIs are defined to sync the ownership.
> >>>>>>>>>
> >>>>>>>>> This commit defines the interfaces between IVSC and camera
> >>>>>>>>> sensor driver in include/linux/ivsc.h. The camera driver
> >>>>>>>>> controls ownership of the CSI-2 link and sensor with the
> >>>>>>>>> acquire/release APIs. When acquiring camera, lane number
> >>>>>>>>> and link freq are also required by IVSC frame router.
> >>>>>>>>
> >>>>>>>> The more I learn about this system, the more I'm inclined to
> >>>>>>>> think this functionality should be exposed as a V4L2 sub-device.
> >>>>>>>> IVSC doesn't really do anything to the data (as long as it
> >>>>>>>> directs it towards the CSI-2 receiver in the SoC), but it is
> >>>>>>>> definitely part of the image pipeline.
> >>>>>>>
> >>>>>>> Yes I happened to discuss this exact same thing with Laurent
> >>>>>>> at FOSDEM and we also came to the conclusion that the IVSC
> >>>>>>> chip should be modeled as a V4L2 sub-device.
> >>>>>>
> >>>>>> Agreed.
> >>>>>
> >>>>> Thanks for your quick review and conclusion, I'm fresh to media
> >>>>> sub-system, is there any convention that I should follow to
> >>>>> upstream this kind of v4l2 sub-device driver so that not too much
> >>>>> back and forth?
> >>>>
> >>>> This is a tentative proposal, as I'm not very familiar with the
> >>>> hardware
> >>>> architecture:
> >>>>
> >>>> - The subdev should have two pads, a sink pad connected to the camera
> >>>> sensor, and a source pad connected to the CSI-2 receiver in the IPU.
> >>>>
> >>>> - As for any new driver, the subdev driver should use the active state
> >>>> managed by the v4l2-subdev core. This requires calling
> >>>> v4l2_subdev_init_finalize() at probe time. See
> >>>> https://git.linuxtv.org/media_tree.git/commit/?id=a2514b9a634a for an
> >>>> example of a subdev driver converted to this mechanism.
> >>>>
> >>>> - As we're talking about CSI-2, the subdev driver should use the streams
> >>>> API that was recently merged, and in particular support the
> >>>> .get_frame_desc(), .set_routing(), .enable_streams() and
> >>>> .disable_streams() operations.
> >>>>
> >>>> - I don't see a need to support V4L2 controls in the subdev driver, but
> >>>> I may be missing something.
> >>>>
> >>>> - The driver should be validated with v4l2-compliance, part of
> >>>> v4l-utils.
> >>>>
> >>>
> >>> Thanks for the detail, but I have one more question, during probe of
> >>> sensor(v4l2-sudev) driver, it will configure camera sensor connected
> >>> to IVSC via I2C, but before that it should acquire camera sensor's
> >>> ownership from IVSC, how v4l2 framework guarantee this?
> >>
> >> Please wrap the lines at 74 characters or so when replying.
> >
> > Thanks, I will.
> >
> >> Do you mean accessing the sensor via I²C also requires acquiring the sensor
> >> from IVSC?
> >
> > Yes
>
> Hmm, that is going to be a problem since we really want to have
> independent driver for the 2 which are not aware of each other.
>
> I think that maybe we can model this part of the ivsc functionality
> as an i2c-mux. But then we will need to somehow change the parent
> of the i2c device for the sensor to the output of this mux ...
>
> I guess this means adding some code (some hack likely) to
> drivers/platform/x86/intel/int3472/discrete.c which' proe function
> is already guaranteed to run before the sensor's i2c-client gets
> instantiated, because of the ACPI _DEP on the INT3472 ACPI device
> in the DSDT.
>
> This is going to be a tricky problem to tackle.
From a framework point of view I would also recommend i2c-mux, but when
it comes to the ACPI integration... I can feel your pain and I'm sorry
about that :-(
--
Regards,
Laurent Pinchart
> -----Original Message-----
> From: Sakari Ailus <[email protected]>
> Sent: Tuesday, February 28, 2023 4:24 PM
>
> Hi Wentong,
>
> On Tue, Feb 28, 2023 at 06:35:41AM +0000, Wu, Wentong wrote:
> > Hi Sakari,
> >
> > few questions as switching to v4l2 sub-dev framework.
> >
> > > -----Original Message-----
> > > From: Sakari Ailus <[email protected]>
> > > Sent: Wednesday, February 15, 2023 12:06 AM
> > >
> > > Hi Wentong,
> > >
> > > Thanks for the patchset.
> > >
> > > On Mon, Feb 13, 2023 at 10:23:47AM +0800, Wentong Wu wrote:
> > > > IVSC directly connects to camera sensor on source side, and on
> > > > output side it not only connects ISH via I2C, but also exposes
> > > > MIPI CSI-2 interface to output camera sensor data. IVSC can use
> > > > the camera sensor data to do AI algorithm, and send the results to
> > > > ISH. On the other end, IVSC can share camera sensor to host by
> > > > routing the raw camera sensor data to the exposed MIPI CSI-2
> > > > interface. But they can not work at the same time, so software APIs are
> defined to sync the ownership.
> > > >
> > > > This commit defines the interfaces between IVSC and camera sensor
> > > > driver in include/linux/ivsc.h. The camera driver controls
> > > > ownership of the CSI-2 link and sensor with the acquire/release
> > > > APIs. When acquiring camera, lane number and link freq are also
> > > > required by IVSC frame router.
> > >
> > > The more I learn about this system, the more I'm inclined to think
> > > this functionality should be exposed as a V4L2 sub-device. IVSC
> > > doesn't really do anything to the data (as long as it directs it
> > > towards the CSI-2 receiver in the SoC), but it is definitely part of the image
> pipeline.
> > >
> > > I suppose the intended use cases assume a single instance of IVSC
> > > (as well as
> > > MEI) but there can, and often are, be multiple camera sensors in the
> > > system. The decision whether to request pass-through from IVCS can't
> > > be done in the camera sensor driver, and should not be visible to
> > > the camera sensor driver. Exposing IVSC as a V4L2 sub-device makes
> > > this trivial to address, as the IVSC driver's V4L2 sub-device video s_stream()
> operation gets called before streaming is started.
> > >
> > > The information whether IVSC is found between the camera sensor and
> > > the host's CSI-2 receiver (IPU in this case) should come from system
> > > firmware and accessed most probably by what is called cio2-bridge at the
> moment.
> > >
> > > The privacy status can be a V4L2 control.
> >
> > This should be a control or event? If control, how user-space handle
> > privacy stuff?
>
> Changing control events generates events for the user space.
>
> <URL:https://hverkuil.home.xs4all.nl/spec/userspace-api/v4l/dev-event.html>
Ok, V4L2_EVENT_CTRL event which reports 'struct v4l2_event_ctrl' data to user space
>
> >
> > For the required link freq and lane number, is v4l2 control the
> > correct way to configure them? If yes, seems there is no CID value for
> > them so that we should custom some CID value(link freqm, lane number,
> > and
> > privacy) for ivsc in linux/v4l2-controls.h, is this acceptable?
>
> You should obtain these using the V4L2 fwnode interface. Please see e.g.
> drivers/media/platform/renesas/rzg2l-cru/rzg2l-csi2.c how that driver registers
> a V4L2 async sub-device and a V4L2 async notifier.
Ok, something like v4l2_ctrl_find(source->ctrl_handler, V4L2_CID_LINK_FREQ); is to get sensor's link
frequency, and the code like 'v4l2_subdev_call(source, pad, get_mbus_config, source, &mbus_config);
num_of_lanes = mbus_config.bus.mipi_csi2.num_data_lanes;' is to get sensor's lane number.
BR,
Wentong
>
> --
> Kind regards,
>
> Sakari Ailus
Hi Wentong,
On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
> Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a
> companion chip designed to provide secure and low power vision capability
> to IA platforms. IVSC is available in existing commercial platforms from
> multiple OEMs.
>
> The primary use case of IVSC is to bring in context awareness. IVSC
> interfaces directly with the platform main camera sensor via a CSI-2 link
> and processes the image data with the embedded AI engine. The detected
> events are sent over I2C to ISH (Intel Sensor Hub) for additional data
> fusion from multiple sensors. The fusion results are used to implement
> advanced use cases like:
> - Face detection to unlock screen
> - Detect user presence to manage backlight setting or waking up system
>
> Since the Image Processing Unit(IPU) used on the host processor needs to
> configure the CSI-2 link in normal camera usages, the CSI-2 link and
> camera sensor can only be used in mutually-exclusive ways by host IPU and
> IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU
> driver can take ownership of the CSI-2 link and camera sensor using
> interfaces provided by this IVSC driver.
>
> Switching ownership requires an interface with two different hardware
> modules inside IVSC. The software interface to these modules is via Intel
> MEI (The Intel Management Engine) commands. These two hardware modules
> have two different MEI UUIDs to enumerate. These hardware modules are:
> - ACE (Algorithm Context Engine): This module is for algorithm computing
> when IVSC owns camera sensor. Also ACE module controls camera sensor's
> ownership. This hardware module is used to set ownership of camera sensor.
> - CSI (Camera Serial Interface): This module is used to route camera
> sensor data either to IVSC or to host for IPU driver and application.
>
> IVSC also provides a privacy mode. When privacy mode is turned on,
> camera sensor can't be used. This means that both ACE and host IPU can't
> get image data. And when this mode is turned on, host IPU driver is
> informed via a registered callback, so that user can be notified.
>
> In summary, to acquire ownership of camera by IPU driver, first ACE
> module needs to be informed of ownership and then to setup MIPI CSI-2
> link for the camera sensor and IPU.
I thought this for a while and did some research, and I can suggest the
following:
- The IVSC sub-device implements a control for privacy (V4L2_CID_PRIVACY
is a good fit).
- Camera sensor access needs to be requested from IVSC before accessing the
sensor via I?C. The IVSC ownership control needs to be in the right
setting for this to work, and device links can be used for that purpose
(see device_link_add()). With DL_FLAG_PM_RUNTIME and DL_FLAG_RPM_ACTIVE,
the supplier devices will be PM runtime resumed before the consumer
(camera sensor). As these devices are purely virtual on host side and has
no power state as such, you can use runtime PM callbacks to transfer the
ownership.
- The CSI-2 configuration should take place when streaming starts on the
sensor (as well as IVSC).
- Device links need to be set up via IPU bridge which now is called CIO2
bridge (cio2-bridge.c).
Any questions, comments?
--
Kind regards,
Sakari Ailus
Hi,
On 3/1/23 11:34, Sakari Ailus wrote:
> Hi Wentong,
>
> On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
>> Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a
>> companion chip designed to provide secure and low power vision capability
>> to IA platforms. IVSC is available in existing commercial platforms from
>> multiple OEMs.
>>
>> The primary use case of IVSC is to bring in context awareness. IVSC
>> interfaces directly with the platform main camera sensor via a CSI-2 link
>> and processes the image data with the embedded AI engine. The detected
>> events are sent over I2C to ISH (Intel Sensor Hub) for additional data
>> fusion from multiple sensors. The fusion results are used to implement
>> advanced use cases like:
>> - Face detection to unlock screen
>> - Detect user presence to manage backlight setting or waking up system
>>
>> Since the Image Processing Unit(IPU) used on the host processor needs to
>> configure the CSI-2 link in normal camera usages, the CSI-2 link and
>> camera sensor can only be used in mutually-exclusive ways by host IPU and
>> IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU
>> driver can take ownership of the CSI-2 link and camera sensor using
>> interfaces provided by this IVSC driver.
>>
>> Switching ownership requires an interface with two different hardware
>> modules inside IVSC. The software interface to these modules is via Intel
>> MEI (The Intel Management Engine) commands. These two hardware modules
>> have two different MEI UUIDs to enumerate. These hardware modules are:
>> - ACE (Algorithm Context Engine): This module is for algorithm computing
>> when IVSC owns camera sensor. Also ACE module controls camera sensor's
>> ownership. This hardware module is used to set ownership of camera sensor.
>> - CSI (Camera Serial Interface): This module is used to route camera
>> sensor data either to IVSC or to host for IPU driver and application.
>>
>> IVSC also provides a privacy mode. When privacy mode is turned on,
>> camera sensor can't be used. This means that both ACE and host IPU can't
>> get image data. And when this mode is turned on, host IPU driver is
>> informed via a registered callback, so that user can be notified.
>>
>> In summary, to acquire ownership of camera by IPU driver, first ACE
>> module needs to be informed of ownership and then to setup MIPI CSI-2
>> link for the camera sensor and IPU.
>
> I thought this for a while and did some research, and I can suggest the
> following:
>
> - The IVSC sub-device implements a control for privacy (V4L2_CID_PRIVACY
> is a good fit).
>
> - Camera sensor access needs to be requested from IVSC before accessing the
> sensor via I²C. The IVSC ownership control needs to be in the right
> setting for this to work, and device links can be used for that purpose
> (see device_link_add()). With DL_FLAG_PM_RUNTIME and DL_FLAG_RPM_ACTIVE,
> the supplier devices will be PM runtime resumed before the consumer
> (camera sensor). As these devices are purely virtual on host side and has
> no power state as such, you can use runtime PM callbacks to transfer the
> ownership.
Interesting proposal to use device-links + runtime-pm for this instead
of modelling this as an i2c-mux. FWIW I'm fine with going this route instead
of using an i2c-mux approach.
I have been thinking about the i2c-mux approach a bit and the problem is
that we are not really muxing but want to turn on/off control and AFAIK
the i2c-mux framework simply leaves the mux muxed to the last used i2c-chain,
so control will never be released when the i2c transfers are done.
And if were to somehow modify things (or maybe there already is some
release callback) then the downside becomes that the i2c-mux core code
operates at the i2c transfer level. So each i2c read/write would then
enable + disavle control.
Modelling this using something like runtime pm as such is a much better
fit because then we request control once on probe / stream-on and
release it once we are fully done, rather then requesting + releasing
control once per i2c-transfer.
Regards,
Hans
>
> - The CSI-2 configuration should take place when streaming starts on the
> sensor (as well as IVSC).
>
> - Device links need to be set up via IPU bridge which now is called CIO2
> bridge (cio2-bridge.c).
>
> Any questions, comments?
>
Hi Wentong,
On Wed, Mar 01, 2023 at 07:26:40AM +0000, Wu, Wentong wrote:
>
>
> > -----Original Message-----
> > From: Sakari Ailus <[email protected]>
> > Sent: Tuesday, February 28, 2023 4:24 PM
> >
> > Hi Wentong,
> >
> > On Tue, Feb 28, 2023 at 06:35:41AM +0000, Wu, Wentong wrote:
> > > Hi Sakari,
> > >
> > > few questions as switching to v4l2 sub-dev framework.
> > >
> > > > -----Original Message-----
> > > > From: Sakari Ailus <[email protected]>
> > > > Sent: Wednesday, February 15, 2023 12:06 AM
> > > >
> > > > Hi Wentong,
> > > >
> > > > Thanks for the patchset.
> > > >
> > > > On Mon, Feb 13, 2023 at 10:23:47AM +0800, Wentong Wu wrote:
> > > > > IVSC directly connects to camera sensor on source side, and on
> > > > > output side it not only connects ISH via I2C, but also exposes
> > > > > MIPI CSI-2 interface to output camera sensor data. IVSC can use
> > > > > the camera sensor data to do AI algorithm, and send the results to
> > > > > ISH. On the other end, IVSC can share camera sensor to host by
> > > > > routing the raw camera sensor data to the exposed MIPI CSI-2
> > > > > interface. But they can not work at the same time, so software APIs are
> > defined to sync the ownership.
> > > > >
> > > > > This commit defines the interfaces between IVSC and camera sensor
> > > > > driver in include/linux/ivsc.h. The camera driver controls
> > > > > ownership of the CSI-2 link and sensor with the acquire/release
> > > > > APIs. When acquiring camera, lane number and link freq are also
> > > > > required by IVSC frame router.
> > > >
> > > > The more I learn about this system, the more I'm inclined to think
> > > > this functionality should be exposed as a V4L2 sub-device. IVSC
> > > > doesn't really do anything to the data (as long as it directs it
> > > > towards the CSI-2 receiver in the SoC), but it is definitely part of the image
> > pipeline.
> > > >
> > > > I suppose the intended use cases assume a single instance of IVSC
> > > > (as well as
> > > > MEI) but there can, and often are, be multiple camera sensors in the
> > > > system. The decision whether to request pass-through from IVCS can't
> > > > be done in the camera sensor driver, and should not be visible to
> > > > the camera sensor driver. Exposing IVSC as a V4L2 sub-device makes
> > > > this trivial to address, as the IVSC driver's V4L2 sub-device video s_stream()
> > operation gets called before streaming is started.
> > > >
> > > > The information whether IVSC is found between the camera sensor and
> > > > the host's CSI-2 receiver (IPU in this case) should come from system
> > > > firmware and accessed most probably by what is called cio2-bridge at the
> > moment.
> > > >
> > > > The privacy status can be a V4L2 control.
> > >
> > > This should be a control or event? If control, how user-space handle
> > > privacy stuff?
> >
> > Changing control events generates events for the user space.
> >
> > <URL:https://hverkuil.home.xs4all.nl/spec/userspace-api/v4l/dev-event.html>
>
> Ok, V4L2_EVENT_CTRL event which reports 'struct v4l2_event_ctrl' data to user space
>
> >
> > >
> > > For the required link freq and lane number, is v4l2 control the
> > > correct way to configure them? If yes, seems there is no CID value for
> > > them so that we should custom some CID value(link freqm, lane number,
> > > and
> > > privacy) for ivsc in linux/v4l2-controls.h, is this acceptable?
> >
> > You should obtain these using the V4L2 fwnode interface. Please see e.g.
> > drivers/media/platform/renesas/rzg2l-cru/rzg2l-csi2.c how that driver registers
> > a V4L2 async sub-device and a V4L2 async notifier.
>
> Ok, something like v4l2_ctrl_find(source->ctrl_handler,
> V4L2_CID_LINK_FREQ); is to get sensor's link frequency, and the code like
> 'v4l2_subdev_call(source, pad, get_mbus_config, source, &mbus_config);
> num_of_lanes = mbus_config.bus.mipi_csi2.num_data_lanes;' is to get
> sensor's lane number.
Please use the V4L2 fwnode interface instead.
The privacy control should only be needed in the user space.
--
Regards,
Sakari Ailus
> -----Original Message-----
> From: Sakari Ailus <[email protected]>
> Sent: Wednesday, March 1, 2023 6:34 PM
>
> Hi Wentong,
>
> On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
> > Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a
> > companion chip designed to provide secure and low power vision
> > capability to IA platforms. IVSC is available in existing commercial
> > platforms from multiple OEMs.
> >
> > The primary use case of IVSC is to bring in context awareness. IVSC
> > interfaces directly with the platform main camera sensor via a CSI-2
> > link and processes the image data with the embedded AI engine. The
> > detected events are sent over I2C to ISH (Intel Sensor Hub) for
> > additional data fusion from multiple sensors. The fusion results are
> > used to implement advanced use cases like:
> > - Face detection to unlock screen
> > - Detect user presence to manage backlight setting or waking up
> > system
> >
> > Since the Image Processing Unit(IPU) used on the host processor needs
> > to configure the CSI-2 link in normal camera usages, the CSI-2 link
> > and camera sensor can only be used in mutually-exclusive ways by host
> > IPU and IVSC. By default the IVSC owns the CSI-2 link and camera
> > sensor. The IPU driver can take ownership of the CSI-2 link and camera
> > sensor using interfaces provided by this IVSC driver.
> >
> > Switching ownership requires an interface with two different hardware
> > modules inside IVSC. The software interface to these modules is via
> > Intel MEI (The Intel Management Engine) commands. These two hardware
> > modules have two different MEI UUIDs to enumerate. These hardware
> modules are:
> > - ACE (Algorithm Context Engine): This module is for algorithm
> > computing when IVSC owns camera sensor. Also ACE module controls
> > camera sensor's ownership. This hardware module is used to set ownership of
> camera sensor.
> > - CSI (Camera Serial Interface): This module is used to route camera
> > sensor data either to IVSC or to host for IPU driver and application.
> >
> > IVSC also provides a privacy mode. When privacy mode is turned on,
> > camera sensor can't be used. This means that both ACE and host IPU
> > can't get image data. And when this mode is turned on, host IPU driver
> > is informed via a registered callback, so that user can be notified.
> >
> > In summary, to acquire ownership of camera by IPU driver, first ACE
> > module needs to be informed of ownership and then to setup MIPI CSI-2
> > link for the camera sensor and IPU.
>
> I thought this for a while and did some research, and I can suggest the
> following:
Thanks a lot
>
> - The IVSC sub-device implements a control for privacy (V4L2_CID_PRIVACY
> is a good fit).
Agree, thanks
>
> - Camera sensor access needs to be requested from IVSC before accessing the
> sensor via I?C. The IVSC ownership control needs to be in the right
> setting for this to work, and device links can be used for that purpose
> (see device_link_add()). With DL_FLAG_PM_RUNTIME and
> DL_FLAG_RPM_ACTIVE,
> the supplier devices will be PM runtime resumed before the consumer
> (camera sensor). As these devices are purely virtual on host side and has
> no power state as such, you can use runtime PM callbacks to transfer the
> ownership.
>
That's pretty cool, and the device link will be setup during mei_ace's probe with
device_link_add, but we should guarantee there is no i2c transfer during sensor
driver's probe. And most of upstream sensor drivers don't initial i2c transfer
during probe, but I have no idea what will be like for our upstreaming sensor drivers.
> - The CSI-2 configuration should take place when streaming starts on the
> sensor (as well as IVSC).
IPU driver should make sure start sensor streaming first and then IVSC sub-dev,
which will match the command downloading sequence required by firmware.
>
> - Device links need to be set up via IPU bridge which now is called CIO2
> bridge (cio2-bridge.c).
Reviewing the code
>
> Any questions, comments?
>
> --
> Kind regards,
>
> Sakari Ailus
> -----Original Message-----
> From: Hans de Goede <[email protected]>
> Sent: Wednesday, March 1, 2023 6:42 PM
>
> Hi,
>
> On 3/1/23 11:34, Sakari Ailus wrote:
> > Hi Wentong,
> >
> > On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
> >> Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is
> >> a companion chip designed to provide secure and low power vision
> >> capability to IA platforms. IVSC is available in existing commercial
> >> platforms from multiple OEMs.
> >>
> >> The primary use case of IVSC is to bring in context awareness. IVSC
> >> interfaces directly with the platform main camera sensor via a CSI-2
> >> link and processes the image data with the embedded AI engine. The
> >> detected events are sent over I2C to ISH (Intel Sensor Hub) for
> >> additional data fusion from multiple sensors. The fusion results are
> >> used to implement advanced use cases like:
> >> - Face detection to unlock screen
> >> - Detect user presence to manage backlight setting or waking up
> >> system
> >>
> >> Since the Image Processing Unit(IPU) used on the host processor needs
> >> to configure the CSI-2 link in normal camera usages, the CSI-2 link
> >> and camera sensor can only be used in mutually-exclusive ways by host
> >> IPU and IVSC. By default the IVSC owns the CSI-2 link and camera
> >> sensor. The IPU driver can take ownership of the CSI-2 link and
> >> camera sensor using interfaces provided by this IVSC driver.
> >>
> >> Switching ownership requires an interface with two different hardware
> >> modules inside IVSC. The software interface to these modules is via
> >> Intel MEI (The Intel Management Engine) commands. These two hardware
> >> modules have two different MEI UUIDs to enumerate. These hardware
> modules are:
> >> - ACE (Algorithm Context Engine): This module is for algorithm
> >> computing when IVSC owns camera sensor. Also ACE module controls
> >> camera sensor's ownership. This hardware module is used to set ownership
> of camera sensor.
> >> - CSI (Camera Serial Interface): This module is used to route camera
> >> sensor data either to IVSC or to host for IPU driver and application.
> >>
> >> IVSC also provides a privacy mode. When privacy mode is turned on,
> >> camera sensor can't be used. This means that both ACE and host IPU
> >> can't get image data. And when this mode is turned on, host IPU
> >> driver is informed via a registered callback, so that user can be notified.
> >>
> >> In summary, to acquire ownership of camera by IPU driver, first ACE
> >> module needs to be informed of ownership and then to setup MIPI CSI-2
> >> link for the camera sensor and IPU.
> >
> > I thought this for a while and did some research, and I can suggest
> > the
> > following:
> >
> > - The IVSC sub-device implements a control for privacy (V4L2_CID_PRIVACY
> > is a good fit).
> >
> > - Camera sensor access needs to be requested from IVSC before accessing the
> > sensor via I²C. The IVSC ownership control needs to be in the right
> > setting for this to work, and device links can be used for that purpose
> > (see device_link_add()). With DL_FLAG_PM_RUNTIME and
> DL_FLAG_RPM_ACTIVE,
> > the supplier devices will be PM runtime resumed before the consumer
> > (camera sensor). As these devices are purely virtual on host side and has
> > no power state as such, you can use runtime PM callbacks to transfer the
> > ownership.
>
> Interesting proposal to use device-links + runtime-pm for this instead of
> modelling this as an i2c-mux. FWIW I'm fine with going this route instead of
> using an i2c-mux approach.
>
> I have been thinking about the i2c-mux approach a bit and the problem is that
> we are not really muxing but want to turn on/off control and AFAIK the i2c-mux
> framework simply leaves the mux muxed to the last used i2c-chain, so control
> will never be released when the i2c transfers are done.
>
> And if were to somehow modify things (or maybe there already is some release
> callback) then the downside becomes that the i2c-mux core code operates at
> the i2c transfer level. So each i2c read/write would then enable + disavle control.
>
> Modelling this using something like runtime pm as such is a much better fit
> because then we request control once on probe / stream-on and release it once
> we are fully done, rather then requesting + releasing control once per i2c-
> transfer.
Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe,
probably we have to switch to i2c-mux modeling way.
Thanks
Wentong
>
> Regards,
>
> Hans
>
>
>
> >
> > - The CSI-2 configuration should take place when streaming starts on the
> > sensor (as well as IVSC).
> >
> > - Device links need to be set up via IPU bridge which now is called CIO2
> > bridge (cio2-bridge.c).
> >
> > Any questions, comments?
> >
Hi Wentong,
On Tue, Mar 07, 2023 at 08:17:04AM +0000, Wu, Wentong wrote:
>
>
> > -----Original Message-----
> > From: Hans de Goede <[email protected]>
> > Sent: Wednesday, March 1, 2023 6:42 PM
> >
> > Hi,
> >
> > On 3/1/23 11:34, Sakari Ailus wrote:
> > > Hi Wentong,
> > >
> > > On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
> > >> Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is
> > >> a companion chip designed to provide secure and low power vision
> > >> capability to IA platforms. IVSC is available in existing commercial
> > >> platforms from multiple OEMs.
> > >>
> > >> The primary use case of IVSC is to bring in context awareness. IVSC
> > >> interfaces directly with the platform main camera sensor via a CSI-2
> > >> link and processes the image data with the embedded AI engine. The
> > >> detected events are sent over I2C to ISH (Intel Sensor Hub) for
> > >> additional data fusion from multiple sensors. The fusion results are
> > >> used to implement advanced use cases like:
> > >> - Face detection to unlock screen
> > >> - Detect user presence to manage backlight setting or waking up
> > >> system
> > >>
> > >> Since the Image Processing Unit(IPU) used on the host processor needs
> > >> to configure the CSI-2 link in normal camera usages, the CSI-2 link
> > >> and camera sensor can only be used in mutually-exclusive ways by host
> > >> IPU and IVSC. By default the IVSC owns the CSI-2 link and camera
> > >> sensor. The IPU driver can take ownership of the CSI-2 link and
> > >> camera sensor using interfaces provided by this IVSC driver.
> > >>
> > >> Switching ownership requires an interface with two different hardware
> > >> modules inside IVSC. The software interface to these modules is via
> > >> Intel MEI (The Intel Management Engine) commands. These two hardware
> > >> modules have two different MEI UUIDs to enumerate. These hardware
> > modules are:
> > >> - ACE (Algorithm Context Engine): This module is for algorithm
> > >> computing when IVSC owns camera sensor. Also ACE module controls
> > >> camera sensor's ownership. This hardware module is used to set ownership
> > of camera sensor.
> > >> - CSI (Camera Serial Interface): This module is used to route camera
> > >> sensor data either to IVSC or to host for IPU driver and application.
> > >>
> > >> IVSC also provides a privacy mode. When privacy mode is turned on,
> > >> camera sensor can't be used. This means that both ACE and host IPU
> > >> can't get image data. And when this mode is turned on, host IPU
> > >> driver is informed via a registered callback, so that user can be notified.
> > >>
> > >> In summary, to acquire ownership of camera by IPU driver, first ACE
> > >> module needs to be informed of ownership and then to setup MIPI CSI-2
> > >> link for the camera sensor and IPU.
> > >
> > > I thought this for a while and did some research, and I can suggest
> > > the
> > > following:
> > >
> > > - The IVSC sub-device implements a control for privacy (V4L2_CID_PRIVACY
> > > is a good fit).
> > >
> > > - Camera sensor access needs to be requested from IVSC before accessing the
> > > sensor via I?C. The IVSC ownership control needs to be in the right
> > > setting for this to work, and device links can be used for that purpose
> > > (see device_link_add()). With DL_FLAG_PM_RUNTIME and
> > DL_FLAG_RPM_ACTIVE,
> > > the supplier devices will be PM runtime resumed before the consumer
> > > (camera sensor). As these devices are purely virtual on host side and has
> > > no power state as such, you can use runtime PM callbacks to transfer the
> > > ownership.
> >
> > Interesting proposal to use device-links + runtime-pm for this instead of
> > modelling this as an i2c-mux. FWIW I'm fine with going this route instead of
> > using an i2c-mux approach.
> >
> > I have been thinking about the i2c-mux approach a bit and the problem is that
> > we are not really muxing but want to turn on/off control and AFAIK the i2c-mux
> > framework simply leaves the mux muxed to the last used i2c-chain, so control
> > will never be released when the i2c transfers are done.
> >
> > And if were to somehow modify things (or maybe there already is some release
> > callback) then the downside becomes that the i2c-mux core code operates at
> > the i2c transfer level. So each i2c read/write would then enable + disavle control.
> >
> > Modelling this using something like runtime pm as such is a much better fit
> > because then we request control once on probe / stream-on and release it once
> > we are fully done, rather then requesting + releasing control once per i2c-
> > transfer.
>
> Seems runtime pm can't fix the problem of initial i2c transfer during sensor driver probe,
> probably we have to switch to i2c-mux modeling way.
What do you mean? The supplier devices are resumed before the driver's
probe is called.
--
Regards,
Sakari Ailus
> -----Original Message-----
> From: Sakari Ailus <[email protected]>
> Sent: Tuesday, March 7, 2023 4:30 PM
>
> Hi Wentong,
>
> On Tue, Mar 07, 2023 at 08:17:04AM +0000, Wu, Wentong wrote:
> >
> >
> > > -----Original Message-----
> > > From: Hans de Goede <[email protected]>
> > > Sent: Wednesday, March 1, 2023 6:42 PM
> > >
> > > Hi,
> > >
> > > On 3/1/23 11:34, Sakari Ailus wrote:
> > > > Hi Wentong,
> > > >
> > > > On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
> > > >> Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls",
> > > >> is a companion chip designed to provide secure and low power
> > > >> vision capability to IA platforms. IVSC is available in existing
> > > >> commercial platforms from multiple OEMs.
> > > >>
> > > >> The primary use case of IVSC is to bring in context awareness.
> > > >> IVSC interfaces directly with the platform main camera sensor via
> > > >> a CSI-2 link and processes the image data with the embedded AI
> > > >> engine. The detected events are sent over I2C to ISH (Intel
> > > >> Sensor Hub) for additional data fusion from multiple sensors. The
> > > >> fusion results are used to implement advanced use cases like:
> > > >> - Face detection to unlock screen
> > > >> - Detect user presence to manage backlight setting or waking up
> > > >> system
> > > >>
> > > >> Since the Image Processing Unit(IPU) used on the host processor
> > > >> needs to configure the CSI-2 link in normal camera usages, the
> > > >> CSI-2 link and camera sensor can only be used in
> > > >> mutually-exclusive ways by host IPU and IVSC. By default the IVSC
> > > >> owns the CSI-2 link and camera sensor. The IPU driver can take
> > > >> ownership of the CSI-2 link and camera sensor using interfaces provided
> by this IVSC driver.
> > > >>
> > > >> Switching ownership requires an interface with two different
> > > >> hardware modules inside IVSC. The software interface to these
> > > >> modules is via Intel MEI (The Intel Management Engine) commands.
> > > >> These two hardware modules have two different MEI UUIDs to
> > > >> enumerate. These hardware
> > > modules are:
> > > >> - ACE (Algorithm Context Engine): This module is for algorithm
> > > >> computing when IVSC owns camera sensor. Also ACE module controls
> > > >> camera sensor's ownership. This hardware module is used to set
> > > >> ownership
> > > of camera sensor.
> > > >> - CSI (Camera Serial Interface): This module is used to route
> > > >> camera sensor data either to IVSC or to host for IPU driver and
> application.
> > > >>
> > > >> IVSC also provides a privacy mode. When privacy mode is turned
> > > >> on, camera sensor can't be used. This means that both ACE and
> > > >> host IPU can't get image data. And when this mode is turned on,
> > > >> host IPU driver is informed via a registered callback, so that user can be
> notified.
> > > >>
> > > >> In summary, to acquire ownership of camera by IPU driver, first
> > > >> ACE module needs to be informed of ownership and then to setup
> > > >> MIPI CSI-2 link for the camera sensor and IPU.
> > > >
> > > > I thought this for a while and did some research, and I can
> > > > suggest the
> > > > following:
> > > >
> > > > - The IVSC sub-device implements a control for privacy (V4L2_CID_PRIVACY
> > > > is a good fit).
> > > >
> > > > - Camera sensor access needs to be requested from IVSC before accessing
> the
> > > > sensor via I?C. The IVSC ownership control needs to be in the right
> > > > setting for this to work, and device links can be used for that purpose
> > > > (see device_link_add()). With DL_FLAG_PM_RUNTIME and
> > > DL_FLAG_RPM_ACTIVE,
> > > > the supplier devices will be PM runtime resumed before the consumer
> > > > (camera sensor). As these devices are purely virtual on host side and has
> > > > no power state as such, you can use runtime PM callbacks to transfer the
> > > > ownership.
> > >
> > > Interesting proposal to use device-links + runtime-pm for this
> > > instead of modelling this as an i2c-mux. FWIW I'm fine with going
> > > this route instead of using an i2c-mux approach.
> > >
> > > I have been thinking about the i2c-mux approach a bit and the
> > > problem is that we are not really muxing but want to turn on/off
> > > control and AFAIK the i2c-mux framework simply leaves the mux muxed
> > > to the last used i2c-chain, so control will never be released when the i2c
> transfers are done.
> > >
> > > And if were to somehow modify things (or maybe there already is some
> > > release
> > > callback) then the downside becomes that the i2c-mux core code
> > > operates at the i2c transfer level. So each i2c read/write would then enable +
> disavle control.
> > >
> > > Modelling this using something like runtime pm as such is a much
> > > better fit because then we request control once on probe / stream-on
> > > and release it once we are fully done, rather then requesting +
> > > releasing control once per i2c- transfer.
> >
> > Seems runtime pm can't fix the problem of initial i2c transfer during
> > sensor driver probe, probably we have to switch to i2c-mux modeling way.
>
> What do you mean? The supplier devices are resumed before the driver's probe
> is called.
But we setup the link with device_link_add during IVSC driver's probe, we can't
guarantee driver probe's sequence.
>
> --
> Regards,
>
> Sakari Ailus
Hi,
On 3/7/23 09:40, Wu, Wentong wrote:
>
>
>> -----Original Message-----
>> From: Sakari Ailus <[email protected]>
>> Sent: Tuesday, March 7, 2023 4:30 PM
>>
>> Hi Wentong,
>>
>> On Tue, Mar 07, 2023 at 08:17:04AM +0000, Wu, Wentong wrote:
>>>
>>>
>>>> -----Original Message-----
>>>> From: Hans de Goede <[email protected]>
>>>> Sent: Wednesday, March 1, 2023 6:42 PM
>>>>
>>>> Hi,
>>>>
>>>> On 3/1/23 11:34, Sakari Ailus wrote:
>>>>> Hi Wentong,
>>>>>
>>>>> On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
>>>>>> Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls",
>>>>>> is a companion chip designed to provide secure and low power
>>>>>> vision capability to IA platforms. IVSC is available in existing
>>>>>> commercial platforms from multiple OEMs.
>>>>>>
>>>>>> The primary use case of IVSC is to bring in context awareness.
>>>>>> IVSC interfaces directly with the platform main camera sensor via
>>>>>> a CSI-2 link and processes the image data with the embedded AI
>>>>>> engine. The detected events are sent over I2C to ISH (Intel
>>>>>> Sensor Hub) for additional data fusion from multiple sensors. The
>>>>>> fusion results are used to implement advanced use cases like:
>>>>>> - Face detection to unlock screen
>>>>>> - Detect user presence to manage backlight setting or waking up
>>>>>> system
>>>>>>
>>>>>> Since the Image Processing Unit(IPU) used on the host processor
>>>>>> needs to configure the CSI-2 link in normal camera usages, the
>>>>>> CSI-2 link and camera sensor can only be used in
>>>>>> mutually-exclusive ways by host IPU and IVSC. By default the IVSC
>>>>>> owns the CSI-2 link and camera sensor. The IPU driver can take
>>>>>> ownership of the CSI-2 link and camera sensor using interfaces provided
>> by this IVSC driver.
>>>>>>
>>>>>> Switching ownership requires an interface with two different
>>>>>> hardware modules inside IVSC. The software interface to these
>>>>>> modules is via Intel MEI (The Intel Management Engine) commands.
>>>>>> These two hardware modules have two different MEI UUIDs to
>>>>>> enumerate. These hardware
>>>> modules are:
>>>>>> - ACE (Algorithm Context Engine): This module is for algorithm
>>>>>> computing when IVSC owns camera sensor. Also ACE module controls
>>>>>> camera sensor's ownership. This hardware module is used to set
>>>>>> ownership
>>>> of camera sensor.
>>>>>> - CSI (Camera Serial Interface): This module is used to route
>>>>>> camera sensor data either to IVSC or to host for IPU driver and
>> application.
>>>>>>
>>>>>> IVSC also provides a privacy mode. When privacy mode is turned
>>>>>> on, camera sensor can't be used. This means that both ACE and
>>>>>> host IPU can't get image data. And when this mode is turned on,
>>>>>> host IPU driver is informed via a registered callback, so that user can be
>> notified.
>>>>>>
>>>>>> In summary, to acquire ownership of camera by IPU driver, first
>>>>>> ACE module needs to be informed of ownership and then to setup
>>>>>> MIPI CSI-2 link for the camera sensor and IPU.
>>>>>
>>>>> I thought this for a while and did some research, and I can
>>>>> suggest the
>>>>> following:
>>>>>
>>>>> - The IVSC sub-device implements a control for privacy (V4L2_CID_PRIVACY
>>>>> is a good fit).
>>>>>
>>>>> - Camera sensor access needs to be requested from IVSC before accessing
>> the
>>>>> sensor via I²C. The IVSC ownership control needs to be in the right
>>>>> setting for this to work, and device links can be used for that purpose
>>>>> (see device_link_add()). With DL_FLAG_PM_RUNTIME and
>>>> DL_FLAG_RPM_ACTIVE,
>>>>> the supplier devices will be PM runtime resumed before the consumer
>>>>> (camera sensor). As these devices are purely virtual on host side and has
>>>>> no power state as such, you can use runtime PM callbacks to transfer the
>>>>> ownership.
>>>>
>>>> Interesting proposal to use device-links + runtime-pm for this
>>>> instead of modelling this as an i2c-mux. FWIW I'm fine with going
>>>> this route instead of using an i2c-mux approach.
>>>>
>>>> I have been thinking about the i2c-mux approach a bit and the
>>>> problem is that we are not really muxing but want to turn on/off
>>>> control and AFAIK the i2c-mux framework simply leaves the mux muxed
>>>> to the last used i2c-chain, so control will never be released when the i2c
>> transfers are done.
>>>>
>>>> And if were to somehow modify things (or maybe there already is some
>>>> release
>>>> callback) then the downside becomes that the i2c-mux core code
>>>> operates at the i2c transfer level. So each i2c read/write would then enable +
>> disavle control.
>>>>
>>>> Modelling this using something like runtime pm as such is a much
>>>> better fit because then we request control once on probe / stream-on
>>>> and release it once we are fully done, rather then requesting +
>>>> releasing control once per i2c- transfer.
>>>
>>> Seems runtime pm can't fix the problem of initial i2c transfer during
>>> sensor driver probe, probably we have to switch to i2c-mux modeling way.
>>
>> What do you mean? The supplier devices are resumed before the driver's probe
>> is called.
>
> But we setup the link with device_link_add during IVSC driver's probe, we can't
> guarantee driver probe's sequence.
Then maybe we need to do the device_link_add somewhere else.
The mainline kernel delays probing of camera sensors on Intel platforms
until the INT3472 driver has probed the INT3472 device on which the
sensors have an ACPI _DEP.
This is already used to make sure that clock lookups and regulator
info is in place before the sensor's probe() function runs.
So that when the driver does clk_get() it succeeds and so that regulator_get()
does not end up returning a dummy regulator.
So I think the code adding the device_link-s for the IVSC should be added
to: drivers/platform/x86/intel/int3472/discrete.c and then the runtime-resume
will happen before the sensor's probe() function runs.
Likewise drivers/platform/x86/intel/int3472/discrete.c should also ensure
that the ivsc driver's probe() has run before it calls
acpi_dev_clear_dependencies().
The acpi_dev_clear_dependencies() call in discrete.c tells the ACPI subsystem
to go ahead and create the i2c-clients for the sensors and allow the sensor
drivers to get loaded and probe the sensor.
Regards,
Hans
> -----Original Message-----
> From: Hans de Goede <[email protected]>
> Sent: Tuesday, March 7, 2023 5:10 PM
>
> Hi,
>
> On 3/7/23 09:40, Wu, Wentong wrote:
> >
> >
> >> -----Original Message-----
> >> From: Sakari Ailus <[email protected]>
> >> Sent: Tuesday, March 7, 2023 4:30 PM
> >>
> >> Hi Wentong,
> >>
> >> On Tue, Mar 07, 2023 at 08:17:04AM +0000, Wu, Wentong wrote:
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: Hans de Goede <[email protected]>
> >>>> Sent: Wednesday, March 1, 2023 6:42 PM
> >>>>
> >>>> Hi,
> >>>>
> >>>> On 3/1/23 11:34, Sakari Ailus wrote:
> >>>>> Hi Wentong,
> >>>>>
> >>>>> On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
> >>>>>> Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls",
> >>>>>> is a companion chip designed to provide secure and low power
> >>>>>> vision capability to IA platforms. IVSC is available in existing
> >>>>>> commercial platforms from multiple OEMs.
> >>>>>>
> >>>>>> The primary use case of IVSC is to bring in context awareness.
> >>>>>> IVSC interfaces directly with the platform main camera sensor via
> >>>>>> a CSI-2 link and processes the image data with the embedded AI
> >>>>>> engine. The detected events are sent over I2C to ISH (Intel
> >>>>>> Sensor Hub) for additional data fusion from multiple sensors. The
> >>>>>> fusion results are used to implement advanced use cases like:
> >>>>>> - Face detection to unlock screen
> >>>>>> - Detect user presence to manage backlight setting or waking up
> >>>>>> system
> >>>>>>
> >>>>>> Since the Image Processing Unit(IPU) used on the host processor
> >>>>>> needs to configure the CSI-2 link in normal camera usages, the
> >>>>>> CSI-2 link and camera sensor can only be used in
> >>>>>> mutually-exclusive ways by host IPU and IVSC. By default the IVSC
> >>>>>> owns the CSI-2 link and camera sensor. The IPU driver can take
> >>>>>> ownership of the CSI-2 link and camera sensor using interfaces
> >>>>>> provided
> >> by this IVSC driver.
> >>>>>>
> >>>>>> Switching ownership requires an interface with two different
> >>>>>> hardware modules inside IVSC. The software interface to these
> >>>>>> modules is via Intel MEI (The Intel Management Engine) commands.
> >>>>>> These two hardware modules have two different MEI UUIDs to
> >>>>>> enumerate. These hardware
> >>>> modules are:
> >>>>>> - ACE (Algorithm Context Engine): This module is for algorithm
> >>>>>> computing when IVSC owns camera sensor. Also ACE module controls
> >>>>>> camera sensor's ownership. This hardware module is used to set
> >>>>>> ownership
> >>>> of camera sensor.
> >>>>>> - CSI (Camera Serial Interface): This module is used to route
> >>>>>> camera sensor data either to IVSC or to host for IPU driver and
> >> application.
> >>>>>>
> >>>>>> IVSC also provides a privacy mode. When privacy mode is turned
> >>>>>> on, camera sensor can't be used. This means that both ACE and
> >>>>>> host IPU can't get image data. And when this mode is turned on,
> >>>>>> host IPU driver is informed via a registered callback, so that
> >>>>>> user can be
> >> notified.
> >>>>>>
> >>>>>> In summary, to acquire ownership of camera by IPU driver, first
> >>>>>> ACE module needs to be informed of ownership and then to setup
> >>>>>> MIPI CSI-2 link for the camera sensor and IPU.
> >>>>>
> >>>>> I thought this for a while and did some research, and I can
> >>>>> suggest the
> >>>>> following:
> >>>>>
> >>>>> - The IVSC sub-device implements a control for privacy
> (V4L2_CID_PRIVACY
> >>>>> is a good fit).
> >>>>>
> >>>>> - Camera sensor access needs to be requested from IVSC before
> >>>>> accessing
> >> the
> >>>>> sensor via I²C. The IVSC ownership control needs to be in the right
> >>>>> setting for this to work, and device links can be used for that purpose
> >>>>> (see device_link_add()). With DL_FLAG_PM_RUNTIME and
> >>>> DL_FLAG_RPM_ACTIVE,
> >>>>> the supplier devices will be PM runtime resumed before the consumer
> >>>>> (camera sensor). As these devices are purely virtual on host side and has
> >>>>> no power state as such, you can use runtime PM callbacks to transfer
> the
> >>>>> ownership.
> >>>>
> >>>> Interesting proposal to use device-links + runtime-pm for this
> >>>> instead of modelling this as an i2c-mux. FWIW I'm fine with going
> >>>> this route instead of using an i2c-mux approach.
> >>>>
> >>>> I have been thinking about the i2c-mux approach a bit and the
> >>>> problem is that we are not really muxing but want to turn on/off
> >>>> control and AFAIK the i2c-mux framework simply leaves the mux muxed
> >>>> to the last used i2c-chain, so control will never be released when
> >>>> the i2c
> >> transfers are done.
> >>>>
> >>>> And if were to somehow modify things (or maybe there already is
> >>>> some release
> >>>> callback) then the downside becomes that the i2c-mux core code
> >>>> operates at the i2c transfer level. So each i2c read/write would
> >>>> then enable +
> >> disavle control.
> >>>>
> >>>> Modelling this using something like runtime pm as such is a much
> >>>> better fit because then we request control once on probe /
> >>>> stream-on and release it once we are fully done, rather then
> >>>> requesting + releasing control once per i2c- transfer.
> >>>
> >>> Seems runtime pm can't fix the problem of initial i2c transfer
> >>> during sensor driver probe, probably we have to switch to i2c-mux modeling
> way.
> >>
> >> What do you mean? The supplier devices are resumed before the
> >> driver's probe is called.
> >
> > But we setup the link with device_link_add during IVSC driver's probe,
> > we can't guarantee driver probe's sequence.
>
> Then maybe we need to do the device_link_add somewhere else.
sensor's parent is the LJCA I2C device whose driver is being upstream
https://www.spinics.net/lists/kernel/msg4702552.htmland and sensor's
power is controlled by IVSC instead of INT3472 if IVSC enabled.
struct device_link *device_link_add(struct device *consumer,
struct device *supplier, u32 flags)
So probably we have to add above device_link_add in LJCA I2C's driver,
and we can find the consumer(camera sensor) with ACPI API, but the
supplier, mei_ace, is mei client device under mei framework and it's
dynamically allocated device instead of ACPI device, probably I can find
its parent with some ACPI lookup from this LJCA I2C device, but
unfortunately mei framework doesn't export the API to find mei client
device with its parent bus device(struct mei_device).
I'm not sure if modeling this mei_ace as LJCA I2C's runtime power
control is acceptable, if yes, probably this mei_ace driver have to go with
LJCA I2C device driver.
BR,
Wentong
>
> The mainline kernel delays probing of camera sensors on Intel platforms until
> the INT3472 driver has probed the INT3472 device on which the sensors have an
> ACPI _DEP.
>
> This is already used to make sure that clock lookups and regulator info is in place
> before the sensor's probe() function runs.
>
> So that when the driver does clk_get() it succeeds and so that regulator_get()
> does not end up returning a dummy regulator.
>
> So I think the code adding the device_link-s for the IVSC should be added
> to: drivers/platform/x86/intel/int3472/discrete.c and then the runtime-resume
> will happen before the sensor's probe() function runs.
>
> Likewise drivers/platform/x86/intel/int3472/discrete.c should also ensure that
> the ivsc driver's probe() has run before it calls acpi_dev_clear_dependencies().
>
> The acpi_dev_clear_dependencies() call in discrete.c tells the ACPI subsystem to
> go ahead and create the i2c-clients for the sensors and allow the sensor drivers
> to get loaded and probe the sensor.
>
> Regards,
>
> Hans
Hi,
On 3/9/23 02:08, Wu, Wentong wrote:
>
>
>> -----Original Message-----
>> From: Hans de Goede <[email protected]>
>> Sent: Tuesday, March 7, 2023 5:10 PM
>>
>> Hi,
>>
>> On 3/7/23 09:40, Wu, Wentong wrote:
>>>
>>>
>>>> -----Original Message-----
>>>> From: Sakari Ailus <[email protected]>
>>>> Sent: Tuesday, March 7, 2023 4:30 PM
>>>>
>>>> Hi Wentong,
>>>>
>>>> On Tue, Mar 07, 2023 at 08:17:04AM +0000, Wu, Wentong wrote:
>>>>>
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Hans de Goede <[email protected]>
>>>>>> Sent: Wednesday, March 1, 2023 6:42 PM
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> On 3/1/23 11:34, Sakari Ailus wrote:
>>>>>>> Hi Wentong,
>>>>>>>
>>>>>>> On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
>>>>>>>> Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls",
>>>>>>>> is a companion chip designed to provide secure and low power
>>>>>>>> vision capability to IA platforms. IVSC is available in existing
>>>>>>>> commercial platforms from multiple OEMs.
>>>>>>>>
>>>>>>>> The primary use case of IVSC is to bring in context awareness.
>>>>>>>> IVSC interfaces directly with the platform main camera sensor via
>>>>>>>> a CSI-2 link and processes the image data with the embedded AI
>>>>>>>> engine. The detected events are sent over I2C to ISH (Intel
>>>>>>>> Sensor Hub) for additional data fusion from multiple sensors. The
>>>>>>>> fusion results are used to implement advanced use cases like:
>>>>>>>> - Face detection to unlock screen
>>>>>>>> - Detect user presence to manage backlight setting or waking up
>>>>>>>> system
>>>>>>>>
>>>>>>>> Since the Image Processing Unit(IPU) used on the host processor
>>>>>>>> needs to configure the CSI-2 link in normal camera usages, the
>>>>>>>> CSI-2 link and camera sensor can only be used in
>>>>>>>> mutually-exclusive ways by host IPU and IVSC. By default the IVSC
>>>>>>>> owns the CSI-2 link and camera sensor. The IPU driver can take
>>>>>>>> ownership of the CSI-2 link and camera sensor using interfaces
>>>>>>>> provided
>>>> by this IVSC driver.
>>>>>>>>
>>>>>>>> Switching ownership requires an interface with two different
>>>>>>>> hardware modules inside IVSC. The software interface to these
>>>>>>>> modules is via Intel MEI (The Intel Management Engine) commands.
>>>>>>>> These two hardware modules have two different MEI UUIDs to
>>>>>>>> enumerate. These hardware
>>>>>> modules are:
>>>>>>>> - ACE (Algorithm Context Engine): This module is for algorithm
>>>>>>>> computing when IVSC owns camera sensor. Also ACE module controls
>>>>>>>> camera sensor's ownership. This hardware module is used to set
>>>>>>>> ownership
>>>>>> of camera sensor.
>>>>>>>> - CSI (Camera Serial Interface): This module is used to route
>>>>>>>> camera sensor data either to IVSC or to host for IPU driver and
>>>> application.
>>>>>>>>
>>>>>>>> IVSC also provides a privacy mode. When privacy mode is turned
>>>>>>>> on, camera sensor can't be used. This means that both ACE and
>>>>>>>> host IPU can't get image data. And when this mode is turned on,
>>>>>>>> host IPU driver is informed via a registered callback, so that
>>>>>>>> user can be
>>>> notified.
>>>>>>>>
>>>>>>>> In summary, to acquire ownership of camera by IPU driver, first
>>>>>>>> ACE module needs to be informed of ownership and then to setup
>>>>>>>> MIPI CSI-2 link for the camera sensor and IPU.
>>>>>>>
>>>>>>> I thought this for a while and did some research, and I can
>>>>>>> suggest the
>>>>>>> following:
>>>>>>>
>>>>>>> - The IVSC sub-device implements a control for privacy
>> (V4L2_CID_PRIVACY
>>>>>>> is a good fit).
>>>>>>>
>>>>>>> - Camera sensor access needs to be requested from IVSC before
>>>>>>> accessing
>>>> the
>>>>>>> sensor via I²C. The IVSC ownership control needs to be in the right
>>>>>>> setting for this to work, and device links can be used for that purpose
>>>>>>> (see device_link_add()). With DL_FLAG_PM_RUNTIME and
>>>>>> DL_FLAG_RPM_ACTIVE,
>>>>>>> the supplier devices will be PM runtime resumed before the consumer
>>>>>>> (camera sensor). As these devices are purely virtual on host side and has
>>>>>>> no power state as such, you can use runtime PM callbacks to transfer
>> the
>>>>>>> ownership.
>>>>>>
>>>>>> Interesting proposal to use device-links + runtime-pm for this
>>>>>> instead of modelling this as an i2c-mux. FWIW I'm fine with going
>>>>>> this route instead of using an i2c-mux approach.
>>>>>>
>>>>>> I have been thinking about the i2c-mux approach a bit and the
>>>>>> problem is that we are not really muxing but want to turn on/off
>>>>>> control and AFAIK the i2c-mux framework simply leaves the mux muxed
>>>>>> to the last used i2c-chain, so control will never be released when
>>>>>> the i2c
>>>> transfers are done.
>>>>>>
>>>>>> And if were to somehow modify things (or maybe there already is
>>>>>> some release
>>>>>> callback) then the downside becomes that the i2c-mux core code
>>>>>> operates at the i2c transfer level. So each i2c read/write would
>>>>>> then enable +
>>>> disavle control.
>>>>>>
>>>>>> Modelling this using something like runtime pm as such is a much
>>>>>> better fit because then we request control once on probe /
>>>>>> stream-on and release it once we are fully done, rather then
>>>>>> requesting + releasing control once per i2c- transfer.
>>>>>
>>>>> Seems runtime pm can't fix the problem of initial i2c transfer
>>>>> during sensor driver probe, probably we have to switch to i2c-mux modeling
>> way.
>>>>
>>>> What do you mean? The supplier devices are resumed before the
>>>> driver's probe is called.
>>>
>>> But we setup the link with device_link_add during IVSC driver's probe,
>>> we can't guarantee driver probe's sequence.
>>
>> Then maybe we need to do the device_link_add somewhere else.
>
> sensor's parent is the LJCA I2C device whose driver is being upstream
> https://www.spinics.net/lists/kernel/msg4702552.htmland and sensor's
> power is controlled by IVSC instead of INT3472 if IVSC enabled.
I believe that the INT3472 code is still involved at least on
a Dell Latitude 9420 the INT3472 code still needs to set the
clock-enable and the privacy-LED GPIOs otherwise the main camera won't
work.
So I'm not sure what you mean with "sensor's power is controlled
by IVSC instead of INT3472" ?
> struct device_link *device_link_add(struct device *consumer,
> struct device *supplier, u32 flags)
>
> So probably we have to add above device_link_add in LJCA I2C's driver,
> and we can find the consumer(camera sensor) with ACPI API, but the
> supplier, mei_ace, is mei client device under mei framework and it's
> dynamically allocated device instead of ACPI device, probably I can find
> its parent with some ACPI lookup from this LJCA I2C device, but
> unfortunately mei framework doesn't export the API to find mei client
> device with its parent bus device(struct mei_device).
>
> I'm not sure if modeling this mei_ace as LJCA I2C's runtime power
> control is acceptable, if yes, probably this mei_ace driver have to go with
> LJCA I2C device driver.
Looking at the ACPI table the sensor ACPI device has 2 _DEP-s listed
the I2C controller and the INT3472 device. Since we are already doing
similar setup in the INT3472 device that seems like a good place
to add the device_link()-s (it can return -EPROBE_DEFER to wait
for the mei_ace to show up).
But when the INT3472 code runs, the consumer device does not exist
yet and AFAICT the same is true when the LCJA i2c-controller driver
is getting registered. The consumer only exists when the i2c_client
is instantiated and at that point the sensor drivers probe() method
can run immediately and we are too late to add the device_link.
As a hobby project I have been working on atomisp2 support and
I have a similar issue there. There is no INT3472 device there,
but there is a _DSM method which needs to be used to figure out
which ACPI GPIO resource is reset / powerdown and if the GPIOs
are active-low or active high.
I have written a little helper function to call the _DSM and
to then turn this into lookups and call devm_acpi_dev_add_driver_gpios().
Since on atomisp2 we cannot use the INT3472 driver to delay
the sensor-driver probe and have the INT3472 driver setup
the GPIO lookup, at least for the sensor drivers used with
atomisp2 there is going to be a need to add a single line
to probe() like this:
v4l2_get_acpi_sensor_info(&i2c_client->dev, NULL);
To me it sounds like we need to do something similar here
and extend the helper function which I have written
(but not yet submitted upstream) :
https://github.com/jwrdegoede/linux-sunxi/commit/e2287979db43d46fa7d354c1bde92eb6219b613d
To also setup the device-links needed for the runtime-pm
solution to getting the i2c passed through to the sensor.
Ideally v4l2_get_acpi_sensor_info() should return void
(easier to use in the sensor drivers) but I think it should return
an int, so that it can e.g. return -EPROBE_DEFER to wait for
the mei_ace.
Regards,
Hans
>> The mainline kernel delays probing of camera sensors on Intel platforms until
>> the INT3472 driver has probed the INT3472 device on which the sensors have an
>> ACPI _DEP.
>>
>> This is already used to make sure that clock lookups and regulator info is in place
>> before the sensor's probe() function runs.
>>
>> So that when the driver does clk_get() it succeeds and so that regulator_get()
>> does not end up returning a dummy regulator.
>>
>> So I think the code adding the device_link-s for the IVSC should be added
>> to: drivers/platform/x86/intel/int3472/discrete.c and then the runtime-resume
>> will happen before the sensor's probe() function runs.
>>
>> Likewise drivers/platform/x86/intel/int3472/discrete.c should also ensure that
>> the ivsc driver's probe() has run before it calls acpi_dev_clear_dependencies().
>>
>> The acpi_dev_clear_dependencies() call in discrete.c tells the ACPI subsystem to
>> go ahead and create the i2c-clients for the sensors and allow the sensor drivers
>> to get loaded and probe the sensor.
>>
>> Regards,
>>
>> Hans
>
Hi
> -----Original Message-----
> From: Hans de Goede <[email protected]>
> Sent: Thursday, March 9, 2023 5:28 PM
>
> Hi,
>
> On 3/9/23 02:08, Wu, Wentong wrote:
> >
> >
> >> -----Original Message-----
> >> From: Hans de Goede <[email protected]>
> >> Sent: Tuesday, March 7, 2023 5:10 PM
> >>
> >> Hi,
> >>
> >> On 3/7/23 09:40, Wu, Wentong wrote:
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: Sakari Ailus <[email protected]>
> >>>> Sent: Tuesday, March 7, 2023 4:30 PM
> >>>>
> >>>> Hi Wentong,
> >>>>
> >>>> On Tue, Mar 07, 2023 at 08:17:04AM +0000, Wu, Wentong wrote:
> >>>>>
> >>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Hans de Goede <[email protected]>
> >>>>>> Sent: Wednesday, March 1, 2023 6:42 PM
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> On 3/1/23 11:34, Sakari Ailus wrote:
> >>>>>>> Hi Wentong,
> >>>>>>>
> >>>>>>> On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
> >>>>>>>> Intel Visual Sensing Controller (IVSC), codenamed "Clover
> >>>>>>>> Falls", is a companion chip designed to provide secure and low
> >>>>>>>> power vision capability to IA platforms. IVSC is available in
> >>>>>>>> existing commercial platforms from multiple OEMs.
> >>>>>>>>
> >>>>>>>> The primary use case of IVSC is to bring in context awareness.
> >>>>>>>> IVSC interfaces directly with the platform main camera sensor
> >>>>>>>> via a CSI-2 link and processes the image data with the embedded
> >>>>>>>> AI engine. The detected events are sent over I2C to ISH (Intel
> >>>>>>>> Sensor Hub) for additional data fusion from multiple sensors.
> >>>>>>>> The fusion results are used to implement advanced use cases like:
> >>>>>>>> - Face detection to unlock screen
> >>>>>>>> - Detect user presence to manage backlight setting or waking
> >>>>>>>> up system
> >>>>>>>>
> >>>>>>>> Since the Image Processing Unit(IPU) used on the host processor
> >>>>>>>> needs to configure the CSI-2 link in normal camera usages, the
> >>>>>>>> CSI-2 link and camera sensor can only be used in
> >>>>>>>> mutually-exclusive ways by host IPU and IVSC. By default the
> >>>>>>>> IVSC owns the CSI-2 link and camera sensor. The IPU driver can
> >>>>>>>> take ownership of the CSI-2 link and camera sensor using
> >>>>>>>> interfaces provided
> >>>> by this IVSC driver.
> >>>>>>>>
> >>>>>>>> Switching ownership requires an interface with two different
> >>>>>>>> hardware modules inside IVSC. The software interface to these
> >>>>>>>> modules is via Intel MEI (The Intel Management Engine) commands.
> >>>>>>>> These two hardware modules have two different MEI UUIDs to
> >>>>>>>> enumerate. These hardware
> >>>>>> modules are:
> >>>>>>>> - ACE (Algorithm Context Engine): This module is for algorithm
> >>>>>>>> computing when IVSC owns camera sensor. Also ACE module
> >>>>>>>> controls camera sensor's ownership. This hardware module is
> >>>>>>>> used to set ownership
> >>>>>> of camera sensor.
> >>>>>>>> - CSI (Camera Serial Interface): This module is used to route
> >>>>>>>> camera sensor data either to IVSC or to host for IPU driver and
> >>>> application.
> >>>>>>>>
> >>>>>>>> IVSC also provides a privacy mode. When privacy mode is turned
> >>>>>>>> on, camera sensor can't be used. This means that both ACE and
> >>>>>>>> host IPU can't get image data. And when this mode is turned on,
> >>>>>>>> host IPU driver is informed via a registered callback, so that
> >>>>>>>> user can be
> >>>> notified.
> >>>>>>>>
> >>>>>>>> In summary, to acquire ownership of camera by IPU driver, first
> >>>>>>>> ACE module needs to be informed of ownership and then to setup
> >>>>>>>> MIPI CSI-2 link for the camera sensor and IPU.
> >>>>>>>
> >>>>>>> I thought this for a while and did some research, and I can
> >>>>>>> suggest the
> >>>>>>> following:
> >>>>>>>
> >>>>>>> - The IVSC sub-device implements a control for privacy
> >> (V4L2_CID_PRIVACY
> >>>>>>> is a good fit).
> >>>>>>>
> >>>>>>> - Camera sensor access needs to be requested from IVSC before
> >>>>>>> accessing
> >>>> the
> >>>>>>> sensor via I²C. The IVSC ownership control needs to be in the right
> >>>>>>> setting for this to work, and device links can be used for that purpose
> >>>>>>> (see device_link_add()). With DL_FLAG_PM_RUNTIME and
> >>>>>> DL_FLAG_RPM_ACTIVE,
> >>>>>>> the supplier devices will be PM runtime resumed before the consumer
> >>>>>>> (camera sensor). As these devices are purely virtual on host side and
> has
> >>>>>>> no power state as such, you can use runtime PM callbacks to
> >>>>>>> transfer
> >> the
> >>>>>>> ownership.
> >>>>>>
> >>>>>> Interesting proposal to use device-links + runtime-pm for this
> >>>>>> instead of modelling this as an i2c-mux. FWIW I'm fine with going
> >>>>>> this route instead of using an i2c-mux approach.
> >>>>>>
> >>>>>> I have been thinking about the i2c-mux approach a bit and the
> >>>>>> problem is that we are not really muxing but want to turn on/off
> >>>>>> control and AFAIK the i2c-mux framework simply leaves the mux
> >>>>>> muxed to the last used i2c-chain, so control will never be
> >>>>>> released when the i2c
> >>>> transfers are done.
> >>>>>>
> >>>>>> And if were to somehow modify things (or maybe there already is
> >>>>>> some release
> >>>>>> callback) then the downside becomes that the i2c-mux core code
> >>>>>> operates at the i2c transfer level. So each i2c read/write would
> >>>>>> then enable +
> >>>> disavle control.
> >>>>>>
> >>>>>> Modelling this using something like runtime pm as such is a much
> >>>>>> better fit because then we request control once on probe /
> >>>>>> stream-on and release it once we are fully done, rather then
> >>>>>> requesting + releasing control once per i2c- transfer.
> >>>>>
> >>>>> Seems runtime pm can't fix the problem of initial i2c transfer
> >>>>> during sensor driver probe, probably we have to switch to i2c-mux
> >>>>> modeling
> >> way.
> >>>>
> >>>> What do you mean? The supplier devices are resumed before the
> >>>> driver's probe is called.
> >>>
> >>> But we setup the link with device_link_add during IVSC driver's
> >>> probe, we can't guarantee driver probe's sequence.
> >>
> >> Then maybe we need to do the device_link_add somewhere else.
> >
> > sensor's parent is the LJCA I2C device whose driver is being upstream
> > https://www.spinics.net/lists/kernel/msg4702552.htmland and sensor's
> > power is controlled by IVSC instead of INT3472 if IVSC enabled.
>
> I believe that the INT3472 code is still involved at least on a Dell Latitude 9420
> the INT3472 code still needs to set the clock-enable and the privacy-LED GPIOs
> otherwise the main camera won't work.
>
> So I'm not sure what you mean with "sensor's power is controlled by IVSC
> instead of INT3472" ?
Could you please share your kernel log and DSDT? Thanks
BR,
Wentong
>
>
> > struct device_link *device_link_add(struct device *consumer,
> > struct device *supplier, u32
> > flags)
> >
> > So probably we have to add above device_link_add in LJCA I2C's driver,
> > and we can find the consumer(camera sensor) with ACPI API, but the
> > supplier, mei_ace, is mei client device under mei framework and it's
> > dynamically allocated device instead of ACPI device, probably I can
> > find its parent with some ACPI lookup from this LJCA I2C device, but
> > unfortunately mei framework doesn't export the API to find mei client
> > device with its parent bus device(struct mei_device).
> >
> > I'm not sure if modeling this mei_ace as LJCA I2C's runtime power
> > control is acceptable, if yes, probably this mei_ace driver have to go
> > with LJCA I2C device driver.
>
> Looking at the ACPI table the sensor ACPI device has 2 _DEP-s listed the I2C
> controller and the INT3472 device. Since we are already doing similar setup in
> the INT3472 device that seems like a good place to add the device_link()-s (it can
> return -EPROBE_DEFER to wait for the mei_ace to show up).
>
> But when the INT3472 code runs, the consumer device does not exist yet and
> AFAICT the same is true when the LCJA i2c-controller driver is getting registered.
> The consumer only exists when the i2c_client is instantiated and at that point
> the sensor drivers probe() method can run immediately and we are too late to
> add the device_link.
>
> As a hobby project I have been working on atomisp2 support and I have a similar
> issue there. There is no INT3472 device there, but there is a _DSM method which
> needs to be used to figure out which ACPI GPIO resource is reset / powerdown
> and if the GPIOs are active-low or active high.
>
> I have written a little helper function to call the _DSM and to then turn this into
> lookups and call devm_acpi_dev_add_driver_gpios().
>
> Since on atomisp2 we cannot use the INT3472 driver to delay the sensor-driver
> probe and have the INT3472 driver setup the GPIO lookup, at least for the
> sensor drivers used with
> atomisp2 there is going to be a need to add a single line to probe() like this:
>
> v4l2_get_acpi_sensor_info(&i2c_client->dev, NULL);
>
> To me it sounds like we need to do something similar here and extend the helper
> function which I have written (but not yet submitted upstream) :
>
> https://github.com/jwrdegoede/linux-
> sunxi/commit/e2287979db43d46fa7d354c1bde92eb6219b613d
>
> To also setup the device-links needed for the runtime-pm solution to getting the
> i2c passed through to the sensor.
>
> Ideally v4l2_get_acpi_sensor_info() should return void (easier to use in the
> sensor drivers) but I think it should return an int, so that it can e.g. return -
> EPROBE_DEFER to wait for the mei_ace.
>
> Regards,
>
> Hans
>
>
>
>
> >> The mainline kernel delays probing of camera sensors on Intel
> >> platforms until the INT3472 driver has probed the INT3472 device on
> >> which the sensors have an ACPI _DEP.
> >>
> >> This is already used to make sure that clock lookups and regulator
> >> info is in place before the sensor's probe() function runs.
> >>
> >> So that when the driver does clk_get() it succeeds and so that
> >> regulator_get() does not end up returning a dummy regulator.
> >>
> >> So I think the code adding the device_link-s for the IVSC should be
> >> added
> >> to: drivers/platform/x86/intel/int3472/discrete.c and then the
> >> runtime-resume will happen before the sensor's probe() function runs.
> >>
> >> Likewise drivers/platform/x86/intel/int3472/discrete.c should also
> >> ensure that the ivsc driver's probe() has run before it calls
> acpi_dev_clear_dependencies().
> >>
> >> The acpi_dev_clear_dependencies() call in discrete.c tells the ACPI
> >> subsystem to go ahead and create the i2c-clients for the sensors and
> >> allow the sensor drivers to get loaded and probe the sensor.
> >>
> >> Regards,
> >>
> >> Hans
> >
<re-added the previous Cc list, which I dropped because of the large attachments>
Hi Wentong,
On 3/9/23 15:29, Wu, Wentong wrote:
> Hi Hans,
>
> Thanks
>
> And AFAICT, there is no IVSC device on your Dell Latitude 9420 where the platform is based on TGL instead of ADL, and I have never heard IVSC runs on TGL, if no IVSC, INT3472 will control sensor's power.
> And I will double confirm with people who know dell product well tomorrow.
Ah, I was under the impression that there was an IVSC there because:
1. The sensor driver for the used sensor (tries to) poke the IVSC
2. Things did not work without building the IVSC drivers, but that might
be due to a dependency on the LCJA GPIO expander instead
But you might very well be right, that would also explain the
"intel vsc not ready" messages in dmesg.
If with the IVSC case the IVSC controls the power to the sensor too,
then another option might be to model the I2C-switch + the power-control
as a powerdown GPIO for the sensor, which most sensor drivers already
try to use. The advantage of doing this would be that GPIO lookups
can reference the GPIO provider + consumer by device-name so then
we don't need to have both devices instantiated at the time of
adding the GPIO lookup. And in that case we could e.g. add the lookup
before registering the I2C controller.
Sakari, what do you think of instead of using runtime-pm + devlinks
having the IVSC code export a powerdown GPIO to the sensor ?
This also decouples the ivsc powerstate from the sensor power-state
which might be useful if we ever want to use some of the more
advanced ivsc features, where AFAICT the ivsc fully controls the sensor.
Regards,
Hans
>> -----Original Message-----
>> From: Hans de Goede <[email protected]>
>> Sent: Thursday, March 9, 2023 9:30 PM
>> To: Wu, Wentong <[email protected]>
>> Subject: Re: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel Visual
>> Sensing Controller(IVSC)
>>
>> Hi Wentong,
>>
>> Attached are the requested dmesg + acpidump for the Dell Latitude 9420.
>>
>> Regards,
>>
>> Hans
>>
>>
>>
>>
>> On 3/9/23 14:21, Wu, Wentong wrote:
>>> Hi
>>>
>>>> -----Original Message-----
>>>> From: Hans de Goede <[email protected]>
>>>> Sent: Thursday, March 9, 2023 5:28 PM
>>>>
>>>> Hi,
>>>>
>>>> On 3/9/23 02:08, Wu, Wentong wrote:
>>>>>
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Hans de Goede <[email protected]>
>>>>>> Sent: Tuesday, March 7, 2023 5:10 PM
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> On 3/7/23 09:40, Wu, Wentong wrote:
>>>>>>>
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Sakari Ailus <[email protected]>
>>>>>>>> Sent: Tuesday, March 7, 2023 4:30 PM
>>>>>>>>
>>>>>>>> Hi Wentong,
>>>>>>>>
>>>>>>>> On Tue, Mar 07, 2023 at 08:17:04AM +0000, Wu, Wentong wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Hans de Goede <[email protected]>
>>>>>>>>>> Sent: Wednesday, March 1, 2023 6:42 PM
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> On 3/1/23 11:34, Sakari Ailus wrote:
>>>>>>>>>>> Hi Wentong,
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
>>>>>>>>>>>> Intel Visual Sensing Controller (IVSC), codenamed "Clover
>>>>>>>>>>>> Falls", is a companion chip designed to provide secure and
>>>>>>>>>>>> low power vision capability to IA platforms. IVSC is
>>>>>>>>>>>> available in existing commercial platforms from multiple OEMs.
>>>>>>>>>>>>
>>>>>>>>>>>> The primary use case of IVSC is to bring in context awareness.
>>>>>>>>>>>> IVSC interfaces directly with the platform main camera sensor
>>>>>>>>>>>> via a CSI-2 link and processes the image data with the
>>>>>>>>>>>> embedded AI engine. The detected events are sent over I2C to
>>>>>>>>>>>> ISH (Intel Sensor Hub) for additional data fusion from multiple
>> sensors.
>>>>>>>>>>>> The fusion results are used to implement advanced use cases like:
>>>>>>>>>>>> - Face detection to unlock screen
>>>>>>>>>>>> - Detect user presence to manage backlight setting or waking
>>>>>>>>>>>> up system
>>>>>>>>>>>>
>>>>>>>>>>>> Since the Image Processing Unit(IPU) used on the host
>>>>>>>>>>>> processor needs to configure the CSI-2 link in normal camera
>>>>>>>>>>>> usages, the
>>>>>>>>>>>> CSI-2 link and camera sensor can only be used in
>>>>>>>>>>>> mutually-exclusive ways by host IPU and IVSC. By default the
>>>>>>>>>>>> IVSC owns the CSI-2 link and camera sensor. The IPU driver
>>>>>>>>>>>> can take ownership of the CSI-2 link and camera sensor using
>>>>>>>>>>>> interfaces provided
>>>>>>>> by this IVSC driver.
>>>>>>>>>>>>
>>>>>>>>>>>> Switching ownership requires an interface with two different
>>>>>>>>>>>> hardware modules inside IVSC. The software interface to these
>>>>>>>>>>>> modules is via Intel MEI (The Intel Management Engine) commands.
>>>>>>>>>>>> These two hardware modules have two different MEI UUIDs to
>>>>>>>>>>>> enumerate. These hardware
>>>>>>>>>> modules are:
>>>>>>>>>>>> - ACE (Algorithm Context Engine): This module is for
>>>>>>>>>>>> algorithm computing when IVSC owns camera sensor. Also ACE
>>>>>>>>>>>> module controls camera sensor's ownership. This hardware
>>>>>>>>>>>> module is used to set ownership
>>>>>>>>>> of camera sensor.
>>>>>>>>>>>> - CSI (Camera Serial Interface): This module is used to
>>>>>>>>>>>> route camera sensor data either to IVSC or to host for IPU
>>>>>>>>>>>> driver and
>>>>>>>> application.
>>>>>>>>>>>>
>>>>>>>>>>>> IVSC also provides a privacy mode. When privacy mode is
>>>>>>>>>>>> turned on, camera sensor can't be used. This means that both
>>>>>>>>>>>> ACE and host IPU can't get image data. And when this mode is
>>>>>>>>>>>> turned on, host IPU driver is informed via a registered
>>>>>>>>>>>> callback, so that user can be
>>>>>>>> notified.
>>>>>>>>>>>>
>>>>>>>>>>>> In summary, to acquire ownership of camera by IPU driver,
>>>>>>>>>>>> first ACE module needs to be informed of ownership and then
>>>>>>>>>>>> to setup MIPI CSI-2 link for the camera sensor and IPU.
>>>>>>>>>>>
>>>>>>>>>>> I thought this for a while and did some research, and I can
>>>>>>>>>>> suggest the
>>>>>>>>>>> following:
>>>>>>>>>>>
>>>>>>>>>>> - The IVSC sub-device implements a control for privacy
>>>>>> (V4L2_CID_PRIVACY
>>>>>>>>>>> is a good fit).
>>>>>>>>>>>
>>>>>>>>>>> - Camera sensor access needs to be requested from IVSC before
>>>>>>>>>>> accessing
>>>>>>>> the
>>>>>>>>>>> sensor via I²C. The IVSC ownership control needs to be in the right
>>>>>>>>>>> setting for this to work, and device links can be used for that
>> purpose
>>>>>>>>>>> (see device_link_add()). With DL_FLAG_PM_RUNTIME and
>>>>>>>>>> DL_FLAG_RPM_ACTIVE,
>>>>>>>>>>> the supplier devices will be PM runtime resumed before the
>> consumer
>>>>>>>>>>> (camera sensor). As these devices are purely virtual on host
>>>>>>>>>>> side and
>>>> has
>>>>>>>>>>> no power state as such, you can use runtime PM callbacks to
>>>>>>>>>>> transfer
>>>>>> the
>>>>>>>>>>> ownership.
>>>>>>>>>>
>>>>>>>>>> Interesting proposal to use device-links + runtime-pm for this
>>>>>>>>>> instead of modelling this as an i2c-mux. FWIW I'm fine with
>>>>>>>>>> going this route instead of using an i2c-mux approach.
>>>>>>>>>>
>>>>>>>>>> I have been thinking about the i2c-mux approach a bit and the
>>>>>>>>>> problem is that we are not really muxing but want to turn
>>>>>>>>>> on/off control and AFAIK the i2c-mux framework simply leaves
>>>>>>>>>> the mux muxed to the last used i2c-chain, so control will never
>>>>>>>>>> be released when the i2c
>>>>>>>> transfers are done.
>>>>>>>>>>
>>>>>>>>>> And if were to somehow modify things (or maybe there already is
>>>>>>>>>> some release
>>>>>>>>>> callback) then the downside becomes that the i2c-mux core code
>>>>>>>>>> operates at the i2c transfer level. So each i2c read/write
>>>>>>>>>> would then enable +
>>>>>>>> disavle control.
>>>>>>>>>>
>>>>>>>>>> Modelling this using something like runtime pm as such is a
>>>>>>>>>> much better fit because then we request control once on probe /
>>>>>>>>>> stream-on and release it once we are fully done, rather then
>>>>>>>>>> requesting + releasing control once per i2c- transfer.
>>>>>>>>>
>>>>>>>>> Seems runtime pm can't fix the problem of initial i2c transfer
>>>>>>>>> during sensor driver probe, probably we have to switch to
>>>>>>>>> i2c-mux modeling
>>>>>> way.
>>>>>>>>
>>>>>>>> What do you mean? The supplier devices are resumed before the
>>>>>>>> driver's probe is called.
>>>>>>>
>>>>>>> But we setup the link with device_link_add during IVSC driver's
>>>>>>> probe, we can't guarantee driver probe's sequence.
>>>>>>
>>>>>> Then maybe we need to do the device_link_add somewhere else.
>>>>>
>>>>> sensor's parent is the LJCA I2C device whose driver is being
>>>>> upstream https://www.spinics.net/lists/kernel/msg4702552.htmland and
>>>>> sensor's power is controlled by IVSC instead of INT3472 if IVSC enabled.
>>>>
>>>> I believe that the INT3472 code is still involved at least on a Dell
>>>> Latitude 9420 the INT3472 code still needs to set the clock-enable
>>>> and the privacy-LED GPIOs otherwise the main camera won't work.
>>>>
>>>> So I'm not sure what you mean with "sensor's power is controlled by
>>>> IVSC instead of INT3472" ?
>>>
>>> Could you please share your kernel log and DSDT? Thanks
>>>
>>> BR,
>>> Wentong
>>>>
>>>>
>>>>> struct device_link *device_link_add(struct device *consumer,
>>>>> struct device *supplier, u32
>>>>> flags)
>>>>>
>>>>> So probably we have to add above device_link_add in LJCA I2C's
>>>>> driver, and we can find the consumer(camera sensor) with ACPI API,
>>>>> but the supplier, mei_ace, is mei client device under mei framework
>>>>> and it's dynamically allocated device instead of ACPI device,
>>>>> probably I can find its parent with some ACPI lookup from this LJCA
>>>>> I2C device, but unfortunately mei framework doesn't export the API
>>>>> to find mei client device with its parent bus device(struct mei_device).
>>>>>
>>>>> I'm not sure if modeling this mei_ace as LJCA I2C's runtime power
>>>>> control is acceptable, if yes, probably this mei_ace driver have to
>>>>> go with LJCA I2C device driver.
>>>>
>>>> Looking at the ACPI table the sensor ACPI device has 2 _DEP-s listed
>>>> the I2C controller and the INT3472 device. Since we are already doing
>>>> similar setup in the INT3472 device that seems like a good place to
>>>> add the device_link()-s (it can return -EPROBE_DEFER to wait for the mei_ace
>> to show up).
>>>>
>>>> But when the INT3472 code runs, the consumer device does not exist
>>>> yet and AFAICT the same is true when the LCJA i2c-controller driver is getting
>> registered.
>>>> The consumer only exists when the i2c_client is instantiated and at
>>>> that point the sensor drivers probe() method can run immediately and
>>>> we are too late to add the device_link.
>>>>
>>>> As a hobby project I have been working on atomisp2 support and I have
>>>> a similar issue there. There is no INT3472 device there, but there is
>>>> a _DSM method which needs to be used to figure out which ACPI GPIO
>>>> resource is reset / powerdown and if the GPIOs are active-low or active high.
>>>>
>>>> I have written a little helper function to call the _DSM and to then
>>>> turn this into lookups and call devm_acpi_dev_add_driver_gpios().
>>>>
>>>> Since on atomisp2 we cannot use the INT3472 driver to delay the
>>>> sensor-driver probe and have the INT3472 driver setup the GPIO
>>>> lookup, at least for the sensor drivers used with
>>>> atomisp2 there is going to be a need to add a single line to probe() like this:
>>>>
>>>> v4l2_get_acpi_sensor_info(&i2c_client->dev, NULL);
>>>>
>>>> To me it sounds like we need to do something similar here and extend
>>>> the helper function which I have written (but not yet submitted upstream) :
>>>>
>>>> https://github.com/jwrdegoede/linux-
>>>> sunxi/commit/e2287979db43d46fa7d354c1bde92eb6219b613d
>>>>
>>>> To also setup the device-links needed for the runtime-pm solution to
>>>> getting the i2c passed through to the sensor.
>>>>
>>>> Ideally v4l2_get_acpi_sensor_info() should return void (easier to use
>>>> in the sensor drivers) but I think it should return an int, so that
>>>> it can e.g. return - EPROBE_DEFER to wait for the mei_ace.
>>>>
>>>> Regards,
>>>>
>>>> Hans
>>>>
>>>>
>>>>
>>>>
>>>>>> The mainline kernel delays probing of camera sensors on Intel
>>>>>> platforms until the INT3472 driver has probed the INT3472 device on
>>>>>> which the sensors have an ACPI _DEP.
>>>>>>
>>>>>> This is already used to make sure that clock lookups and regulator
>>>>>> info is in place before the sensor's probe() function runs.
>>>>>>
>>>>>> So that when the driver does clk_get() it succeeds and so that
>>>>>> regulator_get() does not end up returning a dummy regulator.
>>>>>>
>>>>>> So I think the code adding the device_link-s for the IVSC should be
>>>>>> added
>>>>>> to: drivers/platform/x86/intel/int3472/discrete.c and then the
>>>>>> runtime-resume will happen before the sensor's probe() function runs.
>>>>>>
>>>>>> Likewise drivers/platform/x86/intel/int3472/discrete.c should also
>>>>>> ensure that the ivsc driver's probe() has run before it calls
>>>> acpi_dev_clear_dependencies().
>>>>>>
>>>>>> The acpi_dev_clear_dependencies() call in discrete.c tells the ACPI
>>>>>> subsystem to go ahead and create the i2c-clients for the sensors
>>>>>> and allow the sensor drivers to get loaded and probe the sensor.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Hans
>>>>>
>>>
> -----Original Message-----
> From: Hans de Goede <[email protected]>
> Sent: Thursday, March 9, 2023 11:24 PM
>
> <re-added the previous Cc list, which I dropped because of the large
> attachments>
>
> Hi Wentong,
>
> On 3/9/23 15:29, Wu, Wentong wrote:
> > Hi Hans,
> >
> > Thanks
> >
> > And AFAICT, there is no IVSC device on your Dell Latitude 9420 where the
> platform is based on TGL instead of ADL, and I have never heard IVSC runs on
> TGL, if no IVSC, INT3472 will control sensor's power.
> > And I will double confirm with people who know dell product well tomorrow.
>
> Ah, I was under the impression that there was an IVSC there because:
>
> 1. The sensor driver for the used sensor (tries to) poke the IVSC 2. Things did not
> work without building the IVSC drivers, but that might
> be due to a dependency on the LCJA GPIO expander instead
Below is your dmesg log, the required SPI controller for IVSC isn't here.
[ 35.538114] ljca 2-6:1.0: acked sem wait timed out ret:0 timeout:20 ack:0
[ 35.538129] ljca 2-6:1.0: MNG_ENUM_SPI failed ret:-110 len:7 num:0
[ 35.538621] ljca 2-6:1.0: LJCA USB device init success
[ 35.538776] usbcore: registered new interface driver ljca
Also I checked your SSDT, there is no IVSC device and the sensor device depends on
INT3472 instead of IVSC device as on my setup.
>
> But you might very well be right, that would also explain the "intel vsc not ready"
> messages in dmesg.
>
> If with the IVSC case the IVSC controls the power to the sensor too, then
> another option might be to model the I2C-switch + the power-control as a
> powerdown GPIO for the sensor, which most sensor drivers already try to use.
> The advantage of doing this would be that GPIO lookups can reference the GPIO
> provider + consumer by device-name so then we don't need to have both
> devices instantiated at the time of
> adding the GPIO lookup. And in that case we could e.g. add the lookup
> before registering the I2C controller.
Can we add IVSC device to acpi_honor_dep_ids, so that when everything is done during
mei_ace probe, acpi_dev_clear_dependencies can make sensor start probe?
BR,
Wentong
>
> Sakari, what do you think of instead of using runtime-pm + devlinks having the
> IVSC code export a powerdown GPIO to the sensor ?
>
> This also decouples the ivsc powerstate from the sensor power-state which
> might be useful if we ever want to use some of the more advanced ivsc features,
> where AFAICT the ivsc fully controls the sensor.
>
> Regards,
>
> Hans
>
>
>
>
> >> -----Original Message-----
> >> From: Hans de Goede <[email protected]>
> >> Sent: Thursday, March 9, 2023 9:30 PM
> >> To: Wu, Wentong <[email protected]>
> >> Subject: Re: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of
> >> Intel Visual Sensing Controller(IVSC)
> >>
> >> Hi Wentong,
> >>
> >> Attached are the requested dmesg + acpidump for the Dell Latitude 9420.
> >>
> >> Regards,
> >>
> >> Hans
> >>
> >>
> >>
> >>
> >> On 3/9/23 14:21, Wu, Wentong wrote:
> >>> Hi
> >>>
> >>>> -----Original Message-----
> >>>> From: Hans de Goede <[email protected]>
> >>>> Sent: Thursday, March 9, 2023 5:28 PM
> >>>>
> >>>> Hi,
> >>>>
> >>>> On 3/9/23 02:08, Wu, Wentong wrote:
> >>>>>
> >>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Hans de Goede <[email protected]>
> >>>>>> Sent: Tuesday, March 7, 2023 5:10 PM
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> On 3/7/23 09:40, Wu, Wentong wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>> -----Original Message-----
> >>>>>>>> From: Sakari Ailus <[email protected]>
> >>>>>>>> Sent: Tuesday, March 7, 2023 4:30 PM
> >>>>>>>>
> >>>>>>>> Hi Wentong,
> >>>>>>>>
> >>>>>>>> On Tue, Mar 07, 2023 at 08:17:04AM +0000, Wu, Wentong wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> -----Original Message-----
> >>>>>>>>>> From: Hans de Goede <[email protected]>
> >>>>>>>>>> Sent: Wednesday, March 1, 2023 6:42 PM
> >>>>>>>>>>
> >>>>>>>>>> Hi,
> >>>>>>>>>>
> >>>>>>>>>> On 3/1/23 11:34, Sakari Ailus wrote:
> >>>>>>>>>>> Hi Wentong,
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
> >>>>>>>>>>>> Intel Visual Sensing Controller (IVSC), codenamed "Clover
> >>>>>>>>>>>> Falls", is a companion chip designed to provide secure and
> >>>>>>>>>>>> low power vision capability to IA platforms. IVSC is
> >>>>>>>>>>>> available in existing commercial platforms from multiple OEMs.
> >>>>>>>>>>>>
> >>>>>>>>>>>> The primary use case of IVSC is to bring in context awareness.
> >>>>>>>>>>>> IVSC interfaces directly with the platform main camera
> >>>>>>>>>>>> sensor via a CSI-2 link and processes the image data with
> >>>>>>>>>>>> the embedded AI engine. The detected events are sent over
> >>>>>>>>>>>> I2C to ISH (Intel Sensor Hub) for additional data fusion
> >>>>>>>>>>>> from multiple
> >> sensors.
> >>>>>>>>>>>> The fusion results are used to implement advanced use cases like:
> >>>>>>>>>>>> - Face detection to unlock screen
> >>>>>>>>>>>> - Detect user presence to manage backlight setting or
> >>>>>>>>>>>> waking up system
> >>>>>>>>>>>>
> >>>>>>>>>>>> Since the Image Processing Unit(IPU) used on the host
> >>>>>>>>>>>> processor needs to configure the CSI-2 link in normal
> >>>>>>>>>>>> camera usages, the
> >>>>>>>>>>>> CSI-2 link and camera sensor can only be used in
> >>>>>>>>>>>> mutually-exclusive ways by host IPU and IVSC. By default
> >>>>>>>>>>>> the IVSC owns the CSI-2 link and camera sensor. The IPU
> >>>>>>>>>>>> driver can take ownership of the CSI-2 link and camera
> >>>>>>>>>>>> sensor using interfaces provided
> >>>>>>>> by this IVSC driver.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Switching ownership requires an interface with two
> >>>>>>>>>>>> different hardware modules inside IVSC. The software
> >>>>>>>>>>>> interface to these modules is via Intel MEI (The Intel
> Management Engine) commands.
> >>>>>>>>>>>> These two hardware modules have two different MEI UUIDs to
> >>>>>>>>>>>> enumerate. These hardware
> >>>>>>>>>> modules are:
> >>>>>>>>>>>> - ACE (Algorithm Context Engine): This module is for
> >>>>>>>>>>>> algorithm computing when IVSC owns camera sensor. Also ACE
> >>>>>>>>>>>> module controls camera sensor's ownership. This hardware
> >>>>>>>>>>>> module is used to set ownership
> >>>>>>>>>> of camera sensor.
> >>>>>>>>>>>> - CSI (Camera Serial Interface): This module is used to
> >>>>>>>>>>>> route camera sensor data either to IVSC or to host for IPU
> >>>>>>>>>>>> driver and
> >>>>>>>> application.
> >>>>>>>>>>>>
> >>>>>>>>>>>> IVSC also provides a privacy mode. When privacy mode is
> >>>>>>>>>>>> turned on, camera sensor can't be used. This means that
> >>>>>>>>>>>> both ACE and host IPU can't get image data. And when this
> >>>>>>>>>>>> mode is turned on, host IPU driver is informed via a
> >>>>>>>>>>>> registered callback, so that user can be
> >>>>>>>> notified.
> >>>>>>>>>>>>
> >>>>>>>>>>>> In summary, to acquire ownership of camera by IPU driver,
> >>>>>>>>>>>> first ACE module needs to be informed of ownership and then
> >>>>>>>>>>>> to setup MIPI CSI-2 link for the camera sensor and IPU.
> >>>>>>>>>>>
> >>>>>>>>>>> I thought this for a while and did some research, and I can
> >>>>>>>>>>> suggest the
> >>>>>>>>>>> following:
> >>>>>>>>>>>
> >>>>>>>>>>> - The IVSC sub-device implements a control for privacy
> >>>>>> (V4L2_CID_PRIVACY
> >>>>>>>>>>> is a good fit).
> >>>>>>>>>>>
> >>>>>>>>>>> - Camera sensor access needs to be requested from IVSC
> >>>>>>>>>>> before accessing
> >>>>>>>> the
> >>>>>>>>>>> sensor via I²C. The IVSC ownership control needs to be in the
> right
> >>>>>>>>>>> setting for this to work, and device links can be used for
> >>>>>>>>>>> that
> >> purpose
> >>>>>>>>>>> (see device_link_add()). With DL_FLAG_PM_RUNTIME and
> >>>>>>>>>> DL_FLAG_RPM_ACTIVE,
> >>>>>>>>>>> the supplier devices will be PM runtime resumed before the
> >> consumer
> >>>>>>>>>>> (camera sensor). As these devices are purely virtual on
> >>>>>>>>>>> host side and
> >>>> has
> >>>>>>>>>>> no power state as such, you can use runtime PM callbacks
> >>>>>>>>>>> to transfer
> >>>>>> the
> >>>>>>>>>>> ownership.
> >>>>>>>>>>
> >>>>>>>>>> Interesting proposal to use device-links + runtime-pm for
> >>>>>>>>>> this instead of modelling this as an i2c-mux. FWIW I'm fine
> >>>>>>>>>> with going this route instead of using an i2c-mux approach.
> >>>>>>>>>>
> >>>>>>>>>> I have been thinking about the i2c-mux approach a bit and the
> >>>>>>>>>> problem is that we are not really muxing but want to turn
> >>>>>>>>>> on/off control and AFAIK the i2c-mux framework simply leaves
> >>>>>>>>>> the mux muxed to the last used i2c-chain, so control will
> >>>>>>>>>> never be released when the i2c
> >>>>>>>> transfers are done.
> >>>>>>>>>>
> >>>>>>>>>> And if were to somehow modify things (or maybe there already
> >>>>>>>>>> is some release
> >>>>>>>>>> callback) then the downside becomes that the i2c-mux core
> >>>>>>>>>> code operates at the i2c transfer level. So each i2c
> >>>>>>>>>> read/write would then enable +
> >>>>>>>> disavle control.
> >>>>>>>>>>
> >>>>>>>>>> Modelling this using something like runtime pm as such is a
> >>>>>>>>>> much better fit because then we request control once on probe
> >>>>>>>>>> / stream-on and release it once we are fully done, rather
> >>>>>>>>>> then requesting + releasing control once per i2c- transfer.
> >>>>>>>>>
> >>>>>>>>> Seems runtime pm can't fix the problem of initial i2c transfer
> >>>>>>>>> during sensor driver probe, probably we have to switch to
> >>>>>>>>> i2c-mux modeling
> >>>>>> way.
> >>>>>>>>
> >>>>>>>> What do you mean? The supplier devices are resumed before the
> >>>>>>>> driver's probe is called.
> >>>>>>>
> >>>>>>> But we setup the link with device_link_add during IVSC driver's
> >>>>>>> probe, we can't guarantee driver probe's sequence.
> >>>>>>
> >>>>>> Then maybe we need to do the device_link_add somewhere else.
> >>>>>
> >>>>> sensor's parent is the LJCA I2C device whose driver is being
> >>>>> upstream https://www.spinics.net/lists/kernel/msg4702552.htmland
> >>>>> and sensor's power is controlled by IVSC instead of INT3472 if IVSC
> enabled.
> >>>>
> >>>> I believe that the INT3472 code is still involved at least on a
> >>>> Dell Latitude 9420 the INT3472 code still needs to set the
> >>>> clock-enable and the privacy-LED GPIOs otherwise the main camera won't
> work.
> >>>>
> >>>> So I'm not sure what you mean with "sensor's power is controlled by
> >>>> IVSC instead of INT3472" ?
> >>>
> >>> Could you please share your kernel log and DSDT? Thanks
> >>>
> >>> BR,
> >>> Wentong
> >>>>
> >>>>
> >>>>> struct device_link *device_link_add(struct device *consumer,
> >>>>> struct device *supplier, u32
> >>>>> flags)
> >>>>>
> >>>>> So probably we have to add above device_link_add in LJCA I2C's
> >>>>> driver, and we can find the consumer(camera sensor) with ACPI API,
> >>>>> but the supplier, mei_ace, is mei client device under mei
> >>>>> framework and it's dynamically allocated device instead of ACPI
> >>>>> device, probably I can find its parent with some ACPI lookup from
> >>>>> this LJCA I2C device, but unfortunately mei framework doesn't
> >>>>> export the API to find mei client device with its parent bus device(struct
> mei_device).
> >>>>>
> >>>>> I'm not sure if modeling this mei_ace as LJCA I2C's runtime power
> >>>>> control is acceptable, if yes, probably this mei_ace driver have
> >>>>> to go with LJCA I2C device driver.
> >>>>
> >>>> Looking at the ACPI table the sensor ACPI device has 2 _DEP-s
> >>>> listed the I2C controller and the INT3472 device. Since we are
> >>>> already doing similar setup in the INT3472 device that seems like a
> >>>> good place to add the device_link()-s (it can return -EPROBE_DEFER
> >>>> to wait for the mei_ace
> >> to show up).
> >>>>
> >>>> But when the INT3472 code runs, the consumer device does not exist
> >>>> yet and AFAICT the same is true when the LCJA i2c-controller driver
> >>>> is getting
> >> registered.
> >>>> The consumer only exists when the i2c_client is instantiated and at
> >>>> that point the sensor drivers probe() method can run immediately
> >>>> and we are too late to add the device_link.
> >>>>
> >>>> As a hobby project I have been working on atomisp2 support and I
> >>>> have a similar issue there. There is no INT3472 device there, but
> >>>> there is a _DSM method which needs to be used to figure out which
> >>>> ACPI GPIO resource is reset / powerdown and if the GPIOs are active-low
> or active high.
> >>>>
> >>>> I have written a little helper function to call the _DSM and to
> >>>> then turn this into lookups and call devm_acpi_dev_add_driver_gpios().
> >>>>
> >>>> Since on atomisp2 we cannot use the INT3472 driver to delay the
> >>>> sensor-driver probe and have the INT3472 driver setup the GPIO
> >>>> lookup, at least for the sensor drivers used with
> >>>> atomisp2 there is going to be a need to add a single line to probe() like this:
> >>>>
> >>>> v4l2_get_acpi_sensor_info(&i2c_client->dev, NULL);
> >>>>
> >>>> To me it sounds like we need to do something similar here and
> >>>> extend the helper function which I have written (but not yet submitted
> upstream) :
> >>>>
> >>>> https://github.com/jwrdegoede/linux-
> >>>> sunxi/commit/e2287979db43d46fa7d354c1bde92eb6219b613d
> >>>>
> >>>> To also setup the device-links needed for the runtime-pm solution
> >>>> to getting the i2c passed through to the sensor.
> >>>>
> >>>> Ideally v4l2_get_acpi_sensor_info() should return void (easier to
> >>>> use in the sensor drivers) but I think it should return an int, so
> >>>> that it can e.g. return - EPROBE_DEFER to wait for the mei_ace.
> >>>>
> >>>> Regards,
> >>>>
> >>>> Hans
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>> The mainline kernel delays probing of camera sensors on Intel
> >>>>>> platforms until the INT3472 driver has probed the INT3472 device
> >>>>>> on which the sensors have an ACPI _DEP.
> >>>>>>
> >>>>>> This is already used to make sure that clock lookups and
> >>>>>> regulator info is in place before the sensor's probe() function runs.
> >>>>>>
> >>>>>> So that when the driver does clk_get() it succeeds and so that
> >>>>>> regulator_get() does not end up returning a dummy regulator.
> >>>>>>
> >>>>>> So I think the code adding the device_link-s for the IVSC should
> >>>>>> be added
> >>>>>> to: drivers/platform/x86/intel/int3472/discrete.c and then the
> >>>>>> runtime-resume will happen before the sensor's probe() function runs.
> >>>>>>
> >>>>>> Likewise drivers/platform/x86/intel/int3472/discrete.c should
> >>>>>> also ensure that the ivsc driver's probe() has run before it
> >>>>>> calls
> >>>> acpi_dev_clear_dependencies().
> >>>>>>
> >>>>>> The acpi_dev_clear_dependencies() call in discrete.c tells the
> >>>>>> ACPI subsystem to go ahead and create the i2c-clients for the
> >>>>>> sensors and allow the sensor drivers to get loaded and probe the sensor.
> >>>>>>
> >>>>>> Regards,
> >>>>>>
> >>>>>> Hans
> >>>>>
> >>>
> -----Original Message-----
> From: Hans de Goede <[email protected]>
> Sent: Thursday, March 9, 2023 11:24 PM
>
> <re-added the previous Cc list, which I dropped because of the large
> attachments>
>
> Hi Wentong,
>
> On 3/9/23 15:29, Wu, Wentong wrote:
> > Hi Hans,
> >
> > Thanks
> >
> > And AFAICT, there is no IVSC device on your Dell Latitude 9420 where the
> platform is based on TGL instead of ADL, and I have never heard IVSC runs on
> TGL, if no IVSC, INT3472 will control sensor's power.
> > And I will double confirm with people who know dell product well tomorrow.
>
> Ah, I was under the impression that there was an IVSC there because:
>
> 1. The sensor driver for the used sensor (tries to) poke the IVSC 2. Things did not
> work without building the IVSC drivers, but that might
> be due to a dependency on the LCJA GPIO expander instead
>
> But you might very well be right, that would also explain the "intel vsc not ready"
> messages in dmesg.
>
> If with the IVSC case the IVSC controls the power to the sensor too, then
> another option might be to model the I2C-switch + the power-control as a
> powerdown GPIO for the sensor, which most sensor drivers already try to use.
> The advantage of doing this would be that GPIO lookups can reference the GPIO
> provider + consumer by device-name so then we don't need to have both
> devices instantiated at the time of
> adding the GPIO lookup. And in that case we could e.g. add the lookup
> before registering the I2C controller.
Thanks,
So the drivers of sensors connected to IVSC have to add power up/down code.
BR,
Wentong
>
> Sakari, what do you think of instead of using runtime-pm + devlinks having the
> IVSC code export a powerdown GPIO to the sensor ?
>
> This also decouples the ivsc powerstate from the sensor power-state which
> might be useful if we ever want to use some of the more advanced ivsc features,
> where AFAICT the ivsc fully controls the sensor.
>
> Regards,
>
> Hans
>
>
>
>
> >> -----Original Message-----
> >> From: Hans de Goede <[email protected]>
> >> Sent: Thursday, March 9, 2023 9:30 PM
> >> To: Wu, Wentong <[email protected]>
> >> Subject: Re: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of
> >> Intel Visual Sensing Controller(IVSC)
> >>
> >> Hi Wentong,
> >>
> >> Attached are the requested dmesg + acpidump for the Dell Latitude 9420.
> >>
> >> Regards,
> >>
> >> Hans
> >>
> >>
> >>
> >>
> >> On 3/9/23 14:21, Wu, Wentong wrote:
> >>> Hi
> >>>
> >>>> -----Original Message-----
> >>>> From: Hans de Goede <[email protected]>
> >>>> Sent: Thursday, March 9, 2023 5:28 PM
> >>>>
> >>>> Hi,
> >>>>
> >>>> On 3/9/23 02:08, Wu, Wentong wrote:
> >>>>>
> >>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Hans de Goede <[email protected]>
> >>>>>> Sent: Tuesday, March 7, 2023 5:10 PM
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> On 3/7/23 09:40, Wu, Wentong wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>> -----Original Message-----
> >>>>>>>> From: Sakari Ailus <[email protected]>
> >>>>>>>> Sent: Tuesday, March 7, 2023 4:30 PM
> >>>>>>>>
> >>>>>>>> Hi Wentong,
> >>>>>>>>
> >>>>>>>> On Tue, Mar 07, 2023 at 08:17:04AM +0000, Wu, Wentong wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> -----Original Message-----
> >>>>>>>>>> From: Hans de Goede <[email protected]>
> >>>>>>>>>> Sent: Wednesday, March 1, 2023 6:42 PM
> >>>>>>>>>>
> >>>>>>>>>> Hi,
> >>>>>>>>>>
> >>>>>>>>>> On 3/1/23 11:34, Sakari Ailus wrote:
> >>>>>>>>>>> Hi Wentong,
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
> >>>>>>>>>>>> Intel Visual Sensing Controller (IVSC), codenamed "Clover
> >>>>>>>>>>>> Falls", is a companion chip designed to provide secure and
> >>>>>>>>>>>> low power vision capability to IA platforms. IVSC is
> >>>>>>>>>>>> available in existing commercial platforms from multiple OEMs.
> >>>>>>>>>>>>
> >>>>>>>>>>>> The primary use case of IVSC is to bring in context awareness.
> >>>>>>>>>>>> IVSC interfaces directly with the platform main camera
> >>>>>>>>>>>> sensor via a CSI-2 link and processes the image data with
> >>>>>>>>>>>> the embedded AI engine. The detected events are sent over
> >>>>>>>>>>>> I2C to ISH (Intel Sensor Hub) for additional data fusion
> >>>>>>>>>>>> from multiple
> >> sensors.
> >>>>>>>>>>>> The fusion results are used to implement advanced use cases like:
> >>>>>>>>>>>> - Face detection to unlock screen
> >>>>>>>>>>>> - Detect user presence to manage backlight setting or
> >>>>>>>>>>>> waking up system
> >>>>>>>>>>>>
> >>>>>>>>>>>> Since the Image Processing Unit(IPU) used on the host
> >>>>>>>>>>>> processor needs to configure the CSI-2 link in normal
> >>>>>>>>>>>> camera usages, the
> >>>>>>>>>>>> CSI-2 link and camera sensor can only be used in
> >>>>>>>>>>>> mutually-exclusive ways by host IPU and IVSC. By default
> >>>>>>>>>>>> the IVSC owns the CSI-2 link and camera sensor. The IPU
> >>>>>>>>>>>> driver can take ownership of the CSI-2 link and camera
> >>>>>>>>>>>> sensor using interfaces provided
> >>>>>>>> by this IVSC driver.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Switching ownership requires an interface with two
> >>>>>>>>>>>> different hardware modules inside IVSC. The software
> >>>>>>>>>>>> interface to these modules is via Intel MEI (The Intel
> Management Engine) commands.
> >>>>>>>>>>>> These two hardware modules have two different MEI UUIDs to
> >>>>>>>>>>>> enumerate. These hardware
> >>>>>>>>>> modules are:
> >>>>>>>>>>>> - ACE (Algorithm Context Engine): This module is for
> >>>>>>>>>>>> algorithm computing when IVSC owns camera sensor. Also ACE
> >>>>>>>>>>>> module controls camera sensor's ownership. This hardware
> >>>>>>>>>>>> module is used to set ownership
> >>>>>>>>>> of camera sensor.
> >>>>>>>>>>>> - CSI (Camera Serial Interface): This module is used to
> >>>>>>>>>>>> route camera sensor data either to IVSC or to host for IPU
> >>>>>>>>>>>> driver and
> >>>>>>>> application.
> >>>>>>>>>>>>
> >>>>>>>>>>>> IVSC also provides a privacy mode. When privacy mode is
> >>>>>>>>>>>> turned on, camera sensor can't be used. This means that
> >>>>>>>>>>>> both ACE and host IPU can't get image data. And when this
> >>>>>>>>>>>> mode is turned on, host IPU driver is informed via a
> >>>>>>>>>>>> registered callback, so that user can be
> >>>>>>>> notified.
> >>>>>>>>>>>>
> >>>>>>>>>>>> In summary, to acquire ownership of camera by IPU driver,
> >>>>>>>>>>>> first ACE module needs to be informed of ownership and then
> >>>>>>>>>>>> to setup MIPI CSI-2 link for the camera sensor and IPU.
> >>>>>>>>>>>
> >>>>>>>>>>> I thought this for a while and did some research, and I can
> >>>>>>>>>>> suggest the
> >>>>>>>>>>> following:
> >>>>>>>>>>>
> >>>>>>>>>>> - The IVSC sub-device implements a control for privacy
> >>>>>> (V4L2_CID_PRIVACY
> >>>>>>>>>>> is a good fit).
> >>>>>>>>>>>
> >>>>>>>>>>> - Camera sensor access needs to be requested from IVSC
> >>>>>>>>>>> before accessing
> >>>>>>>> the
> >>>>>>>>>>> sensor via I²C. The IVSC ownership control needs to be in the
> right
> >>>>>>>>>>> setting for this to work, and device links can be used for
> >>>>>>>>>>> that
> >> purpose
> >>>>>>>>>>> (see device_link_add()). With DL_FLAG_PM_RUNTIME and
> >>>>>>>>>> DL_FLAG_RPM_ACTIVE,
> >>>>>>>>>>> the supplier devices will be PM runtime resumed before the
> >> consumer
> >>>>>>>>>>> (camera sensor). As these devices are purely virtual on
> >>>>>>>>>>> host side and
> >>>> has
> >>>>>>>>>>> no power state as such, you can use runtime PM callbacks
> >>>>>>>>>>> to transfer
> >>>>>> the
> >>>>>>>>>>> ownership.
> >>>>>>>>>>
> >>>>>>>>>> Interesting proposal to use device-links + runtime-pm for
> >>>>>>>>>> this instead of modelling this as an i2c-mux. FWIW I'm fine
> >>>>>>>>>> with going this route instead of using an i2c-mux approach.
> >>>>>>>>>>
> >>>>>>>>>> I have been thinking about the i2c-mux approach a bit and the
> >>>>>>>>>> problem is that we are not really muxing but want to turn
> >>>>>>>>>> on/off control and AFAIK the i2c-mux framework simply leaves
> >>>>>>>>>> the mux muxed to the last used i2c-chain, so control will
> >>>>>>>>>> never be released when the i2c
> >>>>>>>> transfers are done.
> >>>>>>>>>>
> >>>>>>>>>> And if were to somehow modify things (or maybe there already
> >>>>>>>>>> is some release
> >>>>>>>>>> callback) then the downside becomes that the i2c-mux core
> >>>>>>>>>> code operates at the i2c transfer level. So each i2c
> >>>>>>>>>> read/write would then enable +
> >>>>>>>> disavle control.
> >>>>>>>>>>
> >>>>>>>>>> Modelling this using something like runtime pm as such is a
> >>>>>>>>>> much better fit because then we request control once on probe
> >>>>>>>>>> / stream-on and release it once we are fully done, rather
> >>>>>>>>>> then requesting + releasing control once per i2c- transfer.
> >>>>>>>>>
> >>>>>>>>> Seems runtime pm can't fix the problem of initial i2c transfer
> >>>>>>>>> during sensor driver probe, probably we have to switch to
> >>>>>>>>> i2c-mux modeling
> >>>>>> way.
> >>>>>>>>
> >>>>>>>> What do you mean? The supplier devices are resumed before the
> >>>>>>>> driver's probe is called.
> >>>>>>>
> >>>>>>> But we setup the link with device_link_add during IVSC driver's
> >>>>>>> probe, we can't guarantee driver probe's sequence.
> >>>>>>
> >>>>>> Then maybe we need to do the device_link_add somewhere else.
> >>>>>
> >>>>> sensor's parent is the LJCA I2C device whose driver is being
> >>>>> upstream https://www.spinics.net/lists/kernel/msg4702552.htmland
> >>>>> and sensor's power is controlled by IVSC instead of INT3472 if IVSC
> enabled.
> >>>>
> >>>> I believe that the INT3472 code is still involved at least on a
> >>>> Dell Latitude 9420 the INT3472 code still needs to set the
> >>>> clock-enable and the privacy-LED GPIOs otherwise the main camera won't
> work.
> >>>>
> >>>> So I'm not sure what you mean with "sensor's power is controlled by
> >>>> IVSC instead of INT3472" ?
> >>>
> >>> Could you please share your kernel log and DSDT? Thanks
> >>>
> >>> BR,
> >>> Wentong
> >>>>
> >>>>
> >>>>> struct device_link *device_link_add(struct device *consumer,
> >>>>> struct device *supplier, u32
> >>>>> flags)
> >>>>>
> >>>>> So probably we have to add above device_link_add in LJCA I2C's
> >>>>> driver, and we can find the consumer(camera sensor) with ACPI API,
> >>>>> but the supplier, mei_ace, is mei client device under mei
> >>>>> framework and it's dynamically allocated device instead of ACPI
> >>>>> device, probably I can find its parent with some ACPI lookup from
> >>>>> this LJCA I2C device, but unfortunately mei framework doesn't
> >>>>> export the API to find mei client device with its parent bus device(struct
> mei_device).
> >>>>>
> >>>>> I'm not sure if modeling this mei_ace as LJCA I2C's runtime power
> >>>>> control is acceptable, if yes, probably this mei_ace driver have
> >>>>> to go with LJCA I2C device driver.
> >>>>
> >>>> Looking at the ACPI table the sensor ACPI device has 2 _DEP-s
> >>>> listed the I2C controller and the INT3472 device. Since we are
> >>>> already doing similar setup in the INT3472 device that seems like a
> >>>> good place to add the device_link()-s (it can return -EPROBE_DEFER
> >>>> to wait for the mei_ace
> >> to show up).
> >>>>
> >>>> But when the INT3472 code runs, the consumer device does not exist
> >>>> yet and AFAICT the same is true when the LCJA i2c-controller driver
> >>>> is getting
> >> registered.
> >>>> The consumer only exists when the i2c_client is instantiated and at
> >>>> that point the sensor drivers probe() method can run immediately
> >>>> and we are too late to add the device_link.
> >>>>
> >>>> As a hobby project I have been working on atomisp2 support and I
> >>>> have a similar issue there. There is no INT3472 device there, but
> >>>> there is a _DSM method which needs to be used to figure out which
> >>>> ACPI GPIO resource is reset / powerdown and if the GPIOs are active-low
> or active high.
> >>>>
> >>>> I have written a little helper function to call the _DSM and to
> >>>> then turn this into lookups and call devm_acpi_dev_add_driver_gpios().
> >>>>
> >>>> Since on atomisp2 we cannot use the INT3472 driver to delay the
> >>>> sensor-driver probe and have the INT3472 driver setup the GPIO
> >>>> lookup, at least for the sensor drivers used with
> >>>> atomisp2 there is going to be a need to add a single line to probe() like this:
> >>>>
> >>>> v4l2_get_acpi_sensor_info(&i2c_client->dev, NULL);
> >>>>
> >>>> To me it sounds like we need to do something similar here and
> >>>> extend the helper function which I have written (but not yet submitted
> upstream) :
> >>>>
> >>>> https://github.com/jwrdegoede/linux-
> >>>> sunxi/commit/e2287979db43d46fa7d354c1bde92eb6219b613d
> >>>>
> >>>> To also setup the device-links needed for the runtime-pm solution
> >>>> to getting the i2c passed through to the sensor.
> >>>>
> >>>> Ideally v4l2_get_acpi_sensor_info() should return void (easier to
> >>>> use in the sensor drivers) but I think it should return an int, so
> >>>> that it can e.g. return - EPROBE_DEFER to wait for the mei_ace.
> >>>>
> >>>> Regards,
> >>>>
> >>>> Hans
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>> The mainline kernel delays probing of camera sensors on Intel
> >>>>>> platforms until the INT3472 driver has probed the INT3472 device
> >>>>>> on which the sensors have an ACPI _DEP.
> >>>>>>
> >>>>>> This is already used to make sure that clock lookups and
> >>>>>> regulator info is in place before the sensor's probe() function runs.
> >>>>>>
> >>>>>> So that when the driver does clk_get() it succeeds and so that
> >>>>>> regulator_get() does not end up returning a dummy regulator.
> >>>>>>
> >>>>>> So I think the code adding the device_link-s for the IVSC should
> >>>>>> be added
> >>>>>> to: drivers/platform/x86/intel/int3472/discrete.c and then the
> >>>>>> runtime-resume will happen before the sensor's probe() function runs.
> >>>>>>
> >>>>>> Likewise drivers/platform/x86/intel/int3472/discrete.c should
> >>>>>> also ensure that the ivsc driver's probe() has run before it
> >>>>>> calls
> >>>> acpi_dev_clear_dependencies().
> >>>>>>
> >>>>>> The acpi_dev_clear_dependencies() call in discrete.c tells the
> >>>>>> ACPI subsystem to go ahead and create the i2c-clients for the
> >>>>>> sensors and allow the sensor drivers to get loaded and probe the sensor.
> >>>>>>
> >>>>>> Regards,
> >>>>>>
> >>>>>> Hans
> >>>>>
> >>>
Hi,
On 3/16/23 09:11, Wu, Wentong wrote:
>
>
>> -----Original Message-----
>> From: Hans de Goede <[email protected]>
>> Sent: Thursday, March 9, 2023 11:24 PM
>>
>> <re-added the previous Cc list, which I dropped because of the large
>> attachments>
>>
>> Hi Wentong,
>>
>> On 3/9/23 15:29, Wu, Wentong wrote:
>>> Hi Hans,
>>>
>>> Thanks
>>>
>>> And AFAICT, there is no IVSC device on your Dell Latitude 9420 where the
>> platform is based on TGL instead of ADL, and I have never heard IVSC runs on
>> TGL, if no IVSC, INT3472 will control sensor's power.
>>> And I will double confirm with people who know dell product well tomorrow.
>>
>> Ah, I was under the impression that there was an IVSC there because:
>>
>> 1. The sensor driver for the used sensor (tries to) poke the IVSC 2. Things did not
>> work without building the IVSC drivers, but that might
>> be due to a dependency on the LCJA GPIO expander instead
>>
>> But you might very well be right, that would also explain the "intel vsc not ready"
>> messages in dmesg.
>>
>> If with the IVSC case the IVSC controls the power to the sensor too, then
>> another option might be to model the I2C-switch + the power-control as a
>> powerdown GPIO for the sensor, which most sensor drivers already try to use.
>> The advantage of doing this would be that GPIO lookups can reference the GPIO
>> provider + consumer by device-name so then we don't need to have both
>> devices instantiated at the time of
>> adding the GPIO lookup. And in that case we could e.g. add the lookup
>> before registering the I2C controller.
>
> Thanks,
>
> So the drivers of sensors connected to IVSC have to add power up/down code.
If we go the model it as a powerdown GPIO route, yes. Note most sensor drivers already have support for this since this is used in the INT3472 case already.
Regards,
Hans
>
> BR,
> Wentong
>>
>> Sakari, what do you think of instead of using runtime-pm + devlinks having the
>> IVSC code export a powerdown GPIO to the sensor ?
>>
>> This also decouples the ivsc powerstate from the sensor power-state which
>> might be useful if we ever want to use some of the more advanced ivsc features,
>> where AFAICT the ivsc fully controls the sensor.
>>
>> Regards,
>>
>> Hans
>>
>>
>>
>>
>>>> -----Original Message-----
>>>> From: Hans de Goede <[email protected]>
>>>> Sent: Thursday, March 9, 2023 9:30 PM
>>>> To: Wu, Wentong <[email protected]>
>>>> Subject: Re: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of
>>>> Intel Visual Sensing Controller(IVSC)
>>>>
>>>> Hi Wentong,
>>>>
>>>> Attached are the requested dmesg + acpidump for the Dell Latitude 9420.
>>>>
>>>> Regards,
>>>>
>>>> Hans
>>>>
>>>>
>>>>
>>>>
>>>> On 3/9/23 14:21, Wu, Wentong wrote:
>>>>> Hi
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Hans de Goede <[email protected]>
>>>>>> Sent: Thursday, March 9, 2023 5:28 PM
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> On 3/9/23 02:08, Wu, Wentong wrote:
>>>>>>>
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Hans de Goede <[email protected]>
>>>>>>>> Sent: Tuesday, March 7, 2023 5:10 PM
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> On 3/7/23 09:40, Wu, Wentong wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Sakari Ailus <[email protected]>
>>>>>>>>>> Sent: Tuesday, March 7, 2023 4:30 PM
>>>>>>>>>>
>>>>>>>>>> Hi Wentong,
>>>>>>>>>>
>>>>>>>>>> On Tue, Mar 07, 2023 at 08:17:04AM +0000, Wu, Wentong wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Hans de Goede <[email protected]>
>>>>>>>>>>>> Sent: Wednesday, March 1, 2023 6:42 PM
>>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> On 3/1/23 11:34, Sakari Ailus wrote:
>>>>>>>>>>>>> Hi Wentong,
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
>>>>>>>>>>>>>> Intel Visual Sensing Controller (IVSC), codenamed "Clover
>>>>>>>>>>>>>> Falls", is a companion chip designed to provide secure and
>>>>>>>>>>>>>> low power vision capability to IA platforms. IVSC is
>>>>>>>>>>>>>> available in existing commercial platforms from multiple OEMs.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The primary use case of IVSC is to bring in context awareness.
>>>>>>>>>>>>>> IVSC interfaces directly with the platform main camera
>>>>>>>>>>>>>> sensor via a CSI-2 link and processes the image data with
>>>>>>>>>>>>>> the embedded AI engine. The detected events are sent over
>>>>>>>>>>>>>> I2C to ISH (Intel Sensor Hub) for additional data fusion
>>>>>>>>>>>>>> from multiple
>>>> sensors.
>>>>>>>>>>>>>> The fusion results are used to implement advanced use cases like:
>>>>>>>>>>>>>> - Face detection to unlock screen
>>>>>>>>>>>>>> - Detect user presence to manage backlight setting or
>>>>>>>>>>>>>> waking up system
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Since the Image Processing Unit(IPU) used on the host
>>>>>>>>>>>>>> processor needs to configure the CSI-2 link in normal
>>>>>>>>>>>>>> camera usages, the
>>>>>>>>>>>>>> CSI-2 link and camera sensor can only be used in
>>>>>>>>>>>>>> mutually-exclusive ways by host IPU and IVSC. By default
>>>>>>>>>>>>>> the IVSC owns the CSI-2 link and camera sensor. The IPU
>>>>>>>>>>>>>> driver can take ownership of the CSI-2 link and camera
>>>>>>>>>>>>>> sensor using interfaces provided
>>>>>>>>>> by this IVSC driver.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Switching ownership requires an interface with two
>>>>>>>>>>>>>> different hardware modules inside IVSC. The software
>>>>>>>>>>>>>> interface to these modules is via Intel MEI (The Intel
>> Management Engine) commands.
>>>>>>>>>>>>>> These two hardware modules have two different MEI UUIDs to
>>>>>>>>>>>>>> enumerate. These hardware
>>>>>>>>>>>> modules are:
>>>>>>>>>>>>>> - ACE (Algorithm Context Engine): This module is for
>>>>>>>>>>>>>> algorithm computing when IVSC owns camera sensor. Also ACE
>>>>>>>>>>>>>> module controls camera sensor's ownership. This hardware
>>>>>>>>>>>>>> module is used to set ownership
>>>>>>>>>>>> of camera sensor.
>>>>>>>>>>>>>> - CSI (Camera Serial Interface): This module is used to
>>>>>>>>>>>>>> route camera sensor data either to IVSC or to host for IPU
>>>>>>>>>>>>>> driver and
>>>>>>>>>> application.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> IVSC also provides a privacy mode. When privacy mode is
>>>>>>>>>>>>>> turned on, camera sensor can't be used. This means that
>>>>>>>>>>>>>> both ACE and host IPU can't get image data. And when this
>>>>>>>>>>>>>> mode is turned on, host IPU driver is informed via a
>>>>>>>>>>>>>> registered callback, so that user can be
>>>>>>>>>> notified.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In summary, to acquire ownership of camera by IPU driver,
>>>>>>>>>>>>>> first ACE module needs to be informed of ownership and then
>>>>>>>>>>>>>> to setup MIPI CSI-2 link for the camera sensor and IPU.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I thought this for a while and did some research, and I can
>>>>>>>>>>>>> suggest the
>>>>>>>>>>>>> following:
>>>>>>>>>>>>>
>>>>>>>>>>>>> - The IVSC sub-device implements a control for privacy
>>>>>>>> (V4L2_CID_PRIVACY
>>>>>>>>>>>>> is a good fit).
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Camera sensor access needs to be requested from IVSC
>>>>>>>>>>>>> before accessing
>>>>>>>>>> the
>>>>>>>>>>>>> sensor via I²C. The IVSC ownership control needs to be in the
>> right
>>>>>>>>>>>>> setting for this to work, and device links can be used for
>>>>>>>>>>>>> that
>>>> purpose
>>>>>>>>>>>>> (see device_link_add()). With DL_FLAG_PM_RUNTIME and
>>>>>>>>>>>> DL_FLAG_RPM_ACTIVE,
>>>>>>>>>>>>> the supplier devices will be PM runtime resumed before the
>>>> consumer
>>>>>>>>>>>>> (camera sensor). As these devices are purely virtual on
>>>>>>>>>>>>> host side and
>>>>>> has
>>>>>>>>>>>>> no power state as such, you can use runtime PM callbacks
>>>>>>>>>>>>> to transfer
>>>>>>>> the
>>>>>>>>>>>>> ownership.
>>>>>>>>>>>>
>>>>>>>>>>>> Interesting proposal to use device-links + runtime-pm for
>>>>>>>>>>>> this instead of modelling this as an i2c-mux. FWIW I'm fine
>>>>>>>>>>>> with going this route instead of using an i2c-mux approach.
>>>>>>>>>>>>
>>>>>>>>>>>> I have been thinking about the i2c-mux approach a bit and the
>>>>>>>>>>>> problem is that we are not really muxing but want to turn
>>>>>>>>>>>> on/off control and AFAIK the i2c-mux framework simply leaves
>>>>>>>>>>>> the mux muxed to the last used i2c-chain, so control will
>>>>>>>>>>>> never be released when the i2c
>>>>>>>>>> transfers are done.
>>>>>>>>>>>>
>>>>>>>>>>>> And if were to somehow modify things (or maybe there already
>>>>>>>>>>>> is some release
>>>>>>>>>>>> callback) then the downside becomes that the i2c-mux core
>>>>>>>>>>>> code operates at the i2c transfer level. So each i2c
>>>>>>>>>>>> read/write would then enable +
>>>>>>>>>> disavle control.
>>>>>>>>>>>>
>>>>>>>>>>>> Modelling this using something like runtime pm as such is a
>>>>>>>>>>>> much better fit because then we request control once on probe
>>>>>>>>>>>> / stream-on and release it once we are fully done, rather
>>>>>>>>>>>> then requesting + releasing control once per i2c- transfer.
>>>>>>>>>>>
>>>>>>>>>>> Seems runtime pm can't fix the problem of initial i2c transfer
>>>>>>>>>>> during sensor driver probe, probably we have to switch to
>>>>>>>>>>> i2c-mux modeling
>>>>>>>> way.
>>>>>>>>>>
>>>>>>>>>> What do you mean? The supplier devices are resumed before the
>>>>>>>>>> driver's probe is called.
>>>>>>>>>
>>>>>>>>> But we setup the link with device_link_add during IVSC driver's
>>>>>>>>> probe, we can't guarantee driver probe's sequence.
>>>>>>>>
>>>>>>>> Then maybe we need to do the device_link_add somewhere else.
>>>>>>>
>>>>>>> sensor's parent is the LJCA I2C device whose driver is being
>>>>>>> upstream https://www.spinics.net/lists/kernel/msg4702552.htmland
>>>>>>> and sensor's power is controlled by IVSC instead of INT3472 if IVSC
>> enabled.
>>>>>>
>>>>>> I believe that the INT3472 code is still involved at least on a
>>>>>> Dell Latitude 9420 the INT3472 code still needs to set the
>>>>>> clock-enable and the privacy-LED GPIOs otherwise the main camera won't
>> work.
>>>>>>
>>>>>> So I'm not sure what you mean with "sensor's power is controlled by
>>>>>> IVSC instead of INT3472" ?
>>>>>
>>>>> Could you please share your kernel log and DSDT? Thanks
>>>>>
>>>>> BR,
>>>>> Wentong
>>>>>>
>>>>>>
>>>>>>> struct device_link *device_link_add(struct device *consumer,
>>>>>>> struct device *supplier, u32
>>>>>>> flags)
>>>>>>>
>>>>>>> So probably we have to add above device_link_add in LJCA I2C's
>>>>>>> driver, and we can find the consumer(camera sensor) with ACPI API,
>>>>>>> but the supplier, mei_ace, is mei client device under mei
>>>>>>> framework and it's dynamically allocated device instead of ACPI
>>>>>>> device, probably I can find its parent with some ACPI lookup from
>>>>>>> this LJCA I2C device, but unfortunately mei framework doesn't
>>>>>>> export the API to find mei client device with its parent bus device(struct
>> mei_device).
>>>>>>>
>>>>>>> I'm not sure if modeling this mei_ace as LJCA I2C's runtime power
>>>>>>> control is acceptable, if yes, probably this mei_ace driver have
>>>>>>> to go with LJCA I2C device driver.
>>>>>>
>>>>>> Looking at the ACPI table the sensor ACPI device has 2 _DEP-s
>>>>>> listed the I2C controller and the INT3472 device. Since we are
>>>>>> already doing similar setup in the INT3472 device that seems like a
>>>>>> good place to add the device_link()-s (it can return -EPROBE_DEFER
>>>>>> to wait for the mei_ace
>>>> to show up).
>>>>>>
>>>>>> But when the INT3472 code runs, the consumer device does not exist
>>>>>> yet and AFAICT the same is true when the LCJA i2c-controller driver
>>>>>> is getting
>>>> registered.
>>>>>> The consumer only exists when the i2c_client is instantiated and at
>>>>>> that point the sensor drivers probe() method can run immediately
>>>>>> and we are too late to add the device_link.
>>>>>>
>>>>>> As a hobby project I have been working on atomisp2 support and I
>>>>>> have a similar issue there. There is no INT3472 device there, but
>>>>>> there is a _DSM method which needs to be used to figure out which
>>>>>> ACPI GPIO resource is reset / powerdown and if the GPIOs are active-low
>> or active high.
>>>>>>
>>>>>> I have written a little helper function to call the _DSM and to
>>>>>> then turn this into lookups and call devm_acpi_dev_add_driver_gpios().
>>>>>>
>>>>>> Since on atomisp2 we cannot use the INT3472 driver to delay the
>>>>>> sensor-driver probe and have the INT3472 driver setup the GPIO
>>>>>> lookup, at least for the sensor drivers used with
>>>>>> atomisp2 there is going to be a need to add a single line to probe() like this:
>>>>>>
>>>>>> v4l2_get_acpi_sensor_info(&i2c_client->dev, NULL);
>>>>>>
>>>>>> To me it sounds like we need to do something similar here and
>>>>>> extend the helper function which I have written (but not yet submitted
>> upstream) :
>>>>>>
>>>>>> https://github.com/jwrdegoede/linux-
>>>>>> sunxi/commit/e2287979db43d46fa7d354c1bde92eb6219b613d
>>>>>>
>>>>>> To also setup the device-links needed for the runtime-pm solution
>>>>>> to getting the i2c passed through to the sensor.
>>>>>>
>>>>>> Ideally v4l2_get_acpi_sensor_info() should return void (easier to
>>>>>> use in the sensor drivers) but I think it should return an int, so
>>>>>> that it can e.g. return - EPROBE_DEFER to wait for the mei_ace.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Hans
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>> The mainline kernel delays probing of camera sensors on Intel
>>>>>>>> platforms until the INT3472 driver has probed the INT3472 device
>>>>>>>> on which the sensors have an ACPI _DEP.
>>>>>>>>
>>>>>>>> This is already used to make sure that clock lookups and
>>>>>>>> regulator info is in place before the sensor's probe() function runs.
>>>>>>>>
>>>>>>>> So that when the driver does clk_get() it succeeds and so that
>>>>>>>> regulator_get() does not end up returning a dummy regulator.
>>>>>>>>
>>>>>>>> So I think the code adding the device_link-s for the IVSC should
>>>>>>>> be added
>>>>>>>> to: drivers/platform/x86/intel/int3472/discrete.c and then the
>>>>>>>> runtime-resume will happen before the sensor's probe() function runs.
>>>>>>>>
>>>>>>>> Likewise drivers/platform/x86/intel/int3472/discrete.c should
>>>>>>>> also ensure that the ivsc driver's probe() has run before it
>>>>>>>> calls
>>>>>> acpi_dev_clear_dependencies().
>>>>>>>>
>>>>>>>> The acpi_dev_clear_dependencies() call in discrete.c tells the
>>>>>>>> ACPI subsystem to go ahead and create the i2c-clients for the
>>>>>>>> sensors and allow the sensor drivers to get loaded and probe the sensor.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Hans
>>>>>>>
>>>>>
>
Hi,
On 3/16/23 03:58, Wu, Wentong wrote:
>
>
>> -----Original Message-----
>> From: Hans de Goede <[email protected]>
>> Sent: Thursday, March 9, 2023 11:24 PM
>>
>> <re-added the previous Cc list, which I dropped because of the large
>> attachments>
>>
>> Hi Wentong,
>>
>> On 3/9/23 15:29, Wu, Wentong wrote:
>>> Hi Hans,
>>>
>>> Thanks
>>>
>>> And AFAICT, there is no IVSC device on your Dell Latitude 9420 where the
>> platform is based on TGL instead of ADL, and I have never heard IVSC runs on
>> TGL, if no IVSC, INT3472 will control sensor's power.
>>> And I will double confirm with people who know dell product well tomorrow.
>>
>> Ah, I was under the impression that there was an IVSC there because:
>>
>> 1. The sensor driver for the used sensor (tries to) poke the IVSC 2. Things did not
>> work without building the IVSC drivers, but that might
>> be due to a dependency on the LCJA GPIO expander instead
>
> Below is your dmesg log, the required SPI controller for IVSC isn't here.
>
> [ 35.538114] ljca 2-6:1.0: acked sem wait timed out ret:0 timeout:20 ack:0
> [ 35.538129] ljca 2-6:1.0: MNG_ENUM_SPI failed ret:-110 len:7 num:0
> [ 35.538621] ljca 2-6:1.0: LJCA USB device init success
> [ 35.538776] usbcore: registered new interface driver ljca
>
> Also I checked your SSDT, there is no IVSC device and the sensor device depends on
> INT3472 instead of IVSC device as on my setup.
Ack.
>> But you might very well be right, that would also explain the "intel vsc not ready"
>> messages in dmesg.
>>
>> If with the IVSC case the IVSC controls the power to the sensor too, then
>> another option might be to model the I2C-switch + the power-control as a
>> powerdown GPIO for the sensor, which most sensor drivers already try to use.
>> The advantage of doing this would be that GPIO lookups can reference the GPIO
>> provider + consumer by device-name so then we don't need to have both
>> devices instantiated at the time of
>> adding the GPIO lookup. And in that case we could e.g. add the lookup
>> before registering the I2C controller.
>
> Can we add IVSC device to acpi_honor_dep_ids, so that when everything is done during
> mei_ace probe, acpi_dev_clear_dependencies can make sensor start probe?
Does the sensor ACPI device node have an ACPI _DEP on the IVSC device ?
If yes, then yes we can add the IVSC device to acpi_honor_dep_id and make mei_ace probe call acpi_dev_clear_dependencies().
Regards,
Hans
>> Sakari, what do you think of instead of using runtime-pm + devlinks having the
>> IVSC code export a powerdown GPIO to the sensor ?
>>
>> This also decouples the ivsc powerstate from the sensor power-state which
>> might be useful if we ever want to use some of the more advanced ivsc features,
>> where AFAICT the ivsc fully controls the sensor.
>>
>> Regards,
>>
>> Hans
>>
>>
>>
>>
>>>> -----Original Message-----
>>>> From: Hans de Goede <[email protected]>
>>>> Sent: Thursday, March 9, 2023 9:30 PM
>>>> To: Wu, Wentong <[email protected]>
>>>> Subject: Re: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of
>>>> Intel Visual Sensing Controller(IVSC)
>>>>
>>>> Hi Wentong,
>>>>
>>>> Attached are the requested dmesg + acpidump for the Dell Latitude 9420.
>>>>
>>>> Regards,
>>>>
>>>> Hans
>>>>
>>>>
>>>>
>>>>
>>>> On 3/9/23 14:21, Wu, Wentong wrote:
>>>>> Hi
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Hans de Goede <[email protected]>
>>>>>> Sent: Thursday, March 9, 2023 5:28 PM
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> On 3/9/23 02:08, Wu, Wentong wrote:
>>>>>>>
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Hans de Goede <[email protected]>
>>>>>>>> Sent: Tuesday, March 7, 2023 5:10 PM
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> On 3/7/23 09:40, Wu, Wentong wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Sakari Ailus <[email protected]>
>>>>>>>>>> Sent: Tuesday, March 7, 2023 4:30 PM
>>>>>>>>>>
>>>>>>>>>> Hi Wentong,
>>>>>>>>>>
>>>>>>>>>> On Tue, Mar 07, 2023 at 08:17:04AM +0000, Wu, Wentong wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Hans de Goede <[email protected]>
>>>>>>>>>>>> Sent: Wednesday, March 1, 2023 6:42 PM
>>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> On 3/1/23 11:34, Sakari Ailus wrote:
>>>>>>>>>>>>> Hi Wentong,
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
>>>>>>>>>>>>>> Intel Visual Sensing Controller (IVSC), codenamed "Clover
>>>>>>>>>>>>>> Falls", is a companion chip designed to provide secure and
>>>>>>>>>>>>>> low power vision capability to IA platforms. IVSC is
>>>>>>>>>>>>>> available in existing commercial platforms from multiple OEMs.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The primary use case of IVSC is to bring in context awareness.
>>>>>>>>>>>>>> IVSC interfaces directly with the platform main camera
>>>>>>>>>>>>>> sensor via a CSI-2 link and processes the image data with
>>>>>>>>>>>>>> the embedded AI engine. The detected events are sent over
>>>>>>>>>>>>>> I2C to ISH (Intel Sensor Hub) for additional data fusion
>>>>>>>>>>>>>> from multiple
>>>> sensors.
>>>>>>>>>>>>>> The fusion results are used to implement advanced use cases like:
>>>>>>>>>>>>>> - Face detection to unlock screen
>>>>>>>>>>>>>> - Detect user presence to manage backlight setting or
>>>>>>>>>>>>>> waking up system
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Since the Image Processing Unit(IPU) used on the host
>>>>>>>>>>>>>> processor needs to configure the CSI-2 link in normal
>>>>>>>>>>>>>> camera usages, the
>>>>>>>>>>>>>> CSI-2 link and camera sensor can only be used in
>>>>>>>>>>>>>> mutually-exclusive ways by host IPU and IVSC. By default
>>>>>>>>>>>>>> the IVSC owns the CSI-2 link and camera sensor. The IPU
>>>>>>>>>>>>>> driver can take ownership of the CSI-2 link and camera
>>>>>>>>>>>>>> sensor using interfaces provided
>>>>>>>>>> by this IVSC driver.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Switching ownership requires an interface with two
>>>>>>>>>>>>>> different hardware modules inside IVSC. The software
>>>>>>>>>>>>>> interface to these modules is via Intel MEI (The Intel
>> Management Engine) commands.
>>>>>>>>>>>>>> These two hardware modules have two different MEI UUIDs to
>>>>>>>>>>>>>> enumerate. These hardware
>>>>>>>>>>>> modules are:
>>>>>>>>>>>>>> - ACE (Algorithm Context Engine): This module is for
>>>>>>>>>>>>>> algorithm computing when IVSC owns camera sensor. Also ACE
>>>>>>>>>>>>>> module controls camera sensor's ownership. This hardware
>>>>>>>>>>>>>> module is used to set ownership
>>>>>>>>>>>> of camera sensor.
>>>>>>>>>>>>>> - CSI (Camera Serial Interface): This module is used to
>>>>>>>>>>>>>> route camera sensor data either to IVSC or to host for IPU
>>>>>>>>>>>>>> driver and
>>>>>>>>>> application.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> IVSC also provides a privacy mode. When privacy mode is
>>>>>>>>>>>>>> turned on, camera sensor can't be used. This means that
>>>>>>>>>>>>>> both ACE and host IPU can't get image data. And when this
>>>>>>>>>>>>>> mode is turned on, host IPU driver is informed via a
>>>>>>>>>>>>>> registered callback, so that user can be
>>>>>>>>>> notified.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In summary, to acquire ownership of camera by IPU driver,
>>>>>>>>>>>>>> first ACE module needs to be informed of ownership and then
>>>>>>>>>>>>>> to setup MIPI CSI-2 link for the camera sensor and IPU.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I thought this for a while and did some research, and I can
>>>>>>>>>>>>> suggest the
>>>>>>>>>>>>> following:
>>>>>>>>>>>>>
>>>>>>>>>>>>> - The IVSC sub-device implements a control for privacy
>>>>>>>> (V4L2_CID_PRIVACY
>>>>>>>>>>>>> is a good fit).
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Camera sensor access needs to be requested from IVSC
>>>>>>>>>>>>> before accessing
>>>>>>>>>> the
>>>>>>>>>>>>> sensor via I²C. The IVSC ownership control needs to be in the
>> right
>>>>>>>>>>>>> setting for this to work, and device links can be used for
>>>>>>>>>>>>> that
>>>> purpose
>>>>>>>>>>>>> (see device_link_add()). With DL_FLAG_PM_RUNTIME and
>>>>>>>>>>>> DL_FLAG_RPM_ACTIVE,
>>>>>>>>>>>>> the supplier devices will be PM runtime resumed before the
>>>> consumer
>>>>>>>>>>>>> (camera sensor). As these devices are purely virtual on
>>>>>>>>>>>>> host side and
>>>>>> has
>>>>>>>>>>>>> no power state as such, you can use runtime PM callbacks
>>>>>>>>>>>>> to transfer
>>>>>>>> the
>>>>>>>>>>>>> ownership.
>>>>>>>>>>>>
>>>>>>>>>>>> Interesting proposal to use device-links + runtime-pm for
>>>>>>>>>>>> this instead of modelling this as an i2c-mux. FWIW I'm fine
>>>>>>>>>>>> with going this route instead of using an i2c-mux approach.
>>>>>>>>>>>>
>>>>>>>>>>>> I have been thinking about the i2c-mux approach a bit and the
>>>>>>>>>>>> problem is that we are not really muxing but want to turn
>>>>>>>>>>>> on/off control and AFAIK the i2c-mux framework simply leaves
>>>>>>>>>>>> the mux muxed to the last used i2c-chain, so control will
>>>>>>>>>>>> never be released when the i2c
>>>>>>>>>> transfers are done.
>>>>>>>>>>>>
>>>>>>>>>>>> And if were to somehow modify things (or maybe there already
>>>>>>>>>>>> is some release
>>>>>>>>>>>> callback) then the downside becomes that the i2c-mux core
>>>>>>>>>>>> code operates at the i2c transfer level. So each i2c
>>>>>>>>>>>> read/write would then enable +
>>>>>>>>>> disavle control.
>>>>>>>>>>>>
>>>>>>>>>>>> Modelling this using something like runtime pm as such is a
>>>>>>>>>>>> much better fit because then we request control once on probe
>>>>>>>>>>>> / stream-on and release it once we are fully done, rather
>>>>>>>>>>>> then requesting + releasing control once per i2c- transfer.
>>>>>>>>>>>
>>>>>>>>>>> Seems runtime pm can't fix the problem of initial i2c transfer
>>>>>>>>>>> during sensor driver probe, probably we have to switch to
>>>>>>>>>>> i2c-mux modeling
>>>>>>>> way.
>>>>>>>>>>
>>>>>>>>>> What do you mean? The supplier devices are resumed before the
>>>>>>>>>> driver's probe is called.
>>>>>>>>>
>>>>>>>>> But we setup the link with device_link_add during IVSC driver's
>>>>>>>>> probe, we can't guarantee driver probe's sequence.
>>>>>>>>
>>>>>>>> Then maybe we need to do the device_link_add somewhere else.
>>>>>>>
>>>>>>> sensor's parent is the LJCA I2C device whose driver is being
>>>>>>> upstream https://www.spinics.net/lists/kernel/msg4702552.htmland
>>>>>>> and sensor's power is controlled by IVSC instead of INT3472 if IVSC
>> enabled.
>>>>>>
>>>>>> I believe that the INT3472 code is still involved at least on a
>>>>>> Dell Latitude 9420 the INT3472 code still needs to set the
>>>>>> clock-enable and the privacy-LED GPIOs otherwise the main camera won't
>> work.
>>>>>>
>>>>>> So I'm not sure what you mean with "sensor's power is controlled by
>>>>>> IVSC instead of INT3472" ?
>>>>>
>>>>> Could you please share your kernel log and DSDT? Thanks
>>>>>
>>>>> BR,
>>>>> Wentong
>>>>>>
>>>>>>
>>>>>>> struct device_link *device_link_add(struct device *consumer,
>>>>>>> struct device *supplier, u32
>>>>>>> flags)
>>>>>>>
>>>>>>> So probably we have to add above device_link_add in LJCA I2C's
>>>>>>> driver, and we can find the consumer(camera sensor) with ACPI API,
>>>>>>> but the supplier, mei_ace, is mei client device under mei
>>>>>>> framework and it's dynamically allocated device instead of ACPI
>>>>>>> device, probably I can find its parent with some ACPI lookup from
>>>>>>> this LJCA I2C device, but unfortunately mei framework doesn't
>>>>>>> export the API to find mei client device with its parent bus device(struct
>> mei_device).
>>>>>>>
>>>>>>> I'm not sure if modeling this mei_ace as LJCA I2C's runtime power
>>>>>>> control is acceptable, if yes, probably this mei_ace driver have
>>>>>>> to go with LJCA I2C device driver.
>>>>>>
>>>>>> Looking at the ACPI table the sensor ACPI device has 2 _DEP-s
>>>>>> listed the I2C controller and the INT3472 device. Since we are
>>>>>> already doing similar setup in the INT3472 device that seems like a
>>>>>> good place to add the device_link()-s (it can return -EPROBE_DEFER
>>>>>> to wait for the mei_ace
>>>> to show up).
>>>>>>
>>>>>> But when the INT3472 code runs, the consumer device does not exist
>>>>>> yet and AFAICT the same is true when the LCJA i2c-controller driver
>>>>>> is getting
>>>> registered.
>>>>>> The consumer only exists when the i2c_client is instantiated and at
>>>>>> that point the sensor drivers probe() method can run immediately
>>>>>> and we are too late to add the device_link.
>>>>>>
>>>>>> As a hobby project I have been working on atomisp2 support and I
>>>>>> have a similar issue there. There is no INT3472 device there, but
>>>>>> there is a _DSM method which needs to be used to figure out which
>>>>>> ACPI GPIO resource is reset / powerdown and if the GPIOs are active-low
>> or active high.
>>>>>>
>>>>>> I have written a little helper function to call the _DSM and to
>>>>>> then turn this into lookups and call devm_acpi_dev_add_driver_gpios().
>>>>>>
>>>>>> Since on atomisp2 we cannot use the INT3472 driver to delay the
>>>>>> sensor-driver probe and have the INT3472 driver setup the GPIO
>>>>>> lookup, at least for the sensor drivers used with
>>>>>> atomisp2 there is going to be a need to add a single line to probe() like this:
>>>>>>
>>>>>> v4l2_get_acpi_sensor_info(&i2c_client->dev, NULL);
>>>>>>
>>>>>> To me it sounds like we need to do something similar here and
>>>>>> extend the helper function which I have written (but not yet submitted
>> upstream) :
>>>>>>
>>>>>> https://github.com/jwrdegoede/linux-
>>>>>> sunxi/commit/e2287979db43d46fa7d354c1bde92eb6219b613d
>>>>>>
>>>>>> To also setup the device-links needed for the runtime-pm solution
>>>>>> to getting the i2c passed through to the sensor.
>>>>>>
>>>>>> Ideally v4l2_get_acpi_sensor_info() should return void (easier to
>>>>>> use in the sensor drivers) but I think it should return an int, so
>>>>>> that it can e.g. return - EPROBE_DEFER to wait for the mei_ace.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Hans
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>> The mainline kernel delays probing of camera sensors on Intel
>>>>>>>> platforms until the INT3472 driver has probed the INT3472 device
>>>>>>>> on which the sensors have an ACPI _DEP.
>>>>>>>>
>>>>>>>> This is already used to make sure that clock lookups and
>>>>>>>> regulator info is in place before the sensor's probe() function runs.
>>>>>>>>
>>>>>>>> So that when the driver does clk_get() it succeeds and so that
>>>>>>>> regulator_get() does not end up returning a dummy regulator.
>>>>>>>>
>>>>>>>> So I think the code adding the device_link-s for the IVSC should
>>>>>>>> be added
>>>>>>>> to: drivers/platform/x86/intel/int3472/discrete.c and then the
>>>>>>>> runtime-resume will happen before the sensor's probe() function runs.
>>>>>>>>
>>>>>>>> Likewise drivers/platform/x86/intel/int3472/discrete.c should
>>>>>>>> also ensure that the ivsc driver's probe() has run before it
>>>>>>>> calls
>>>>>> acpi_dev_clear_dependencies().
>>>>>>>>
>>>>>>>> The acpi_dev_clear_dependencies() call in discrete.c tells the
>>>>>>>> ACPI subsystem to go ahead and create the i2c-clients for the
>>>>>>>> sensors and allow the sensor drivers to get loaded and probe the sensor.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Hans
>>>>>>>
>>>>>
>
> -----Original Message-----
> From: Hans de Goede <[email protected]>
> Sent: Thursday, March 16, 2023 5:04 PM
>
> Hi,
>
> On 3/16/23 03:58, Wu, Wentong wrote:
> >
> >
> >> -----Original Message-----
> >> From: Hans de Goede <[email protected]>
> >> Sent: Thursday, March 9, 2023 11:24 PM
> >>
> >> <re-added the previous Cc list, which I dropped because of the large
> >> attachments>
> >>
> >> Hi Wentong,
> >>
> >> On 3/9/23 15:29, Wu, Wentong wrote:
> >>> Hi Hans,
> >>>
> >>> Thanks
> >>>
> >>> And AFAICT, there is no IVSC device on your Dell Latitude 9420 where
> >>> the
> >> platform is based on TGL instead of ADL, and I have never heard IVSC
> >> runs on TGL, if no IVSC, INT3472 will control sensor's power.
> >>> And I will double confirm with people who know dell product well tomorrow.
> >>
> >> Ah, I was under the impression that there was an IVSC there because:
> >>
> >> 1. The sensor driver for the used sensor (tries to) poke the IVSC 2.
> >> Things did not work without building the IVSC drivers, but that might
> >> be due to a dependency on the LCJA GPIO expander instead
> >
> > Below is your dmesg log, the required SPI controller for IVSC isn't here.
> >
> > [ 35.538114] ljca 2-6:1.0: acked sem wait timed out ret:0 timeout:20 ack:0
> > [ 35.538129] ljca 2-6:1.0: MNG_ENUM_SPI failed ret:-110 len:7 num:0
> > [ 35.538621] ljca 2-6:1.0: LJCA USB device init success
> > [ 35.538776] usbcore: registered new interface driver ljca
> >
> > Also I checked your SSDT, there is no IVSC device and the sensor
> > device depends on
> > INT3472 instead of IVSC device as on my setup.
>
> Ack.
>
> >> But you might very well be right, that would also explain the "intel vsc not
> ready"
> >> messages in dmesg.
> >>
> >> If with the IVSC case the IVSC controls the power to the sensor too,
> >> then another option might be to model the I2C-switch + the
> >> power-control as a powerdown GPIO for the sensor, which most sensor
> drivers already try to use.
> >> The advantage of doing this would be that GPIO lookups can reference
> >> the GPIO provider + consumer by device-name so then we don't need to
> >> have both devices instantiated at the time of
> >> adding the GPIO lookup. And in that case we could e.g. add the lookup
> >> before registering the I2C controller.
> >
> > Can we add IVSC device to acpi_honor_dep_ids, so that when everything
> > is done during mei_ace probe, acpi_dev_clear_dependencies can make sensor
> start probe?
>
> Does the sensor ACPI device node have an ACPI _DEP on the IVSC device ?
Yes,
>
> If yes, then yes we can add the IVSC device to acpi_honor_dep_id and make
> mei_ace probe call acpi_dev_clear_dependencies().
But I prefer the powerdown gpio model, because we have to follow the commands
sequences as below which is required by firmware, runtime pm is hard to achieve this.
+ /* switch camera sensor ownership to host */
+ ret = ace_set_camera_owner(ACE_CAMERA_HOST);
+ if (ret)
+ goto error;
+
+ /* switch CSI-2 link to host */
+ ret = csi_set_link_owner(CSI_LINK_HOST, callback, context);
+ if (ret)
+ goto release_camera;
+
+ /* configure CSI-2 link */
+ ret = csi_set_link_cfg(nr_of_lanes, link_freq);
+ if (ret)
+ goto release_csi;
BR,
Wentong
>
> Regards,
>
> Hans
>
>
>
> >> Sakari, what do you think of instead of using runtime-pm + devlinks
> >> having the IVSC code export a powerdown GPIO to the sensor ?
> >>
> >> This also decouples the ivsc powerstate from the sensor power-state
> >> which might be useful if we ever want to use some of the more
> >> advanced ivsc features, where AFAICT the ivsc fully controls the sensor.
> >>
> >> Regards,
> >>
> >> Hans
> >>
> >>
> >>
> >>
> >>>> -----Original Message-----
> >>>> From: Hans de Goede <[email protected]>
> >>>> Sent: Thursday, March 9, 2023 9:30 PM
> >>>> To: Wu, Wentong <[email protected]>
> >>>> Subject: Re: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of
> >>>> Intel Visual Sensing Controller(IVSC)
> >>>>
> >>>> Hi Wentong,
> >>>>
> >>>> Attached are the requested dmesg + acpidump for the Dell Latitude 9420.
> >>>>
> >>>> Regards,
> >>>>
> >>>> Hans
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On 3/9/23 14:21, Wu, Wentong wrote:
> >>>>> Hi
> >>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Hans de Goede <[email protected]>
> >>>>>> Sent: Thursday, March 9, 2023 5:28 PM
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> On 3/9/23 02:08, Wu, Wentong wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>> -----Original Message-----
> >>>>>>>> From: Hans de Goede <[email protected]>
> >>>>>>>> Sent: Tuesday, March 7, 2023 5:10 PM
> >>>>>>>>
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> On 3/7/23 09:40, Wu, Wentong wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> -----Original Message-----
> >>>>>>>>>> From: Sakari Ailus <[email protected]>
> >>>>>>>>>> Sent: Tuesday, March 7, 2023 4:30 PM
> >>>>>>>>>>
> >>>>>>>>>> Hi Wentong,
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Mar 07, 2023 at 08:17:04AM +0000, Wu, Wentong wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>> From: Hans de Goede <[email protected]>
> >>>>>>>>>>>> Sent: Wednesday, March 1, 2023 6:42 PM
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi,
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 3/1/23 11:34, Sakari Ailus wrote:
> >>>>>>>>>>>>> Hi Wentong,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu
> wrote:
> >>>>>>>>>>>>>> Intel Visual Sensing Controller (IVSC), codenamed "Clover
> >>>>>>>>>>>>>> Falls", is a companion chip designed to provide secure
> >>>>>>>>>>>>>> and low power vision capability to IA platforms. IVSC is
> >>>>>>>>>>>>>> available in existing commercial platforms from multiple OEMs.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> The primary use case of IVSC is to bring in context awareness.
> >>>>>>>>>>>>>> IVSC interfaces directly with the platform main camera
> >>>>>>>>>>>>>> sensor via a CSI-2 link and processes the image data with
> >>>>>>>>>>>>>> the embedded AI engine. The detected events are sent over
> >>>>>>>>>>>>>> I2C to ISH (Intel Sensor Hub) for additional data fusion
> >>>>>>>>>>>>>> from multiple
> >>>> sensors.
> >>>>>>>>>>>>>> The fusion results are used to implement advanced use cases
> like:
> >>>>>>>>>>>>>> - Face detection to unlock screen
> >>>>>>>>>>>>>> - Detect user presence to manage backlight setting or
> >>>>>>>>>>>>>> waking up system
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Since the Image Processing Unit(IPU) used on the host
> >>>>>>>>>>>>>> processor needs to configure the CSI-2 link in normal
> >>>>>>>>>>>>>> camera usages, the
> >>>>>>>>>>>>>> CSI-2 link and camera sensor can only be used in
> >>>>>>>>>>>>>> mutually-exclusive ways by host IPU and IVSC. By default
> >>>>>>>>>>>>>> the IVSC owns the CSI-2 link and camera sensor. The IPU
> >>>>>>>>>>>>>> driver can take ownership of the CSI-2 link and camera
> >>>>>>>>>>>>>> sensor using interfaces provided
> >>>>>>>>>> by this IVSC driver.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Switching ownership requires an interface with two
> >>>>>>>>>>>>>> different hardware modules inside IVSC. The software
> >>>>>>>>>>>>>> interface to these modules is via Intel MEI (The Intel
> >> Management Engine) commands.
> >>>>>>>>>>>>>> These two hardware modules have two different MEI UUIDs
> >>>>>>>>>>>>>> to enumerate. These hardware
> >>>>>>>>>>>> modules are:
> >>>>>>>>>>>>>> - ACE (Algorithm Context Engine): This module is for
> >>>>>>>>>>>>>> algorithm computing when IVSC owns camera sensor. Also
> >>>>>>>>>>>>>> ACE module controls camera sensor's ownership. This
> >>>>>>>>>>>>>> hardware module is used to set ownership
> >>>>>>>>>>>> of camera sensor.
> >>>>>>>>>>>>>> - CSI (Camera Serial Interface): This module is used to
> >>>>>>>>>>>>>> route camera sensor data either to IVSC or to host for
> >>>>>>>>>>>>>> IPU driver and
> >>>>>>>>>> application.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> IVSC also provides a privacy mode. When privacy mode is
> >>>>>>>>>>>>>> turned on, camera sensor can't be used. This means that
> >>>>>>>>>>>>>> both ACE and host IPU can't get image data. And when this
> >>>>>>>>>>>>>> mode is turned on, host IPU driver is informed via a
> >>>>>>>>>>>>>> registered callback, so that user can be
> >>>>>>>>>> notified.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> In summary, to acquire ownership of camera by IPU driver,
> >>>>>>>>>>>>>> first ACE module needs to be informed of ownership and
> >>>>>>>>>>>>>> then to setup MIPI CSI-2 link for the camera sensor and IPU.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I thought this for a while and did some research, and I
> >>>>>>>>>>>>> can suggest the
> >>>>>>>>>>>>> following:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> - The IVSC sub-device implements a control for privacy
> >>>>>>>> (V4L2_CID_PRIVACY
> >>>>>>>>>>>>> is a good fit).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> - Camera sensor access needs to be requested from IVSC
> >>>>>>>>>>>>> before accessing
> >>>>>>>>>> the
> >>>>>>>>>>>>> sensor via I²C. The IVSC ownership control needs to be
> >>>>>>>>>>>>> in the
> >> right
> >>>>>>>>>>>>> setting for this to work, and device links can be used
> >>>>>>>>>>>>> for that
> >>>> purpose
> >>>>>>>>>>>>> (see device_link_add()). With DL_FLAG_PM_RUNTIME and
> >>>>>>>>>>>> DL_FLAG_RPM_ACTIVE,
> >>>>>>>>>>>>> the supplier devices will be PM runtime resumed before
> >>>>>>>>>>>>> the
> >>>> consumer
> >>>>>>>>>>>>> (camera sensor). As these devices are purely virtual on
> >>>>>>>>>>>>> host side and
> >>>>>> has
> >>>>>>>>>>>>> no power state as such, you can use runtime PM callbacks
> >>>>>>>>>>>>> to transfer
> >>>>>>>> the
> >>>>>>>>>>>>> ownership.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Interesting proposal to use device-links + runtime-pm for
> >>>>>>>>>>>> this instead of modelling this as an i2c-mux. FWIW I'm fine
> >>>>>>>>>>>> with going this route instead of using an i2c-mux approach.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I have been thinking about the i2c-mux approach a bit and
> >>>>>>>>>>>> the problem is that we are not really muxing but want to
> >>>>>>>>>>>> turn on/off control and AFAIK the i2c-mux framework simply
> >>>>>>>>>>>> leaves the mux muxed to the last used i2c-chain, so control
> >>>>>>>>>>>> will never be released when the i2c
> >>>>>>>>>> transfers are done.
> >>>>>>>>>>>>
> >>>>>>>>>>>> And if were to somehow modify things (or maybe there
> >>>>>>>>>>>> already is some release
> >>>>>>>>>>>> callback) then the downside becomes that the i2c-mux core
> >>>>>>>>>>>> code operates at the i2c transfer level. So each i2c
> >>>>>>>>>>>> read/write would then enable +
> >>>>>>>>>> disavle control.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Modelling this using something like runtime pm as such is a
> >>>>>>>>>>>> much better fit because then we request control once on
> >>>>>>>>>>>> probe / stream-on and release it once we are fully done,
> >>>>>>>>>>>> rather then requesting + releasing control once per i2c- transfer.
> >>>>>>>>>>>
> >>>>>>>>>>> Seems runtime pm can't fix the problem of initial i2c
> >>>>>>>>>>> transfer during sensor driver probe, probably we have to
> >>>>>>>>>>> switch to i2c-mux modeling
> >>>>>>>> way.
> >>>>>>>>>>
> >>>>>>>>>> What do you mean? The supplier devices are resumed before the
> >>>>>>>>>> driver's probe is called.
> >>>>>>>>>
> >>>>>>>>> But we setup the link with device_link_add during IVSC
> >>>>>>>>> driver's probe, we can't guarantee driver probe's sequence.
> >>>>>>>>
> >>>>>>>> Then maybe we need to do the device_link_add somewhere else.
> >>>>>>>
> >>>>>>> sensor's parent is the LJCA I2C device whose driver is being
> >>>>>>> upstream https://www.spinics.net/lists/kernel/msg4702552.htmland
> >>>>>>> and sensor's power is controlled by IVSC instead of INT3472 if
> >>>>>>> IVSC
> >> enabled.
> >>>>>>
> >>>>>> I believe that the INT3472 code is still involved at least on a
> >>>>>> Dell Latitude 9420 the INT3472 code still needs to set the
> >>>>>> clock-enable and the privacy-LED GPIOs otherwise the main camera
> >>>>>> won't
> >> work.
> >>>>>>
> >>>>>> So I'm not sure what you mean with "sensor's power is controlled
> >>>>>> by IVSC instead of INT3472" ?
> >>>>>
> >>>>> Could you please share your kernel log and DSDT? Thanks
> >>>>>
> >>>>> BR,
> >>>>> Wentong
> >>>>>>
> >>>>>>
> >>>>>>> struct device_link *device_link_add(struct device *consumer,
> >>>>>>> struct device *supplier, u32
> >>>>>>> flags)
> >>>>>>>
> >>>>>>> So probably we have to add above device_link_add in LJCA I2C's
> >>>>>>> driver, and we can find the consumer(camera sensor) with ACPI
> >>>>>>> API, but the supplier, mei_ace, is mei client device under mei
> >>>>>>> framework and it's dynamically allocated device instead of ACPI
> >>>>>>> device, probably I can find its parent with some ACPI lookup
> >>>>>>> from this LJCA I2C device, but unfortunately mei framework
> >>>>>>> doesn't export the API to find mei client device with its parent
> >>>>>>> bus device(struct
> >> mei_device).
> >>>>>>>
> >>>>>>> I'm not sure if modeling this mei_ace as LJCA I2C's runtime
> >>>>>>> power control is acceptable, if yes, probably this mei_ace
> >>>>>>> driver have to go with LJCA I2C device driver.
> >>>>>>
> >>>>>> Looking at the ACPI table the sensor ACPI device has 2 _DEP-s
> >>>>>> listed the I2C controller and the INT3472 device. Since we are
> >>>>>> already doing similar setup in the INT3472 device that seems like
> >>>>>> a good place to add the device_link()-s (it can return
> >>>>>> -EPROBE_DEFER to wait for the mei_ace
> >>>> to show up).
> >>>>>>
> >>>>>> But when the INT3472 code runs, the consumer device does not
> >>>>>> exist yet and AFAICT the same is true when the LCJA
> >>>>>> i2c-controller driver is getting
> >>>> registered.
> >>>>>> The consumer only exists when the i2c_client is instantiated and
> >>>>>> at that point the sensor drivers probe() method can run
> >>>>>> immediately and we are too late to add the device_link.
> >>>>>>
> >>>>>> As a hobby project I have been working on atomisp2 support and I
> >>>>>> have a similar issue there. There is no INT3472 device there, but
> >>>>>> there is a _DSM method which needs to be used to figure out which
> >>>>>> ACPI GPIO resource is reset / powerdown and if the GPIOs are
> >>>>>> active-low
> >> or active high.
> >>>>>>
> >>>>>> I have written a little helper function to call the _DSM and to
> >>>>>> then turn this into lookups and call devm_acpi_dev_add_driver_gpios().
> >>>>>>
> >>>>>> Since on atomisp2 we cannot use the INT3472 driver to delay the
> >>>>>> sensor-driver probe and have the INT3472 driver setup the GPIO
> >>>>>> lookup, at least for the sensor drivers used with
> >>>>>> atomisp2 there is going to be a need to add a single line to probe() like
> this:
> >>>>>>
> >>>>>> v4l2_get_acpi_sensor_info(&i2c_client->dev, NULL);
> >>>>>>
> >>>>>> To me it sounds like we need to do something similar here and
> >>>>>> extend the helper function which I have written (but not yet
> >>>>>> submitted
> >> upstream) :
> >>>>>>
> >>>>>> https://github.com/jwrdegoede/linux-
> >>>>>> sunxi/commit/e2287979db43d46fa7d354c1bde92eb6219b613d
> >>>>>>
> >>>>>> To also setup the device-links needed for the runtime-pm solution
> >>>>>> to getting the i2c passed through to the sensor.
> >>>>>>
> >>>>>> Ideally v4l2_get_acpi_sensor_info() should return void (easier to
> >>>>>> use in the sensor drivers) but I think it should return an int,
> >>>>>> so that it can e.g. return - EPROBE_DEFER to wait for the mei_ace.
> >>>>>>
> >>>>>> Regards,
> >>>>>>
> >>>>>> Hans
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>> The mainline kernel delays probing of camera sensors on Intel
> >>>>>>>> platforms until the INT3472 driver has probed the INT3472
> >>>>>>>> device on which the sensors have an ACPI _DEP.
> >>>>>>>>
> >>>>>>>> This is already used to make sure that clock lookups and
> >>>>>>>> regulator info is in place before the sensor's probe() function runs.
> >>>>>>>>
> >>>>>>>> So that when the driver does clk_get() it succeeds and so that
> >>>>>>>> regulator_get() does not end up returning a dummy regulator.
> >>>>>>>>
> >>>>>>>> So I think the code adding the device_link-s for the IVSC
> >>>>>>>> should be added
> >>>>>>>> to: drivers/platform/x86/intel/int3472/discrete.c and then the
> >>>>>>>> runtime-resume will happen before the sensor's probe() function runs.
> >>>>>>>>
> >>>>>>>> Likewise drivers/platform/x86/intel/int3472/discrete.c should
> >>>>>>>> also ensure that the ivsc driver's probe() has run before it
> >>>>>>>> calls
> >>>>>> acpi_dev_clear_dependencies().
> >>>>>>>>
> >>>>>>>> The acpi_dev_clear_dependencies() call in discrete.c tells the
> >>>>>>>> ACPI subsystem to go ahead and create the i2c-clients for the
> >>>>>>>> sensors and allow the sensor drivers to get loaded and probe the
> sensor.
> >>>>>>>>
> >>>>>>>> Regards,
> >>>>>>>>
> >>>>>>>> Hans
> >>>>>>>
> >>>>>
> >
Hi Wentong,
On Fri, Mar 17, 2023 at 07:30:19AM +0000, Wu, Wentong wrote:
>
>
> > -----Original Message-----
> > From: Hans de Goede <[email protected]>
> > Sent: Thursday, March 16, 2023 5:04 PM
> >
> > Hi,
> >
> > On 3/16/23 03:58, Wu, Wentong wrote:
> > >
> > >
> > >> -----Original Message-----
> > >> From: Hans de Goede <[email protected]>
> > >> Sent: Thursday, March 9, 2023 11:24 PM
> > >>
> > >> <re-added the previous Cc list, which I dropped because of the large
> > >> attachments>
> > >>
> > >> Hi Wentong,
> > >>
> > >> On 3/9/23 15:29, Wu, Wentong wrote:
> > >>> Hi Hans,
> > >>>
> > >>> Thanks
> > >>>
> > >>> And AFAICT, there is no IVSC device on your Dell Latitude 9420 where
> > >>> the
> > >> platform is based on TGL instead of ADL, and I have never heard IVSC
> > >> runs on TGL, if no IVSC, INT3472 will control sensor's power.
> > >>> And I will double confirm with people who know dell product well tomorrow.
> > >>
> > >> Ah, I was under the impression that there was an IVSC there because:
> > >>
> > >> 1. The sensor driver for the used sensor (tries to) poke the IVSC 2.
> > >> Things did not work without building the IVSC drivers, but that might
> > >> be due to a dependency on the LCJA GPIO expander instead
> > >
> > > Below is your dmesg log, the required SPI controller for IVSC isn't here.
> > >
> > > [ 35.538114] ljca 2-6:1.0: acked sem wait timed out ret:0 timeout:20 ack:0
> > > [ 35.538129] ljca 2-6:1.0: MNG_ENUM_SPI failed ret:-110 len:7 num:0
> > > [ 35.538621] ljca 2-6:1.0: LJCA USB device init success
> > > [ 35.538776] usbcore: registered new interface driver ljca
> > >
> > > Also I checked your SSDT, there is no IVSC device and the sensor
> > > device depends on
> > > INT3472 instead of IVSC device as on my setup.
> >
> > Ack.
> >
> > >> But you might very well be right, that would also explain the "intel vsc not
> > ready"
> > >> messages in dmesg.
> > >>
> > >> If with the IVSC case the IVSC controls the power to the sensor too,
> > >> then another option might be to model the I2C-switch + the
> > >> power-control as a powerdown GPIO for the sensor, which most sensor
> > drivers already try to use.
> > >> The advantage of doing this would be that GPIO lookups can reference
> > >> the GPIO provider + consumer by device-name so then we don't need to
> > >> have both devices instantiated at the time of
> > >> adding the GPIO lookup. And in that case we could e.g. add the lookup
> > >> before registering the I2C controller.
> > >
> > > Can we add IVSC device to acpi_honor_dep_ids, so that when everything
> > > is done during mei_ace probe, acpi_dev_clear_dependencies can make sensor
> > start probe?
> >
> > Does the sensor ACPI device node have an ACPI _DEP on the IVSC device ?
>
> Yes,
>
> >
> > If yes, then yes we can add the IVSC device to acpi_honor_dep_id and make
> > mei_ace probe call acpi_dev_clear_dependencies().
>
> But I prefer the powerdown gpio model, because we have to follow the commands
> sequences as below which is required by firmware, runtime pm is hard to achieve this.
How so?
I don't insist on the runtime PM based solution but I'd rather not have
changes to virtually all sensor drivers --- this is an external chip to
them.
> + /* switch camera sensor ownership to host */
> + ret = ace_set_camera_owner(ACE_CAMERA_HOST);
> + if (ret)
> + goto error;
> +
> + /* switch CSI-2 link to host */
> + ret = csi_set_link_owner(CSI_LINK_HOST, callback, context);
> + if (ret)
> + goto release_camera;
> +
> + /* configure CSI-2 link */
> + ret = csi_set_link_cfg(nr_of_lanes, link_freq);
> + if (ret)
> + goto release_csi;
--
Kind regards,
Sakari Ailus
> -----Original Message-----
> From: Sakari Ailus <[email protected]>
> Sent: Friday, March 17, 2023 4:59 PM
>
> Hi Wentong,
>
> On Fri, Mar 17, 2023 at 07:30:19AM +0000, Wu, Wentong wrote:
> >
> >
> > > -----Original Message-----
> > > From: Hans de Goede <[email protected]>
> > > Sent: Thursday, March 16, 2023 5:04 PM
> > >
> > > Hi,
> > >
> > > On 3/16/23 03:58, Wu, Wentong wrote:
> > > >
> > > >
> > > >> -----Original Message-----
> > > >> From: Hans de Goede <[email protected]>
> > > >> Sent: Thursday, March 9, 2023 11:24 PM
> > > >>
> > > >> <re-added the previous Cc list, which I dropped because of the
> > > >> large
> > > >> attachments>
> > > >>
> > > >> Hi Wentong,
> > > >>
> > > >> On 3/9/23 15:29, Wu, Wentong wrote:
> > > >>> Hi Hans,
> > > >>>
> > > >>> Thanks
> > > >>>
> > > >>> And AFAICT, there is no IVSC device on your Dell Latitude 9420
> > > >>> where the
> > > >> platform is based on TGL instead of ADL, and I have never heard
> > > >> IVSC runs on TGL, if no IVSC, INT3472 will control sensor's power.
> > > >>> And I will double confirm with people who know dell product well
> tomorrow.
> > > >>
> > > >> Ah, I was under the impression that there was an IVSC there because:
> > > >>
> > > >> 1. The sensor driver for the used sensor (tries to) poke the IVSC 2.
> > > >> Things did not work without building the IVSC drivers, but that might
> > > >> be due to a dependency on the LCJA GPIO expander instead
> > > >
> > > > Below is your dmesg log, the required SPI controller for IVSC isn't here.
> > > >
> > > > [ 35.538114] ljca 2-6:1.0: acked sem wait timed out ret:0 timeout:20
> ack:0
> > > > [ 35.538129] ljca 2-6:1.0: MNG_ENUM_SPI failed ret:-110 len:7 num:0
> > > > [ 35.538621] ljca 2-6:1.0: LJCA USB device init success
> > > > [ 35.538776] usbcore: registered new interface driver ljca
> > > >
> > > > Also I checked your SSDT, there is no IVSC device and the sensor
> > > > device depends on
> > > > INT3472 instead of IVSC device as on my setup.
> > >
> > > Ack.
> > >
> > > >> But you might very well be right, that would also explain the
> > > >> "intel vsc not
> > > ready"
> > > >> messages in dmesg.
> > > >>
> > > >> If with the IVSC case the IVSC controls the power to the sensor
> > > >> too, then another option might be to model the I2C-switch + the
> > > >> power-control as a powerdown GPIO for the sensor, which most
> > > >> sensor
> > > drivers already try to use.
> > > >> The advantage of doing this would be that GPIO lookups can
> > > >> reference the GPIO provider + consumer by device-name so then we
> > > >> don't need to have both devices instantiated at the time of
> > > >> adding the GPIO lookup. And in that case we could e.g. add the lookup
> > > >> before registering the I2C controller.
> > > >
> > > > Can we add IVSC device to acpi_honor_dep_ids, so that when
> > > > everything is done during mei_ace probe,
> > > > acpi_dev_clear_dependencies can make sensor
> > > start probe?
> > >
> > > Does the sensor ACPI device node have an ACPI _DEP on the IVSC device ?
> >
> > Yes,
> >
> > >
> > > If yes, then yes we can add the IVSC device to acpi_honor_dep_id and
> > > make mei_ace probe call acpi_dev_clear_dependencies().
> >
> > But I prefer the powerdown gpio model, because we have to follow the
> > commands sequences as below which is required by firmware, runtime pm is
> hard to achieve this.
>
> How so?
But we have to find a way to download commands to firmware following below
sequence before writing camera sensor's registers to start camera sensor's steam.
BR,
Wentong
>
> I don't insist on the runtime PM based solution but I'd rather not have changes
> to virtually all sensor drivers --- this is an external chip to them.
>
> > + /* switch camera sensor ownership to host */
> > + ret = ace_set_camera_owner(ACE_CAMERA_HOST);
> > + if (ret)
> > + goto error;
> > +
> > + /* switch CSI-2 link to host */
> > + ret = csi_set_link_owner(CSI_LINK_HOST, callback, context);
> > + if (ret)
> > + goto release_camera;
> > +
> > + /* configure CSI-2 link */
> > + ret = csi_set_link_cfg(nr_of_lanes, link_freq);
> > + if (ret)
> > + goto release_csi;
>
> --
> Kind regards,
>
> Sakari Ailus