Hi,
The following series is a driver for a new PCIe device from Intel named NNP-I
(Nirvana Neural Processor for Inference). NNP-I is a PCIe connected compute
device used for acceleration of AI deep learning inference applications in the
data-center.
The reason that this driver should be in the kernel is that it aims to serve
multiple users and user-space applications which might share the same NNP-I
card. Workloads from multiple applications can be processed simultanously by
the NNP-I card if enough compute resources exist.
Overview of the NNP-I device, driver structure and ABIs used in the driver is in
patch#1, which adds the info as a document as it might be a useful info for
anyone trying to understand the driver even past review.
In order to ease the review process, there will be multiple series for the
entire driver code. This is the first series, and it implements everything
necessary to initialize the NNP-I device and allow a user-space inference
application to use it. Other features, which are mostly related to maintenance,
device status visibility and error-handling, will be submitted on the next stage.
A basic user-space library and test application which illustrates the flow of
an NNP-I inference application can be found here: https://github.com/IntelAI/nnpi-host
(This series is enough for the test application to run)
This patchset has gone through internal review inside Intel, the summary of the
change log from the internal review follows.
I would appreciate any feedback, questions or comments to this series.
Changes in v22:
- Patch#13: remove back-to-back mutex acquire when calling to check if response
ring buffer is empty.
- Patch#15: remove not needed output argument to do_map_hostres() by calling
to send_map_hostres_req() from do_map_hostres().
Changes in v21:
- Fixed kerneldoc warnings.
- Patch#15 - break ioctl handling function map_hostres() to smaller functions.
- Few code style issues
- Removed not-needed error printouts.
- Removed channel "respq_corrupted" state, instead made chan_read return 0
without modifying the response ring-buffer.
Changes in v20:
- Patcg#13: Handle response ring-buffer corruption by adding "respq_corruped"
state.
- Patch#15: protect host resource mappings using existing dev_mutex lock
instead of a new spin lock.
- Patch#12: removed not needed "force" arg to nnpdev_submit_device_event_to_channels()
- Replaced two boolean states of cmd_chan to a single channel status state
- Few code style issues
Changes in v19:
- Use global mutex to protect device chardev clients list instead of per-device
mutex. (to simplify handling of device disconnect during device removal).
Changes in v18:
- Added kref to device chardev client structure to deal with device removal
possible race condition.
- Fixed lkp build failure on 32-bit arch.
- Some code style issues
Changes in v17:
- Fixes v16 review comments only.
- Added Alex's reviewed-by tag to patches 2,3,5,6,8,9
(after fixing requested comments)
- Created new patch#7 as suggested by Alexander Shishkin to pull some
functionality from patch#6 to a separate patch.
- Fixed lkp build warning on arm arch by using:
ifdef CONFIG_ARCH_DMA_ADDR_T_64BIT.
(This is in patch#2)
- Rebased on top of 5.12-rc3
Changes in v16:
- Fixed v15 review comments
- Added MODULE_DEVICE_TABLE() usage for pci driver.
Changes in v15:
- fixed v14 review comments
- few bug fixes found during testing:
+ made channel's resosponse ring buffer be resizable as some stress apps
stretch it.
+ Use mutex instead of spin_lock for protecting nnpdev pointer in a channel.
This is since it needs to be locked when applications do write to the channel
file.
Changes in v14:
- converted overview document to .rst file
- refactor into two modules, NNP-I framework and PCIe device driver.
(as suggested by Andy Shevchenko)
- changed device remove flow to disconnect device from active clients,
rather then waiting for clients to exit.
- Fixed all review comments.
Changes in v13:
- Fixed errors from codespell
- Fixed errors reported by sparse and cppcheck
- Modified commit messages
- Fixed all comments from v12
Changes in v12:
- Removed sharing of boot image between multiple devices
- Fixed comments from v11
Changes in v11:
- Fixed comments from v10
- Removed the following features, reducing the size of the patch-set:
- Sysfs visibility
- Handling of device errors
- Visibility to static device system info (total memory, ice count, steping, ...)
- Access to device crash log
- BIOS capsule update
- enable/disable device for new inference contexts
- PCIe reset and error handling
- Visibility to device post code and bios flash progress
- Ability to get more command queue BW for inference contexts comparing to nnpiml channels
- Device hang detection
- PCIe error injection
- Host SW traces
- Support host resources from dma-buf objects (created by another driver)
- Block non-privileged users to do any nnpi_ctl commands and query SW counters of ALL contexts.
Changes in v10:
- removed bitfield in unions from protocol (using bit masks instead)
(as suggested by Andy Shevchenko)
- renamed function names to be more consistant
- changed logical/pci "layers" to be called device and HW layers.
- removed host resource allocation method from IOCTL. Support only pinning user memory.
- re-wrote most of the commit messages, based on Alexander Shishkin comments.
- fixed errors reported by lkp
Changes in v9:
- fixed potential dead-lock in boot load flow
- IOCTL structs alignment issues
- sysfs - one value per-attribute
- code-style: func arguments and multi-line alignment
- sparse errors reported by lkp
Changes in v8:
- broke the most of the big patches to little smaller ones.
- Reviewed and fixed all comments from v7.
Changes in v7:
- Added documentation sections to many structs and functions.
- Added "Terminology" section to Documentation/misc-devices/intel-nnpi.txt
- Removed use of function pointers with interface to PCIe "h/w layer"
- Fixed IOCTLs backward compatability support.
- Patch#13 has removed - ipc s/w counters
- Patch#15 has removed - handling PCIe link-down in absense of pciehp
- Byteorder consideration - Fixed data packets sent to the device to be
in little-endian. Other communications with the device is through
mmio which is little-endian anyway.
- Removed sysfs "reset" attribute
- Removed sysfs attribute which outputs application pids.
- Fixed and cleaned all other comments.
Changes in v6:
- lkp build fixes
- fixes build errors when tracing is enabled
- made trace compiled by default, define NNP_DISABLE_TRACE to disable.
- fixed reviewed-by tag to mention first name before last name.
- serialize pcie reset and remove flows.
- rebased on top of current linux master
Changes in v5:
- Makefile fix for out-of-tree builds
(added $(srctree) when specifying include path)
Changes in v4:
- Per Dave-Hansen suggestion, abandon patch#20
(misc: nnpi: Added hostres_min_order module parameter)
Realized that the benefit it brings not worth the risk.
- Fixes build failures found by lkp
- Some bug fixes found in v3
- Clean W=1 build warnings
Changes in v3:
- Few small BUG fixes found during testing
- Add device bios update flow - boot flow has changed to allow
booting the device either with OS boot image or bios capsule image.
The device bios will start bios update if capsule image is used.
- Fixed comments from previous version
Changes in v2:
- used --strict flag to checkpatch.pl, only left CHECK comments which
will break the 80 chars line length limit if fixed.
- removed CONFIG_DMA_SHARED_BUFFER ifdefs
- moved high order page allocation optimization to separete (last) patch
- removed device list array
- removed all c++ style comments
Thanks,
Guy.
Guy Zadicario (15):
misc: nnpi: Document NNP-I's driver overview
misc: nnpi: Initialize NNP-I framework and PCIe modules
misc: nnpi: Manage and schedule messages to device
misc: nnpi: Define host/card ipc protocol
misc: nnpi: Manage host memory resources
misc: nnpi: Allow usermode to manage host resources
misc: nnpi: Disallow host memory resource access if no NNP-I devices
exist
misc: nnpi: Boot NNP-I device
misc: nnpi: Process device response messages
misc: nnpi: Query and verify device protocol
misc: nnpi: Create comm channel from app to device
misc: nnpi: Route device response messages
misc: nnpi: Expose command channel file interface
misc: nnpi: Create command channel from userspace
misc: nnpi: Map host resources to device channel
Documentation/ABI/testing/sysfs-driver-intel_nnpi | 5 +
Documentation/misc-devices/index.rst | 1 +
Documentation/misc-devices/intel-nnpi.rst | 237 +++++
MAINTAINERS | 6 +
drivers/misc/Kconfig | 1 +
drivers/misc/Makefile | 1 +
drivers/misc/intel-nnpi/Kconfig | 18 +
drivers/misc/intel-nnpi/Makefile | 13 +
drivers/misc/intel-nnpi/bootimage.c | 246 +++++
drivers/misc/intel-nnpi/bootimage.h | 43 +
drivers/misc/intel-nnpi/cmd_chan.c | 790 ++++++++++++++
drivers/misc/intel-nnpi/cmd_chan.h | 134 +++
drivers/misc/intel-nnpi/device.c | 1081 ++++++++++++++++++++
drivers/misc/intel-nnpi/device.h | 182 ++++
drivers/misc/intel-nnpi/device_chardev.c | 789 ++++++++++++++
drivers/misc/intel-nnpi/device_chardev.h | 14 +
drivers/misc/intel-nnpi/host_chardev.c | 353 +++++++
drivers/misc/intel-nnpi/host_chardev.h | 12 +
drivers/misc/intel-nnpi/hostres.c | 627 ++++++++++++
drivers/misc/intel-nnpi/hostres.h | 167 +++
.../misc/intel-nnpi/ipc_include/ipc_c2h_events.h | 198 ++++
drivers/misc/intel-nnpi/ipc_include/ipc_protocol.h | 340 ++++++
.../misc/intel-nnpi/ipc_include/nnp_boot_defs.h | 71 ++
drivers/misc/intel-nnpi/ipc_include/nnp_elbi.h | 91 ++
drivers/misc/intel-nnpi/msg_scheduler.c | 319 ++++++
drivers/misc/intel-nnpi/msg_scheduler.h | 153 +++
drivers/misc/intel-nnpi/nnp_pcie.c | 530 ++++++++++
drivers/misc/intel-nnpi/nnp_user.c | 131 +++
drivers/misc/intel-nnpi/nnp_user.h | 79 ++
include/uapi/misc/intel_nnpi.h | 304 ++++++
30 files changed, 6936 insertions(+)
create mode 100644 Documentation/ABI/testing/sysfs-driver-intel_nnpi
create mode 100644 Documentation/misc-devices/intel-nnpi.rst
create mode 100644 drivers/misc/intel-nnpi/Kconfig
create mode 100644 drivers/misc/intel-nnpi/Makefile
create mode 100644 drivers/misc/intel-nnpi/bootimage.c
create mode 100644 drivers/misc/intel-nnpi/bootimage.h
create mode 100644 drivers/misc/intel-nnpi/cmd_chan.c
create mode 100644 drivers/misc/intel-nnpi/cmd_chan.h
create mode 100644 drivers/misc/intel-nnpi/device.c
create mode 100644 drivers/misc/intel-nnpi/device.h
create mode 100644 drivers/misc/intel-nnpi/device_chardev.c
create mode 100644 drivers/misc/intel-nnpi/device_chardev.h
create mode 100644 drivers/misc/intel-nnpi/host_chardev.c
create mode 100644 drivers/misc/intel-nnpi/host_chardev.h
create mode 100644 drivers/misc/intel-nnpi/hostres.c
create mode 100644 drivers/misc/intel-nnpi/hostres.h
create mode 100644 drivers/misc/intel-nnpi/ipc_include/ipc_c2h_events.h
create mode 100644 drivers/misc/intel-nnpi/ipc_include/ipc_protocol.h
create mode 100644 drivers/misc/intel-nnpi/ipc_include/nnp_boot_defs.h
create mode 100644 drivers/misc/intel-nnpi/ipc_include/nnp_elbi.h
create mode 100644 drivers/misc/intel-nnpi/msg_scheduler.c
create mode 100644 drivers/misc/intel-nnpi/msg_scheduler.h
create mode 100644 drivers/misc/intel-nnpi/nnp_pcie.c
create mode 100644 drivers/misc/intel-nnpi/nnp_user.c
create mode 100644 drivers/misc/intel-nnpi/nnp_user.h
create mode 100644 include/uapi/misc/intel_nnpi.h
--
1.8.3.1
---------------------------------------------------------------------
Intel Israel (74) Limited
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
Provide an IOCTL interface for creating and destroying host memory
resources through a character device (/dev/nnpi_host).
There is a single instance of this character device in the system
regardless of the number of NNP-I devices attached because it
controls host resources which may be shared between different devices.
A nnp_user object, created when an application opens this character
device, identifies the user (client) of the driver and holds a list of
all host resources allocated by that user through the opened file
descriptor.
Host memory resources created through this character device can be mapped
to device access through IOCTLs made to a different, per-device, chardev
(will be introduced on next commits).
All resources will be destroyed when the application closes the connection
or exits.
The IOCTL interface is defined in: include/uapi/misc/intel_nnpi.h
Signed-off-by: Guy Zadicario <[email protected]>
Reviewed-by: Alexander Shishkin <[email protected]>
---
MAINTAINERS | 1 +
drivers/misc/intel-nnpi/Makefile | 2 +-
drivers/misc/intel-nnpi/device.c | 14 ++
drivers/misc/intel-nnpi/host_chardev.c | 346 +++++++++++++++++++++++++++++++++
drivers/misc/intel-nnpi/host_chardev.h | 12 ++
drivers/misc/intel-nnpi/nnp_user.c | 131 +++++++++++++
drivers/misc/intel-nnpi/nnp_user.h | 79 ++++++++
include/uapi/misc/intel_nnpi.h | 150 ++++++++++++++
8 files changed, 734 insertions(+), 1 deletion(-)
create mode 100644 drivers/misc/intel-nnpi/host_chardev.c
create mode 100644 drivers/misc/intel-nnpi/host_chardev.h
create mode 100644 drivers/misc/intel-nnpi/nnp_user.c
create mode 100644 drivers/misc/intel-nnpi/nnp_user.h
create mode 100644 include/uapi/misc/intel_nnpi.h
diff --git a/MAINTAINERS b/MAINTAINERS
index ff0c3d7..42f3e54 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -9320,6 +9320,7 @@ INTEL NNP-I PCI DRIVER
M: Guy Zadicario <[email protected]>
S: Supported
F: drivers/misc/intel-nnpi/
+F: include/uapi/misc/intel_nnpi.h
INTEL P-Unit IPC DRIVER
M: Zha Qipeng <[email protected]>
diff --git a/drivers/misc/intel-nnpi/Makefile b/drivers/misc/intel-nnpi/Makefile
index 3713d51..da2863b 100644
--- a/drivers/misc/intel-nnpi/Makefile
+++ b/drivers/misc/intel-nnpi/Makefile
@@ -5,7 +5,7 @@
obj-$(CONFIG_INTEL_NNPI) := intel_nnpi.o intel_nnpi_pcie.o
-intel_nnpi-y := device.o msg_scheduler.o hostres.o
+intel_nnpi-y := device.o msg_scheduler.o hostres.o host_chardev.o nnp_user.o
intel_nnpi_pcie-y := nnp_pcie.o
diff --git a/drivers/misc/intel-nnpi/device.c b/drivers/misc/intel-nnpi/device.c
index 60c5a94..0f98398 100644
--- a/drivers/misc/intel-nnpi/device.c
+++ b/drivers/misc/intel-nnpi/device.c
@@ -8,6 +8,7 @@
#include <linux/module.h>
#include "device.h"
+#include "host_chardev.h"
#include "msg_scheduler.h"
static DEFINE_IDA(dev_ida);
@@ -99,6 +100,19 @@ void nnpdev_destroy(struct nnp_device *nnpdev)
}
EXPORT_SYMBOL(nnpdev_destroy);
+static int __init nnp_init(void)
+{
+ return nnp_init_host_interface();
+}
+subsys_initcall(nnp_init);
+
+static void __exit nnp_cleanup(void)
+{
+ nnp_release_host_interface();
+ /* dev_ida is already empty here - no point calling ida_destroy */
+}
+module_exit(nnp_cleanup);
+
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Intel(R) NNPI Framework");
MODULE_AUTHOR("Intel Corporation");
diff --git a/drivers/misc/intel-nnpi/host_chardev.c b/drivers/misc/intel-nnpi/host_chardev.c
new file mode 100644
index 0000000..6048fda
--- /dev/null
+++ b/drivers/misc/intel-nnpi/host_chardev.c
@@ -0,0 +1,346 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (C) 2019-2021 Intel Corporation */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/cdev.h>
+#include <linux/device.h>
+#include <linux/file.h>
+#include <linux/fs.h>
+#include <linux/hashtable.h>
+#include <linux/idr.h>
+#include <linux/printk.h>
+#include <linux/slab.h>
+#include <uapi/misc/intel_nnpi.h>
+
+#include "device.h"
+#include "host_chardev.h"
+#include "ipc_protocol.h"
+#include "nnp_user.h"
+
+static struct cdev cdev;
+static dev_t devnum;
+static struct class *class;
+static struct device *dev;
+
+static inline int is_host_file(struct file *f);
+
+static enum dma_data_direction to_dma_dir(unsigned int nnp_dir)
+{
+ /* Ignore IOCTL_INF_RES_NETWORK */
+ switch (nnp_dir & (IOCTL_INF_RES_INPUT | IOCTL_INF_RES_OUTPUT)) {
+ case (IOCTL_INF_RES_INPUT | IOCTL_INF_RES_OUTPUT):
+ return DMA_BIDIRECTIONAL;
+ case IOCTL_INF_RES_INPUT:
+ return DMA_TO_DEVICE;
+ case IOCTL_INF_RES_OUTPUT:
+ return DMA_FROM_DEVICE;
+ default:
+ break;
+ }
+
+ return DMA_NONE;
+}
+
+static long create_hostres(struct nnp_user_info *user_info, void __user *arg,
+ unsigned int size)
+{
+ int ret;
+ struct nnpdrv_ioctl_create_hostres req;
+ struct host_resource *hostres;
+ struct user_hostres *user_hostres_entry;
+ void __user *uptr;
+ unsigned int io_size = sizeof(req);
+
+ if (size != io_size)
+ return -EINVAL;
+
+ if (copy_from_user(&req, arg, io_size))
+ return -EFAULT;
+
+ if (req.usage_flags & ~IOCTL_RES_USAGE_VALID_MASK)
+ return -EINVAL;
+
+ uptr = u64_to_user_ptr(req.user_ptr);
+ hostres = nnp_hostres_from_usermem(uptr, req.size,
+ to_dma_dir(req.usage_flags));
+
+ if (IS_ERR(hostres))
+ return PTR_ERR(hostres);
+
+ ret = nnp_user_add_hostres(user_info, hostres, &user_hostres_entry);
+ if (ret < 0) {
+ nnp_hostres_put(hostres);
+ return ret;
+ }
+
+ req.size = nnp_hostres_size(hostres);
+
+ /*
+ * The created user_hostres_entry holds refcount to the resource,
+ * no need to keep another one here.
+ */
+ nnp_hostres_put(hostres);
+
+ req.user_handle = user_hostres_entry->user_handle;
+ if (copy_to_user(arg, &req, io_size)) {
+ ret = -EFAULT;
+ goto destroy_hostres_entry;
+ }
+
+ return 0;
+
+destroy_hostres_entry:
+ nnp_user_remove_hostres(user_hostres_entry);
+
+ return ret;
+}
+
+static long destroy_hostres(struct nnp_user_info *user_info, void __user *arg,
+ unsigned int size)
+{
+ struct nnpdrv_ioctl_destroy_hostres destroy_args;
+ struct user_hostres *user_hostres_entry;
+ unsigned int io_size = sizeof(destroy_args);
+ int ret = 0;
+
+ if (size != io_size)
+ return -EINVAL;
+
+ if (copy_from_user(&destroy_args, arg, io_size))
+ return -EFAULT;
+
+ /* errno must be cleared on entry */
+ if (destroy_args.o_errno)
+ return -EINVAL;
+
+ mutex_lock(&user_info->mutex);
+ user_hostres_entry = idr_find(&user_info->idr, destroy_args.user_handle);
+ if (user_hostres_entry) {
+ nnp_user_remove_hostres_locked(user_hostres_entry);
+ } else {
+ destroy_args.o_errno = NNPER_NO_SUCH_RESOURCE;
+ if (copy_to_user(arg, &destroy_args, io_size))
+ ret = -EFAULT;
+ }
+
+ mutex_unlock(&user_info->mutex);
+ return ret;
+}
+
+static long lock_hostres(struct nnp_user_info *user_info, void __user *arg,
+ unsigned int size)
+{
+ int ret = 0;
+ struct nnpdrv_ioctl_lock_hostres lock_args;
+ struct user_hostres *user_hostres_entry;
+ unsigned int io_size = sizeof(lock_args);
+
+ if (size != io_size)
+ return -EINVAL;
+
+ if (copy_from_user(&lock_args, arg, io_size))
+ return -EFAULT;
+
+ /* errno must be cleared on entry */
+ if (lock_args.o_errno)
+ return -EINVAL;
+
+ mutex_lock(&user_info->mutex);
+ user_hostres_entry = idr_find(&user_info->idr, lock_args.user_handle);
+ if (user_hostres_entry) {
+ ret = nnp_hostres_user_lock(user_hostres_entry->hostres);
+ } else {
+ lock_args.o_errno = NNPER_NO_SUCH_RESOURCE;
+ if (copy_to_user(arg, &lock_args, io_size))
+ ret = -EFAULT;
+ }
+
+ mutex_unlock(&user_info->mutex);
+ return ret;
+}
+
+static long unlock_hostres(struct nnp_user_info *user_info, void __user *arg,
+ unsigned int size)
+{
+ int ret = 0;
+ struct user_hostres *user_hostres_entry;
+ struct nnpdrv_ioctl_lock_hostres lock_args;
+ unsigned int io_size = sizeof(lock_args);
+
+ if (size != io_size)
+ return -EINVAL;
+
+ if (copy_from_user(&lock_args, arg, io_size))
+ return -EFAULT;
+
+ /* errno must be cleared on entry */
+ if (lock_args.o_errno)
+ return -EINVAL;
+
+ mutex_lock(&user_info->mutex);
+ user_hostres_entry = idr_find(&user_info->idr, lock_args.user_handle);
+ if (user_hostres_entry) {
+ ret = nnp_hostres_user_unlock(user_hostres_entry->hostres);
+ } else {
+ lock_args.o_errno = NNPER_NO_SUCH_RESOURCE;
+ if (copy_to_user(arg, &lock_args, sizeof(lock_args)))
+ ret = -EFAULT;
+ }
+
+ mutex_unlock(&user_info->mutex);
+ return ret;
+}
+
+struct file *nnp_host_file_get(int host_fd)
+{
+ struct file *host_file;
+
+ host_file = fget(host_fd);
+ if (is_host_file(host_file))
+ return host_file;
+
+ if (host_file)
+ fput(host_file);
+
+ return NULL;
+}
+
+/*
+ * Inference host cdev (/dev/nnpi_host) file operation functions
+ */
+
+static int host_open(struct inode *inode, struct file *f)
+{
+ struct nnp_user_info *user_info;
+
+ if (!is_host_file(f))
+ return -EINVAL;
+
+ user_info = kzalloc(sizeof(*user_info), GFP_KERNEL);
+ if (!user_info)
+ return -ENOMEM;
+
+ nnp_user_init(user_info);
+
+ f->private_data = user_info;
+
+ return 0;
+}
+
+static int host_release(struct inode *inode, struct file *f)
+{
+ struct nnp_user_info *user_info;
+
+ if (!is_host_file(f))
+ return -EINVAL;
+
+ user_info = f->private_data;
+
+ nnp_user_destroy_all(user_info);
+ f->private_data = NULL;
+
+ return 0;
+}
+
+static long host_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
+{
+ long ret = 0;
+ struct nnp_user_info *user_info = f->private_data;
+ unsigned int ioc_nr, size;
+
+ if (!is_host_file(f))
+ return -ENOTTY;
+
+ if (_IOC_TYPE(cmd) != 'h')
+ return -EINVAL;
+
+ ioc_nr = _IOC_NR(cmd);
+ size = _IOC_SIZE(cmd);
+
+ switch (ioc_nr) {
+ case _IOC_NR(IOCTL_INF_CREATE_HOST_RESOURCE):
+ ret = create_hostres(user_info, (void __user *)arg, size);
+ break;
+ case _IOC_NR(IOCTL_INF_DESTROY_HOST_RESOURCE):
+ ret = destroy_hostres(user_info, (void __user *)arg, size);
+ break;
+ case _IOC_NR(IOCTL_INF_UNLOCK_HOST_RESOURCE):
+ ret = unlock_hostres(user_info, (void __user *)arg, size);
+ break;
+ case _IOC_NR(IOCTL_INF_LOCK_HOST_RESOURCE):
+ ret = lock_hostres(user_info, (void __user *)arg, size);
+ break;
+ default:
+ ret = -EINVAL;
+ }
+
+ return ret;
+}
+
+static const struct file_operations host_fops = {
+ .owner = THIS_MODULE,
+ .open = host_open,
+ .release = host_release,
+ .unlocked_ioctl = host_ioctl,
+ .compat_ioctl = host_ioctl,
+};
+
+static inline int is_host_file(struct file *f)
+{
+ return f && f->f_op == &host_fops;
+}
+
+int nnp_init_host_interface(void)
+{
+ int ret;
+
+ ret = alloc_chrdev_region(&devnum, 0, 1, NNPDRV_INF_HOST_DEV_NAME);
+ if (ret < 0)
+ return ret;
+
+ cdev_init(&cdev, &host_fops);
+ cdev.owner = THIS_MODULE;
+
+ ret = cdev_add(&cdev, devnum, 1);
+ if (ret < 0)
+ goto err_region;
+
+ class = class_create(THIS_MODULE, NNPDRV_INF_HOST_DEV_NAME);
+ if (IS_ERR(class)) {
+ ret = PTR_ERR(class);
+ goto err_cdev;
+ }
+
+ dev = device_create(class, NULL, devnum, NULL, NNPDRV_INF_HOST_DEV_NAME);
+ if (IS_ERR(dev)) {
+ ret = PTR_ERR(dev);
+ goto err_class;
+ }
+
+ ret = nnp_hostres_init_sysfs(dev);
+ if (ret < 0)
+ goto err_device;
+
+ return 0;
+
+err_device:
+ device_destroy(class, devnum);
+err_class:
+ class_destroy(class);
+err_cdev:
+ cdev_del(&cdev);
+err_region:
+ unregister_chrdev_region(devnum, 1);
+
+ return ret;
+}
+
+void nnp_release_host_interface(void)
+{
+ nnp_hostres_fini_sysfs(dev);
+ device_destroy(class, devnum);
+ class_destroy(class);
+ cdev_del(&cdev);
+ unregister_chrdev_region(devnum, 1);
+}
diff --git a/drivers/misc/intel-nnpi/host_chardev.h b/drivers/misc/intel-nnpi/host_chardev.h
new file mode 100644
index 0000000..5812e0f
--- /dev/null
+++ b/drivers/misc/intel-nnpi/host_chardev.h
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright (C) 2019-2021 Intel Corporation */
+
+#ifndef _NNPDRV_INFERENCE_H
+#define _NNPDRV_INFERENCE_H
+
+int nnp_init_host_interface(void);
+void nnp_release_host_interface(void);
+
+struct file *nnp_host_file_get(int host_fd);
+
+#endif
diff --git a/drivers/misc/intel-nnpi/nnp_user.c b/drivers/misc/intel-nnpi/nnp_user.c
new file mode 100644
index 0000000..51d7418
--- /dev/null
+++ b/drivers/misc/intel-nnpi/nnp_user.c
@@ -0,0 +1,131 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (C) 2019-2021 Intel Corporation */
+
+#include <linux/slab.h>
+
+#include "nnp_user.h"
+
+void nnp_user_init(struct nnp_user_info *user_info)
+{
+ INIT_LIST_HEAD(&user_info->hostres_list);
+ mutex_init(&user_info->mutex);
+ kref_init(&user_info->ref);
+ idr_init(&user_info->idr);
+}
+
+void nnp_user_get(struct nnp_user_info *user_info)
+{
+ kref_get(&user_info->ref);
+}
+
+static void nnp_user_release(struct kref *kref)
+{
+ struct nnp_user_info *user_info =
+ container_of(kref, struct nnp_user_info, ref);
+ struct completion *completion = user_info->close_completion;
+
+ idr_destroy(&user_info->idr);
+ kfree(user_info);
+ complete(completion);
+}
+
+void nnp_user_put(struct nnp_user_info *user_info)
+{
+ kref_put(&user_info->ref, nnp_user_release);
+}
+
+int nnp_user_add_hostres(struct nnp_user_info *user_info,
+ struct host_resource *hostres,
+ struct user_hostres **user_hostres_entry)
+{
+ struct user_hostres *hr_entry;
+ int id;
+
+ hr_entry = kmalloc(sizeof(*hr_entry), GFP_KERNEL);
+ if (!hr_entry)
+ return -ENOMEM;
+
+ /*
+ * Increment refcount to hostres for the entry reference.
+ * (caller holds reference to it, so we know it exist).
+ */
+ nnp_hostres_get(hostres);
+ hr_entry->hostres = hostres;
+
+ /*
+ * We are called from ioctl of file that own this user_info,
+ * So it safe to assume it exist.
+ */
+ nnp_user_get(user_info);
+ hr_entry->user_info = user_info;
+
+ mutex_lock(&user_info->mutex);
+ /*
+ * We allocate handle starting from 1 and not 0 to allow
+ * user-space treat zero as invalid handle
+ */
+ id = idr_alloc(&user_info->idr, hr_entry, 1, -1, GFP_KERNEL);
+ if (id < 0) {
+ nnp_user_put(user_info);
+ nnp_hostres_put(hostres);
+ kfree(hr_entry);
+ mutex_unlock(&user_info->mutex);
+ return -ENOSPC;
+ }
+ hr_entry->user_handle = id;
+ list_add(&hr_entry->node, &user_info->hostres_list);
+ mutex_unlock(&user_info->mutex);
+
+ *user_hostres_entry = hr_entry;
+
+ return 0;
+}
+
+void nnp_user_remove_hostres_locked(struct user_hostres *hr_entry)
+{
+ struct nnp_user_info *user_info = hr_entry->user_info;
+
+ idr_remove(&user_info->idr, hr_entry->user_handle);
+ list_del(&hr_entry->node);
+
+ nnp_hostres_put(hr_entry->hostres);
+
+ kfree(hr_entry);
+ nnp_user_put(user_info);
+}
+
+void nnp_user_remove_hostres(struct user_hostres *hr_entry)
+{
+ struct nnp_user_info *user_info = hr_entry->user_info;
+
+ mutex_lock(&user_info->mutex);
+ nnp_user_remove_hostres_locked(hr_entry);
+ mutex_unlock(&user_info->mutex);
+}
+
+void nnp_user_destroy_all(struct nnp_user_info *user_info)
+{
+ struct user_hostres *user_hostres_entry;
+ DECLARE_COMPLETION_ONSTACK(completion);
+
+ mutex_lock(&user_info->mutex);
+
+ /* destroy all hostreses owned by the "user" */
+ while (!list_empty(&user_info->hostres_list)) {
+ user_hostres_entry = list_first_entry(&user_info->hostres_list,
+ struct user_hostres, node);
+ /*
+ * We can safely destroy this object without checking
+ * its refcount since we get here only after the host char-dev
+ * as well as all cmd_chan char-devs that may hold temporary
+ * reference to this object are already released.
+ */
+ nnp_user_remove_hostres_locked(user_hostres_entry);
+ }
+ mutex_unlock(&user_info->mutex);
+
+ /* wait for all channels and hostreses to be destroyed */
+ user_info->close_completion = &completion;
+ nnp_user_put(user_info);
+ wait_for_completion(&completion);
+}
diff --git a/drivers/misc/intel-nnpi/nnp_user.h b/drivers/misc/intel-nnpi/nnp_user.h
new file mode 100644
index 0000000..429ac13
--- /dev/null
+++ b/drivers/misc/intel-nnpi/nnp_user.h
@@ -0,0 +1,79 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright (C) 2019-2021 Intel Corporation */
+
+#ifndef _NNPDRV_INF_PROC_H
+#define _NNPDRV_INF_PROC_H
+
+#include <linux/kref.h>
+#include <linux/types.h>
+
+#include "hostres.h"
+
+/**
+ * struct nnp_user_info - structure for per-user info
+ * @ref: refcount to this "user" object
+ * @hostres_list: list of host resources
+ * @close_completion: used to wait for all channels of this user to be
+ * destroyed before closing the user.
+ * @mutex: protects hostres_list and idr modifications
+ * @idr: used to generate user handles to created host resources
+ * @user_list_node: list node to attach this struct in "list of users".
+ *
+ * structure to hold per-user info,
+ * a "user" is created for each open made to the host char dev (/dev/nnpi_host).
+ * It holds a list of all host resources created through requests from
+ * the same client ("user").
+ * device communication "channels", created by device char dev (/dev/nnpi%d)
+ * must be correlated with a "user" object which is supplied from user-space
+ * by the opened file descriptor to /dev/nnpi_host. Such "channel" may access
+ * only host resources created by the same "user".
+ * The lifetime of this object last at least for the duration of the host char
+ * device file struct but can last longer if some channel objects still hold
+ * a reference to it (this is why @ref is needed).
+ */
+struct nnp_user_info {
+ struct kref ref;
+ struct list_head hostres_list;
+ struct completion *close_completion;
+ struct mutex mutex;
+ struct idr idr;
+ struct list_head user_list_node;
+};
+
+/**
+ * struct user_hostres - structure for host resource created by user
+ * @node: list node to attach this struct to nnp_user_info::hostres_list
+ * @hostres: the actual host resource object
+ * @user_handle: handle allocated from idr object, used as handle to this
+ * object in ioctl ABI.
+ * @user_info: pointer to "user" which created this resource.
+ * it is used only during destruction of the object.
+ *
+ * structure for a host resource object which created through host char dev
+ * request. The lifetime of this structure ends when the user request to
+ * destroy it through ioctl call. The underlying @hostres may still continue
+ * to exist if command channel (cmd_chan) objects has mapped the resource to
+ * device access.
+ */
+struct user_hostres {
+ struct list_head node;
+ struct host_resource *hostres;
+ int user_handle;
+ struct nnp_user_info *user_info;
+};
+
+void nnp_user_init(struct nnp_user_info *user_info);
+
+void nnp_user_get(struct nnp_user_info *user_info);
+void nnp_user_put(struct nnp_user_info *user_info);
+
+int nnp_user_add_hostres(struct nnp_user_info *user_info,
+ struct host_resource *hostres,
+ struct user_hostres **user_hostres_entry);
+
+void nnp_user_remove_hostres(struct user_hostres *hr_entry);
+void nnp_user_remove_hostres_locked(struct user_hostres *hr_entry);
+
+void nnp_user_destroy_all(struct nnp_user_info *user_info);
+
+#endif
diff --git a/include/uapi/misc/intel_nnpi.h b/include/uapi/misc/intel_nnpi.h
new file mode 100644
index 0000000..5114aea
--- /dev/null
+++ b/include/uapi/misc/intel_nnpi.h
@@ -0,0 +1,150 @@
+/* SPDX-License-Identifier: GPL-2.0-only WITH Linux-syscall-note */
+/* Copyright (C) 2019-2021 Intel Corporation */
+
+#ifndef _NNP_UAPI_H
+#define _NNP_UAPI_H
+
+#include <linux/types.h>
+#include <linux/ioctl.h>
+#include <stdbool.h>
+
+#define NNPDRV_INF_HOST_DEV_NAME "nnpi_host"
+
+/*
+ * ioctls for /dev/nnpi_host device
+ */
+
+/*
+ * IOCTL_INF_CREATE_HOST_RESOURCE:
+ *
+ * A request to create a host memory resource object that can then be mapped
+ * and accessed by the NNP-I device's DMA engine.
+ * The created host resource is pinned in memory for its entire lifecycle.
+ * The memory of the resource is backed by user allocated memory which
+ * get pinned by the IOCTL.
+ *
+ * See description of nnpdrv_ioctl_create_hostres structure for more details.
+ *
+ * The ioctl returns a handle to the created host resource.
+ */
+#define IOCTL_INF_CREATE_HOST_RESOURCE \
+ _IOWR('h', 0, struct nnpdrv_ioctl_create_hostres)
+
+/*
+ * IOCTL_INF_DESTROY_HOST_RESOURCE:
+ *
+ * A request to destoy a host resource object.
+ */
+#define IOCTL_INF_DESTROY_HOST_RESOURCE \
+ _IOWR('h', 2, struct nnpdrv_ioctl_destroy_hostres)
+
+/*
+ * IOCTL_INF_LOCK_HOST_RESOURCE:
+ *
+ * A request to lock a host resource for cpu access for either
+ * read or write.
+ *
+ * This IOCTL does *not* synchronize accessed to host memory between host
+ * cpu and the device's DMA engine. It is used only for either flush or
+ * invalidate cpu caches to let the device see the last writes made from
+ * host cpu and let cpu read up-to-date content of the resource after the
+ * device changed it.
+ *
+ * This synchronization is not required on all platforms, when mapping
+ * the resource for device access, using IOCTL_NNPI_DEVICE_CHANNEL_MAP_HOSTRES,
+ * the application receive an indication if such synchronization is needed
+ * or not with that device.
+ *
+ * When such synchronization is needed:
+ * When application wants to change host resource content to be read by the
+ * device, it should first lock it for write, change its content by accessing
+ * it's mapped virtual address and then call this ioctl again to unlock it
+ * before sending a command to the device which may read the resource.
+ * When the application received indication that the device has changed the
+ * resource content, it should first lock the resource for reading before
+ * accessing its memory.
+ */
+#define IOCTL_INF_LOCK_HOST_RESOURCE \
+ _IOWR('h', 3, struct nnpdrv_ioctl_lock_hostres)
+
+/*
+ * IOCTL_INF_UNLOCK_HOST_RESOURCE:
+ *
+ * A request to unlock a host resource that was previously locked for cpu access.
+ */
+#define IOCTL_INF_UNLOCK_HOST_RESOURCE \
+ _IOWR('h', 4, struct nnpdrv_ioctl_lock_hostres)
+
+/*
+ * The below are possible bit masks that can be specified in
+ * usage_flags field of struct nnpdrv_ioctl_create_hostres.
+ * It specify attribute and usage flags for a host resource.
+ */
+#define IOCTL_INF_RES_INPUT (1u << 0) /* being read by the NNP-I device */
+#define IOCTL_INF_RES_OUTPUT (1u << 1) /* being written by the device */
+#define IOCTL_RES_USAGE_VALID_MASK (IOCTL_INF_RES_INPUT | IOCTL_INF_RES_OUTPUT)
+
+/**
+ * struct nnpdrv_ioctl_create_hostres - IOCTL_INF_CREATE_HOST_RESOURCE payload
+ * @user_ptr: User virtual address.
+ * @size: User memory size on input. Host resource size on output.
+ * @usage_flags: resource usage flag bits, IOCTL_INF_RES_*
+ * @user_handle: resource handle on output.
+ *
+ * argument structure for IOCTL_INF_CREATE_HOST_RESOURCE ioctl
+ *
+ * @user_ptr should be initialized to a user virtual address and @size
+ * should be initialized with it's size, the user memory will be pinned and will
+ * hold the host resource content.
+ *
+ * On output, @user_handle is a handle to the created host resource that can be
+ * used later with other IOCTLs and @size is the size of the host resource.
+ */
+struct nnpdrv_ioctl_create_hostres {
+ __u64 user_ptr;
+ __u64 size;
+ __u32 usage_flags;
+ __s32 user_handle;
+};
+
+/**
+ * struct nnpdrv_ioctl_lock_hostres - IOCTL_INF_LOCK_HOST_RESOURCE payload
+ * @user_handle: handle to host resource object
+ * @o_errno: On input, must be set to 0.
+ * On output, 0 on success, one of the NNPERR_* error codes on error.
+ *
+ * argument structure for IOCTL_INF_LOCK_HOST_RESOURCE and
+ * IOCTL_INF_LOCK_HOST_RESOURCE ioctl calls.
+ */
+struct nnpdrv_ioctl_lock_hostres {
+ __s32 user_handle;
+ __u32 o_errno;
+};
+
+/**
+ * struct nnpdrv_ioctl_destroy_hostres - IOCTL_INF_DESTROY_HOST_RESOURCE payload
+ * @user_handle: handle to host resource object
+ * @o_errno: On input, must be set to 0.
+ * On output, 0 on success, one of the NNPERR_* error codes on error.
+ *
+ * argument structure for IOCTL_INF_DESTROY_HOST_RESOURCE ioctl
+ */
+struct nnpdrv_ioctl_destroy_hostres {
+ __s32 user_handle;
+ __u32 o_errno;
+};
+
+/****************************************************************
+ * Error code values - errors returned in o_errno fields of
+ * above structures.
+ ****************************************************************/
+#define NNP_ERRNO_BASE 200
+#define NNPER_DEVICE_NOT_READY (NNP_ERRNO_BASE + 1)
+#define NNPER_NO_SUCH_RESOURCE (NNP_ERRNO_BASE + 2)
+#define NNPER_INCOMPATIBLE_RESOURCES (NNP_ERRNO_BASE + 3)
+#define NNPER_DEVICE_ERROR (NNP_ERRNO_BASE + 4)
+#define NNPER_NO_SUCH_CHANNEL (NNP_ERRNO_BASE + 5)
+#define NNPER_NO_SUCH_HOSTRES_MAP (NNP_ERRNO_BASE + 6)
+#define NNPER_VERSIONS_MISMATCH (NNP_ERRNO_BASE + 7)
+
+#endif /* of _NNP_UAPI_H */
--
1.8.3.1
---------------------------------------------------------------------
Intel Israel (74) Limited
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
Provide interface for creating a "host resource" - a memory object
which can be mapped to a dma address space of one or more NNP-I
devices. These host resource objects manage memory blocks from which
big chunks of data, such as inference tensors, the device's embedded OS
and telemtry data are DMA'ed to and from the NNP-I device.
Signed-off-by: Guy Zadicario <[email protected]>
Reviewed-by: Alexander Shishkin <[email protected]>
---
Documentation/ABI/testing/sysfs-driver-intel_nnpi | 5 +
drivers/misc/intel-nnpi/Makefile | 2 +-
drivers/misc/intel-nnpi/hostres.c | 627 ++++++++++++++++++++++
drivers/misc/intel-nnpi/hostres.h | 167 ++++++
4 files changed, 800 insertions(+), 1 deletion(-)
create mode 100644 Documentation/ABI/testing/sysfs-driver-intel_nnpi
create mode 100644 drivers/misc/intel-nnpi/hostres.c
create mode 100644 drivers/misc/intel-nnpi/hostres.h
diff --git a/Documentation/ABI/testing/sysfs-driver-intel_nnpi b/Documentation/ABI/testing/sysfs-driver-intel_nnpi
new file mode 100644
index 0000000..b09880f
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-driver-intel_nnpi
@@ -0,0 +1,5 @@
+What: /sys/class/nnpi_host/nnpi_host/total_hostres_size
+Date: Mar 2021
+Kernelversion: 5.13
+Contact: [email protected]
+Description: Total size in bytes of all allocated NNP-I host resources.
diff --git a/drivers/misc/intel-nnpi/Makefile b/drivers/misc/intel-nnpi/Makefile
index 43f09c0..3713d51 100644
--- a/drivers/misc/intel-nnpi/Makefile
+++ b/drivers/misc/intel-nnpi/Makefile
@@ -5,7 +5,7 @@
obj-$(CONFIG_INTEL_NNPI) := intel_nnpi.o intel_nnpi_pcie.o
-intel_nnpi-y := device.o msg_scheduler.o
+intel_nnpi-y := device.o msg_scheduler.o hostres.o
intel_nnpi_pcie-y := nnp_pcie.o
diff --git a/drivers/misc/intel-nnpi/hostres.c b/drivers/misc/intel-nnpi/hostres.c
new file mode 100644
index 0000000..e0d71c0
--- /dev/null
+++ b/drivers/misc/intel-nnpi/hostres.c
@@ -0,0 +1,627 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (C) 2019-2021 Intel Corporation */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/bitfield.h>
+#include <linux/err.h>
+#include <linux/mutex.h>
+#include <linux/pagemap.h>
+#include <linux/pfn.h>
+#include <linux/printk.h>
+#include <linux/sched/mm.h>
+#include <linux/slab.h>
+#include <linux/vmalloc.h>
+
+#include "hostres.h"
+#include "ipc_protocol.h"
+
+/**
+ * struct host_resource - structure for host memory resource object
+ * @ref: kref for that host resource object
+ * @size: size of the memory resource, in bytes
+ * @devices: list of devices this resource is mapped to (list of nnpdev_mapping)
+ * @lock: protects @devices
+ * @dir: DMA direction mask possible for this resource, when mapped to device.
+ * @pinned_mm: mm object used to pin the user allocated resource memory. NULL
+ * if the resource was not allocated by user-space.
+ * @vptr: virtual pointer to the resource memory if allocated by
+ * nnp_hostres_alloc(). NULL otherwise.
+ * @start_offset: holds the offset within the first pinned page where resource
+ * memory starts (relevant only when @pinned_mm is not NULL).
+ * @pages: array of resource memory pages.
+ * @n_pages: size of pages array.
+ */
+struct host_resource {
+ struct kref ref;
+ size_t size;
+ struct list_head devices;
+ spinlock_t lock;
+ enum dma_data_direction dir;
+
+ struct mm_struct *pinned_mm;
+ void *vptr;
+ unsigned int start_offset;
+
+ struct page **pages;
+ unsigned int n_pages;
+};
+
+/**
+ * struct nnpdev_mapping - mapping information of host resource to one device
+ * @ref: kref for that mapping object
+ * @res: pointer to the host resource
+ * @dev: the device the resource is mapped to
+ * @sgt: scatter table of host resource pages in memory
+ * @dma_chain_sgt: sg_table of dma_chain blocks (see description below).
+ * @dma_chain_order: order used to allocate scatterlist of @dma_chain_sgt.
+ * @node: list head to attach this object to a list of mappings
+ *
+ * This structure holds mapping information of one host resource to one
+ * NNP-I device. @sgt is the sg_table describes the DMA addresses of the
+ * resource chunks.
+ *
+ * When mapping a host memory resource for NNP-I device access, we need to send
+ * the DMA page table of the resource to the device. The device uses this page
+ * table when programming its DMA engine to read/write the host resource.
+ *
+ * The format of that page table is a chain of continuous DMA buffers, each
+ * starts with a 24 bytes header (&struct dma_chain_header) followed by 8 bytes
+ * entries, each describe a continuous block of the resource
+ * (&struct nnp_dma_chain_entry).
+ *
+ * The header of the chain has a pointer to the next buffer in the chain for
+ * the case where multiple DMA blocks are required to describe the
+ * entire resource. The address of the first block in the chain is sent to
+ * the device, which then fetches the entire chain when the resource is
+ * mapped. @dma_chain_sgt is an sg_table of memory mapped to the device and
+ * initialized with the resource page table in the above described format.
+ */
+struct nnpdev_mapping {
+ struct kref ref;
+ struct host_resource *res;
+ struct device *dev;
+ struct sg_table *sgt;
+ struct sg_table dma_chain_sgt;
+ unsigned int dma_chain_order;
+ struct list_head node;
+};
+
+/*
+ * Since host resources are pinned for their entire lifetime, it
+ * is useful to monitor the total size of NNP-I host resources
+ * allocated in the system.
+ */
+static size_t total_hostres_size;
+static DEFINE_MUTEX(total_size_mutex);
+
+/* Destroys host resource, when all references to it are released */
+static void release_hostres(struct kref *kref)
+{
+ struct host_resource *r = container_of(kref, struct host_resource, ref);
+
+ if (r->pinned_mm) {
+ unpin_user_pages(r->pages, r->n_pages);
+ account_locked_vm(r->pinned_mm, r->n_pages, false);
+ mmdrop(r->pinned_mm);
+ } else {
+ vfree(r->vptr);
+ }
+
+ kvfree(r->pages);
+ mutex_lock(&total_size_mutex);
+ total_hostres_size -= r->size;
+ mutex_unlock(&total_size_mutex);
+ kfree(r);
+}
+
+void nnp_hostres_get(struct host_resource *res)
+{
+ kref_get(&res->ref);
+};
+
+void nnp_hostres_put(struct host_resource *res)
+{
+ kref_put(&res->ref, release_hostres);
+}
+
+/* Really destroys mapping to device, when refcount is zero */
+static void release_mapping(struct kref *kref)
+{
+ struct nnpdev_mapping *m = container_of(kref, struct nnpdev_mapping, ref);
+
+ spin_lock(&m->res->lock);
+ list_del(&m->node);
+ spin_unlock(&m->res->lock);
+
+ dma_unmap_sgtable(m->dev, &m->dma_chain_sgt, DMA_TO_DEVICE, 0);
+ sgl_free_order(m->dma_chain_sgt.sgl, m->dma_chain_order);
+
+ dma_unmap_sg(m->dev, m->sgt->sgl, m->sgt->orig_nents, m->res->dir);
+ sg_free_table(m->sgt);
+ kfree(m->sgt);
+
+ nnp_hostres_put(m->res);
+
+ kfree(m);
+}
+
+static struct host_resource *alloc_hostres(size_t size, enum dma_data_direction dir)
+{
+ struct host_resource *r;
+
+ r = kzalloc(sizeof(*r), GFP_KERNEL);
+ if (!r)
+ return r;
+
+ kref_init(&r->ref);
+ spin_lock_init(&r->lock);
+ r->dir = dir;
+ r->size = size;
+ INIT_LIST_HEAD(&r->devices);
+
+ return r;
+}
+
+struct host_resource *nnp_hostres_alloc(size_t size, enum dma_data_direction dir)
+{
+ struct host_resource *r;
+ unsigned int i;
+ char *p;
+
+ if (size == 0 || dir == DMA_NONE)
+ return ERR_PTR(-EINVAL);
+
+ r = alloc_hostres(size, dir);
+ if (!r)
+ return ERR_PTR(-ENOMEM);
+
+ r->n_pages = PFN_UP(size);
+ r->vptr = vmalloc(r->n_pages * PAGE_SIZE);
+ if (!r->vptr)
+ goto free_res;
+
+ r->pages = kvmalloc_array(r->n_pages, sizeof(struct page *), GFP_KERNEL);
+ if (!r->pages)
+ goto free_vptr;
+
+ for (i = 0, p = r->vptr; i < r->n_pages; i++, p += PAGE_SIZE) {
+ r->pages[i] = vmalloc_to_page(p);
+ if (!r->pages[i])
+ goto free_pages;
+ }
+
+ mutex_lock(&total_size_mutex);
+ total_hostres_size += size;
+ mutex_unlock(&total_size_mutex);
+
+ return r;
+
+free_pages:
+ kvfree(r->pages);
+free_vptr:
+ vfree(r->vptr);
+free_res:
+ kfree(r);
+ return ERR_PTR(-ENOMEM);
+}
+
+struct host_resource *nnp_hostres_from_usermem(void __user *user_ptr, size_t size,
+ enum dma_data_direction dir)
+{
+ /*
+ * user_ptr is not being accessed, it is only used as parameter to
+ * pin_user_pages(), so it is OK to remove the __user annotation.
+ */
+ uintptr_t user_addr = (__force uintptr_t)user_ptr;
+ struct host_resource *r;
+ int err;
+ int gup_flags = 0;
+ int n, pinned;
+
+ if (size == 0 || dir == DMA_NONE)
+ return ERR_PTR(-EINVAL);
+
+ /* Restrict for 4 byte alignment */
+ if (user_addr & 0x3)
+ return ERR_PTR(-EINVAL);
+
+ if (!access_ok(user_ptr, size))
+ return ERR_PTR(-EFAULT);
+
+ r = alloc_hostres(size, dir);
+ if (!r)
+ return ERR_PTR(-ENOMEM);
+
+ r->start_offset = offset_in_page(user_addr);
+ user_addr &= PAGE_MASK;
+
+ r->n_pages = PFN_UP(size + r->start_offset);
+ r->pages = kvmalloc_array(r->n_pages, sizeof(struct page *), GFP_KERNEL);
+ if (!r->pages) {
+ err = -ENOMEM;
+ goto free_res;
+ }
+
+ err = account_locked_vm(current->mm, r->n_pages, true);
+ if (err)
+ goto free_pages;
+
+ if (nnp_hostres_is_input(r))
+ gup_flags = FOLL_WRITE;
+
+ /*
+ * The host resource is being re-used for multiple DMA
+ * transfers for streaming data into the device.
+ * In most situations will live long term.
+ */
+ gup_flags |= FOLL_LONGTERM;
+
+ for (pinned = 0; pinned < r->n_pages; pinned += n) {
+ n = pin_user_pages(user_addr + pinned * PAGE_SIZE,
+ r->n_pages - pinned, gup_flags,
+ &r->pages[pinned], NULL);
+ if (n < 0) {
+ err = -ENOMEM;
+ goto unaccount;
+ }
+ }
+
+ r->pinned_mm = current->mm;
+ mmgrab(r->pinned_mm);
+
+ mutex_lock(&total_size_mutex);
+ total_hostres_size += size;
+ mutex_unlock(&total_size_mutex);
+
+ return r;
+
+unaccount:
+ account_locked_vm(current->mm, r->n_pages, false);
+ unpin_user_pages(r->pages, pinned);
+free_pages:
+ kvfree(r->pages);
+free_res:
+ kfree(r);
+ return ERR_PTR(err);
+}
+
+/* Finds mapping by device and increase its refcount. NULL if not found */
+static struct nnpdev_mapping *get_mapping_for_dev(struct host_resource *res,
+ struct device *dev)
+{
+ struct nnpdev_mapping *m;
+
+ spin_lock(&res->lock);
+
+ list_for_each_entry(m, &res->devices, node) {
+ if (m->dev == dev) {
+ kref_get(&m->ref);
+ goto out;
+ }
+ }
+
+ m = NULL;
+out:
+ spin_unlock(&res->lock);
+ return m;
+}
+
+static bool entry_valid(struct scatterlist *sgl, u64 ipc_entry)
+{
+ unsigned long long dma_pfn;
+ unsigned long n_pages;
+
+ dma_pfn = FIELD_GET(DMA_CHAIN_ENTRY_PFN_MASK, ipc_entry);
+ if (NNP_IPC_DMA_PFN_TO_ADDR(dma_pfn) != sg_dma_address(sgl))
+ return false;
+
+ n_pages = FIELD_GET(DMA_CHAIN_ENTRY_NPAGES_MASK, ipc_entry);
+ if (n_pages != DIV_ROUND_UP(sg_dma_len(sgl), NNP_PAGE_SIZE))
+ return false;
+
+ return true;
+}
+
+/**
+ * build_ipc_dma_chain_array() - builds page list of the resource for IPC usage
+ * @m: pointer to device mapping info struct
+ * @use_one_entry: if true will generate all page table in one continuous
+ * DMA chunk. otherwise a chain of blocks will be used
+ * each of one page size.
+ * @start_offset: offset in first mapped page where resource memory starts,
+ *
+ * This function allocates scatterlist, map it to device and populate it with
+ * page table of the device mapped resource in format suitable to be used
+ * in the IPC protocol for sending the resource page table to the card.
+ * The format of the page table is described in the documentation of &struct
+ * nnpdev_mapping.
+ *
+ * Return: 0 on success, error value otherwise
+ */
+static int build_ipc_dma_chain_array(struct nnpdev_mapping *m, bool use_one_entry,
+ unsigned int start_offset)
+{
+ unsigned int i, k = 0;
+ u64 *p = NULL;
+ u64 e;
+ unsigned long long dma_addr, dma_pfn, size;
+ struct nnp_dma_chain_header *h;
+ struct scatterlist *sg, *map_sg;
+ struct scatterlist *chain_sg;
+ unsigned long n_pages;
+ unsigned int chain_size;
+ unsigned int chain_order;
+ unsigned int chain_nents;
+ unsigned int nents_per_entry;
+ unsigned int start_off = start_offset;
+ int rc;
+
+ if (use_one_entry) {
+ /*
+ * Allocate enough pages in one chunk that will fit
+ * the header and nnp_dma_chain_entry for all the sg_table
+ * entries.
+ */
+ nents_per_entry = m->sgt->nents;
+ chain_size = sizeof(struct nnp_dma_chain_header) +
+ m->sgt->nents * DMA_CHAIN_ENTRY_SIZE;
+ chain_order = get_order(chain_size);
+ } else {
+ /*
+ * Calc number of one page DMA buffers needed to hold the
+ * entire page table.
+ * NENTS_PER_PAGE is how much DMA chain entries fits
+ * in a single page following the chain header, must be at
+ * positive.
+ */
+ nents_per_entry = NENTS_PER_PAGE;
+ chain_size = DIV_ROUND_UP(m->sgt->nents, nents_per_entry) *
+ NNP_PAGE_SIZE;
+ chain_order = 0;
+ }
+
+ chain_sg = sgl_alloc_order(chain_size, chain_order, false, GFP_KERNEL,
+ &chain_nents);
+ if (!chain_sg)
+ return -ENOMEM;
+
+ m->dma_chain_sgt.sgl = chain_sg;
+ m->dma_chain_sgt.nents = chain_nents;
+ m->dma_chain_sgt.orig_nents = chain_nents;
+ m->dma_chain_order = chain_order;
+ rc = dma_map_sgtable(m->dev, &m->dma_chain_sgt, DMA_TO_DEVICE, 0);
+ if (rc)
+ goto free_chain_sg;
+
+ /* Initialize chain entry blocks */
+ map_sg = m->sgt->sgl;
+ for_each_sg(chain_sg, sg, chain_nents, i) {
+ /*
+ * Check that the allocated DMA address fits in IPC protocol.
+ * In the protocol, DMA addresses are sent as 4K page numbers
+ * and must fit in 45 bits.
+ * Meaning, if the DMA address is larger than 57 bits it will
+ * not fit.
+ */
+ if (sg_dma_address(sg) > NNP_IPC_DMA_MAX_ADDR)
+ goto unmap_chain_sg;
+
+ /* h: points to the header of current block */
+ h = sg_virt(sg);
+
+ /* p: points to current chunk entry in block */
+ p = (u64 *)(h + 1);
+
+ size = 0;
+ for (k = 0; k < nents_per_entry && map_sg; ++k) {
+ /*
+ * Build entry with DMA address as page number and
+ * size in pages
+ */
+ dma_addr = sg_dma_address(map_sg);
+ dma_pfn = NNP_IPC_DMA_ADDR_TO_PFN(dma_addr);
+ n_pages = DIV_ROUND_UP(sg_dma_len(map_sg), NNP_PAGE_SIZE);
+
+ e = FIELD_PREP(DMA_CHAIN_ENTRY_PFN_MASK, dma_pfn);
+ e |= FIELD_PREP(DMA_CHAIN_ENTRY_NPAGES_MASK, n_pages);
+
+ /*
+ * Check that packed entry matches the DMA chunk.
+ * (Will fail if either dma_pfn or n_pages fields overflows)
+ */
+ if (!entry_valid(map_sg, e))
+ goto unmap_chain_sg;
+
+ /* Fill entry value (should be 64-bit little-endian) */
+ p[k] = cpu_to_le64(e);
+
+ size += sg_dma_len(map_sg);
+
+ map_sg = sg_next(map_sg);
+ }
+
+ /* Initialize block header and link to next block */
+ h->total_nents = cpu_to_le32(m->sgt->nents);
+ h->start_offset = cpu_to_le32(start_off);
+ h->size = cpu_to_le64(size);
+ if (sg_next(sg))
+ h->dma_next = cpu_to_le64(sg_dma_address(sg_next(sg)));
+ else
+ h->dma_next = 0;
+ start_off = 0;
+ }
+
+ return 0;
+
+unmap_chain_sg:
+ dma_unmap_sgtable(m->dev, &m->dma_chain_sgt, DMA_TO_DEVICE, 0);
+free_chain_sg:
+ sgl_free_order(chain_sg, chain_order);
+ memset(&m->dma_chain_sgt, 0, sizeof(m->dma_chain_sgt));
+ return -ENOMEM;
+}
+
+struct nnpdev_mapping *nnp_hostres_map_device(struct host_resource *res,
+ struct nnp_device *nnpdev,
+ bool use_one_entry,
+ dma_addr_t *page_list,
+ u32 *total_chunks)
+{
+ int ret;
+ struct nnpdev_mapping *m;
+ struct scatterlist *sge;
+
+ if (!res || !nnpdev || !page_list)
+ return ERR_PTR(-EINVAL);
+
+ /* Check if already mapped for the device */
+ m = get_mapping_for_dev(res, nnpdev->dev);
+ if (m)
+ goto done;
+
+ nnp_hostres_get(res);
+
+ m = kmalloc(sizeof(*m), GFP_KERNEL);
+ if (!m) {
+ ret = -ENOMEM;
+ goto put_resource;
+ }
+
+ kref_init(&m->ref);
+
+ m->dev = nnpdev->dev;
+ m->res = res;
+
+ m->sgt = kmalloc(sizeof(*m->sgt), GFP_KERNEL);
+ if (!m->sgt) {
+ ret = -ENOMEM;
+ goto free_mapping;
+ }
+
+ sge = __sg_alloc_table_from_pages(m->sgt, res->pages, res->n_pages, 0,
+ res->size + res->start_offset,
+ NNP_MAX_CHUNK_SIZE, NULL, 0, GFP_KERNEL);
+ if (IS_ERR(sge)) {
+ ret = PTR_ERR(sge);
+ goto free_sgt_struct;
+ }
+
+ ret = dma_map_sg(m->dev, m->sgt->sgl, m->sgt->orig_nents, res->dir);
+ if (ret <= 0) {
+ /* dma_map_sg returns 0 on error with no error value */
+ ret = -ENOMEM;
+ goto free_sgt;
+ }
+
+ m->sgt->nents = ret;
+
+ ret = build_ipc_dma_chain_array(m, use_one_entry, res->start_offset);
+ if (ret < 0)
+ goto unmap;
+
+ spin_lock(&res->lock);
+ list_add(&m->node, &res->devices);
+ spin_unlock(&res->lock);
+
+done:
+ *page_list = sg_dma_address(m->dma_chain_sgt.sgl);
+ if (total_chunks)
+ *total_chunks = m->sgt->nents;
+
+ return m;
+
+unmap:
+ dma_unmap_sg(m->dev, m->sgt->sgl, m->sgt->orig_nents, res->dir);
+free_sgt:
+ sg_free_table(m->sgt);
+free_sgt_struct:
+ kfree(m->sgt);
+free_mapping:
+ kfree(m);
+put_resource:
+ nnp_hostres_put(res);
+ return ERR_PTR(ret);
+}
+
+void nnp_hostres_unmap_device(struct nnpdev_mapping *mapping)
+{
+ kref_put(&mapping->ref, release_mapping);
+}
+
+int nnp_hostres_user_lock(struct host_resource *res)
+{
+ struct nnpdev_mapping *m;
+
+ spin_lock(&res->lock);
+ list_for_each_entry(m, &res->devices, node)
+ dma_sync_sg_for_cpu(m->dev, m->sgt->sgl, m->sgt->orig_nents, res->dir);
+ spin_unlock(&res->lock);
+
+ return 0;
+}
+
+int nnp_hostres_user_unlock(struct host_resource *res)
+{
+ struct nnpdev_mapping *m;
+
+ spin_lock(&res->lock);
+ list_for_each_entry(m, &res->devices, node)
+ dma_sync_sg_for_device(m->dev, m->sgt->sgl, m->sgt->orig_nents,
+ res->dir);
+ spin_unlock(&res->lock);
+
+ return 0;
+}
+
+bool nnp_hostres_is_input(struct host_resource *res)
+{
+ return res->dir == DMA_TO_DEVICE || res->dir == DMA_BIDIRECTIONAL;
+}
+
+bool nnp_hostres_is_output(struct host_resource *res)
+{
+ return res->dir == DMA_FROM_DEVICE || res->dir == DMA_BIDIRECTIONAL;
+}
+
+size_t nnp_hostres_size(struct host_resource *res)
+{
+ return res->size;
+}
+
+void *nnp_hostres_vptr(struct host_resource *res)
+{
+ return res->vptr;
+}
+
+static ssize_t total_hostres_size_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ ssize_t ret;
+
+ mutex_lock(&total_size_mutex);
+ ret = sysfs_emit(buf, "%zu\n", total_hostres_size);
+ mutex_unlock(&total_size_mutex);
+
+ return ret;
+}
+static DEVICE_ATTR_RO(total_hostres_size);
+
+static struct attribute *nnp_host_attrs[] = {
+ &dev_attr_total_hostres_size.attr,
+ NULL
+};
+
+static struct attribute_group nnp_host_attrs_grp = {
+ .attrs = nnp_host_attrs,
+};
+
+int nnp_hostres_init_sysfs(struct device *dev)
+{
+ return sysfs_create_group(&dev->kobj, &nnp_host_attrs_grp);
+}
+
+void nnp_hostres_fini_sysfs(struct device *dev)
+{
+ sysfs_remove_group(&dev->kobj, &nnp_host_attrs_grp);
+}
diff --git a/drivers/misc/intel-nnpi/hostres.h b/drivers/misc/intel-nnpi/hostres.h
new file mode 100644
index 0000000..6397c73
--- /dev/null
+++ b/drivers/misc/intel-nnpi/hostres.h
@@ -0,0 +1,167 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright (C) 2019-2021 Intel Corporation */
+
+#ifndef _NNPDRV_HOSTRES_H
+#define _NNPDRV_HOSTRES_H
+
+#include <linux/dma-mapping.h>
+#include "device.h"
+
+/**
+ * nnp_hostres_alloc() - allocate memory and create host resource
+ * @size: Size of the host resource to be created
+ * @dir: Resource direction (read or write or both)
+ *
+ * This function allocates memory pages and provides host resource handle.
+ * The memory is mapped to kernel virtual address.
+ * The resource can be Input(read by device), Output(write by device) and both.
+ *
+ * The return handle can be used as argument to one of the other nnpi_hostres*
+ * functions for:
+ * - mapping/unmapping the resource for NNP-I device.
+ * - pointer to the allocated memory can be retrieved by nnp_hostres_vptr()
+ *
+ * The handle should be released when no longer needed by a call to
+ * nnp_hostres_put.
+ *
+ * Return: pointer to created resource or error value
+ */
+struct host_resource *nnp_hostres_alloc(size_t size, enum dma_data_direction dir);
+
+/**
+ * nnp_hostres_from_usermem() - Creates host resource from user-space memory
+ * @user_ptr: user virtual memory to pin
+ * @size: size of user buffer to pin
+ * @dir: Resource direction (read or write or both)
+ *
+ * This function pins the provided user memory and create a host resource
+ * handle managing this memory.
+ * The provided handle can be used the same as the handle created by
+ * nnp_hostres_alloc.
+ * The resource can be Input(read by device), Output(write by device) and both.
+ *
+ * The handle should be released when no longer needed by a call to
+ * nnp_hostres_put.
+ *
+ * Return: pointer to created resource or error value
+ */
+struct host_resource *nnp_hostres_from_usermem(void __user *user_ptr, size_t size,
+ enum dma_data_direction dir);
+
+/**
+ * nnp_hostres_map_device() - Maps the host resource to NNP-I device
+ * @res: handle to host resource
+ * @nnpdev: handle to nnp device struct
+ * @use_one_entry: when true will produce ipc dma chain page table descriptor
+ * of the mapping in a single concurrent dma block.
+ * otherwise a chain of multiple blocks might be generated.
+ * @page_list: returns the dma address of the ipc dma chain page table
+ * descriptor.
+ * @total_chunks: returns the total number of elements in the mapping's
+ * sg_table. Can be NULL if this info is not required.
+ *
+ * This function maps the host resource to be accessible from device
+ * and returns the dma page list of DMA addresses packed in format
+ * suitable to be used in IPC protocol to be sent to the card.
+ *
+ * The resource can be mapped to multiple devices.
+ *
+ * Return: pointer to the mapping object or error on failure.
+ */
+struct nnpdev_mapping *nnp_hostres_map_device(struct host_resource *res,
+ struct nnp_device *nnpdev,
+ bool use_one_entry,
+ dma_addr_t *page_list,
+ u32 *total_chunks);
+
+/**
+ * nnp_hostres_unmap_device() - Unmaps the host resource from NNP-I device
+ * @mapping: mapping pointer, returned from nnp_hostres_map_device
+ *
+ * This function unmaps previously mapped host resource from device.
+ */
+void nnp_hostres_unmap_device(struct nnpdev_mapping *mapping);
+
+/**
+ * nnp_hostres_user_lock() - Lock the host resource to access from userspace
+ * @res: handle to host resource
+ *
+ * This function should be called before user-space application is accessing
+ * the host resource content (either for read or write). The function
+ * invalidates or flashes the cpu caches when necessary.
+ * The function does *not* impose any synchronization between application and
+ * device accesses to the resource memory. Such synchronization is handled
+ * in user-space.
+ *
+ * Return: error on failure.
+ */
+int nnp_hostres_user_lock(struct host_resource *res);
+
+/**
+ * nnp_hostres_user_unlock() - Unlocks the host resource from userspace access
+ * @res: handle to host resource
+ *
+ * This function should be called after user-space application is finished
+ * accessing the host resource content (either for read or write). The function
+ * invalidates or flashes the cpu caches when necessary.
+ *
+ * Return: error on failure.
+ */
+int nnp_hostres_user_unlock(struct host_resource *res);
+
+/**
+ * nnp_hostres_get() - Increases refcount of the hostres
+ * @res: handle to host resource
+ *
+ * This function increases refcount of the host resource.
+ */
+void nnp_hostres_get(struct host_resource *res);
+
+/**
+ * nnp_hostres_put() - Decreases refcount of the hostres
+ * @res: handle to host resource
+ *
+ * This function decreases refcount of the host resource and destroys it
+ * when it reaches 0.
+ */
+void nnp_hostres_put(struct host_resource *res);
+
+/**
+ * nnp_hostres_is_input() - Returns if the host resource is input resource
+ * @res: handle to host resource
+ *
+ * This function returns true if the host resource can be read by device.
+ * The "input" terminology is used here since such resources are usually
+ * used as inputs to device inference network.
+ *
+ * Return: true if the reasource is readable.
+ */
+bool nnp_hostres_is_input(struct host_resource *res);
+
+/**
+ * nnp_hostres_is_output() - Returns if the host resource is output resource
+ * @res: handle to host resource
+ *
+ * This function returns true if the host resource can be modified by device.
+ * The term "output" is used here since usually such resources are used for
+ * outputs of device inference network.
+ *
+ * Return: true if the reasource is writable.
+ */
+bool nnp_hostres_is_output(struct host_resource *res);
+
+size_t nnp_hostres_size(struct host_resource *res);
+
+/**
+ * nnp_hostres_vptr() - returns the virtual pointer to the resource buffer
+ * @res: handle to host resource
+ *
+ * Return: pointer to resource data or NULL if was not allocated by
+ * nnp_hostres_alloc()
+ */
+void *nnp_hostres_vptr(struct host_resource *res);
+
+int nnp_hostres_init_sysfs(struct device *dev);
+void nnp_hostres_fini_sysfs(struct device *dev);
+
+#endif
--
1.8.3.1
---------------------------------------------------------------------
Intel Israel (74) Limited
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
Boot the NNP-I device after the card is powered-on or reset. When the
NNP-I card comes up, it's flashed BIOS starts running.
Since the persistent storage on the device is rather small, it needs to
get its OS boot image from the host driver. So once the card's BIOS is
running, the host driver loads and provides a boot image to the card BIOS,
which uses it to run the Embedded Linux image and SW stack on the card.
Signed-off-by: Guy Zadicario <[email protected]>
Reviewed-by: Alexander Shishkin <[email protected]>
---
drivers/misc/intel-nnpi/Makefile | 3 +-
drivers/misc/intel-nnpi/bootimage.c | 246 ++++++++++++++++++++++++++++++
drivers/misc/intel-nnpi/bootimage.h | 43 ++++++
drivers/misc/intel-nnpi/device.c | 291 ++++++++++++++++++++++++++++++++++++
drivers/misc/intel-nnpi/device.h | 77 ++++++++++
drivers/misc/intel-nnpi/nnp_pcie.c | 16 ++
6 files changed, 675 insertions(+), 1 deletion(-)
create mode 100644 drivers/misc/intel-nnpi/bootimage.c
create mode 100644 drivers/misc/intel-nnpi/bootimage.h
diff --git a/drivers/misc/intel-nnpi/Makefile b/drivers/misc/intel-nnpi/Makefile
index da2863b..e46c89f 100644
--- a/drivers/misc/intel-nnpi/Makefile
+++ b/drivers/misc/intel-nnpi/Makefile
@@ -5,7 +5,8 @@
obj-$(CONFIG_INTEL_NNPI) := intel_nnpi.o intel_nnpi_pcie.o
-intel_nnpi-y := device.o msg_scheduler.o hostres.o host_chardev.o nnp_user.o
+intel_nnpi-y := device.o msg_scheduler.o hostres.o host_chardev.o nnp_user.o \
+ bootimage.o
intel_nnpi_pcie-y := nnp_pcie.o
diff --git a/drivers/misc/intel-nnpi/bootimage.c b/drivers/misc/intel-nnpi/bootimage.c
new file mode 100644
index 0000000..38b5156
--- /dev/null
+++ b/drivers/misc/intel-nnpi/bootimage.c
@@ -0,0 +1,246 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (C) 2019-2021 Intel Corporation */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/bitfield.h>
+#include <linux/device.h>
+#include <linux/firmware.h>
+#include <linux/fs.h>
+#include <linux/namei.h>
+#include <linux/printk.h>
+
+#include "bootimage.h"
+#include "device.h"
+#include "hostres.h"
+#include "ipc_protocol.h"
+#include "nnp_boot_defs.h"
+
+#define MAX_IMAGE_NAME_LEN (NAME_MAX + 1)
+
+void nnpdev_boot_image_init(struct image_info *boot_image)
+{
+ boot_image->state = IMAGE_NONE;
+ boot_image->hostres = NULL;
+ mutex_init(&boot_image->mutex);
+}
+
+static int load_firmware(struct image_info *image_info)
+{
+ const struct firmware *fw;
+ struct nnp_device *nnpdev = container_of(image_info, struct nnp_device,
+ boot_image);
+ struct device *dev = nnpdev->dev;
+ struct kstat stat;
+ struct path path;
+ static const char *fname = "/lib/firmware/" NNP_FIRMWARE_NAME;
+ void *vptr;
+ int ret;
+
+ lockdep_assert_held(&image_info->mutex);
+
+ /*
+ * find image file size
+ *
+ * NOTE: we look for the file under a constant path "/lib/firmware"
+ * since it works and accepted on all platforms that NNP-I device
+ * can be installed.
+ * A better solution would be to look at the same paths that the
+ * firmware API will search however the firmware API does not
+ * export any function to do the search and there is no point
+ * duplicating it here.
+ */
+ ret = kern_path(fname, LOOKUP_FOLLOW, &path);
+ if (ret) {
+ pr_err("Could not find image under /lib/firmware\n");
+ return ret;
+ }
+
+ ret = vfs_getattr(&path, &stat, STATX_SIZE, 0);
+ path_put(&path);
+ if (ret) {
+ pr_err("Failed to get file size for %s error=%d\n", fname, ret);
+ return ret;
+ }
+
+ /* create host resource to hold the boot image content */
+ image_info->hostres = nnp_hostres_alloc(stat.size, DMA_TO_DEVICE);
+ if (IS_ERR(image_info->hostres))
+ return PTR_ERR(image_info->hostres);
+
+ vptr = nnp_hostres_vptr(image_info->hostres);
+
+ /*
+ * load the image into the host resource.
+ * We load directly to pre-allocated host resource memory
+ * in order to prevent caching of the boot image inside
+ * firmware API
+ */
+ ret = request_firmware_into_buf(&fw, NNP_FIRMWARE_NAME, dev, vptr,
+ stat.size);
+ if (ret) {
+ pr_err("failed to load firmware %s ret=%d\n", fname, ret);
+ nnp_hostres_put(image_info->hostres);
+ image_info->hostres = NULL;
+ return ret;
+ }
+
+ release_firmware(fw);
+ image_info->state = IMAGE_AVAILABLE;
+
+ return 0;
+}
+
+static void load_image_handler(struct work_struct *work)
+{
+ struct image_info *image_info = container_of(work, struct image_info,
+ work);
+ struct nnp_device *nnpdev = container_of(image_info, struct nnp_device,
+ boot_image);
+ dma_addr_t page_list_addr;
+ unsigned int total_chunks;
+ unsigned int image_size;
+ u64 cmd[3];
+ u32 val;
+ int ret;
+
+ mutex_lock(&image_info->mutex);
+
+ /* do not load if image load request has canceled */
+ if (image_info->state != IMAGE_REQUESTED) {
+ mutex_unlock(&image_info->mutex);
+ return;
+ }
+
+ /* load boot image from disk */
+ ret = load_firmware(image_info);
+ if (ret) {
+ image_info->state = IMAGE_LOAD_FAILED;
+ goto fail;
+ }
+
+ /* map image to the device */
+ image_info->hostres_map = nnp_hostres_map_device(image_info->hostres,
+ nnpdev, true,
+ &page_list_addr,
+ &total_chunks);
+ if (IS_ERR(image_info->hostres_map)) {
+ nnp_hostres_put(image_info->hostres);
+ image_info->hostres = NULL;
+ image_info->state = IMAGE_NONE;
+ goto fail;
+ }
+
+ mutex_unlock(&image_info->mutex);
+
+ image_size = (unsigned int)nnp_hostres_size(image_info->hostres);
+
+ /* image successfully mapped - send it to the device to boot */
+ dev_dbg(nnpdev->dev,
+ "Mapped boot image num_chunks=%u total_size=%u\n", total_chunks,
+ image_size);
+
+ /* write image address directly to the command Q */
+ cmd[0] = FIELD_PREP(NNP_H2C_BOOT_IMAGE_READY_QW0_OP_MASK,
+ NNP_IPC_H2C_OP_BIOS_PROTOCOL);
+ cmd[0] |= FIELD_PREP(NNP_H2C_BOOT_IMAGE_READY_QW0_TYPE_MASK,
+ NNP_IPC_H2C_TYPE_BOOT_IMAGE_READY);
+ cmd[0] |= FIELD_PREP(NNP_H2C_BOOT_IMAGE_READY_QW0_SIZE_MASK,
+ 2 * sizeof(u64));
+
+ cmd[1] = (u64)page_list_addr + sizeof(struct nnp_dma_chain_header);
+
+ cmd[2] = FIELD_PREP(NNP_H2C_BOOT_IMAGE_READY_QW2_DESC_SIZE_MASK,
+ total_chunks * sizeof(struct nnp_dma_chain_entry));
+ cmd[2] |= FIELD_PREP(NNP_H2C_BOOT_IMAGE_READY_QW2_IMAGE_SIZE_MASK,
+ image_size);
+
+ nnpdev->ops->cmdq_write_mesg(nnpdev, cmd, 3);
+ return;
+
+fail:
+ /* notify card that boot image cannot be loaded */
+ val = FIELD_PREP(NNP_HOST_ERROR_MASK,
+ NNP_HOST_ERROR_CANNOT_LOAD_IMAGE);
+ nnpdev->ops->set_host_doorbell_value(nnpdev, val);
+ mutex_unlock(&image_info->mutex);
+}
+
+/**
+ * nnpdev_load_boot_image() - load boot image and send it to device
+ * @nnpdev: the device requested the image
+ *
+ * This function starts the flow of loading a boot image and map it to the
+ * requesting device. It will launch a work to load the boot image.
+ * It is an error to call this function if boot image load for the same
+ * device is already in progress.
+ *
+ * Return:
+ * * 0 - boot image was successfully loaded, mapped and sent to the device.
+ * * -EINVAL - image load is already in progress
+ */
+int nnpdev_load_boot_image(struct nnp_device *nnpdev)
+{
+ struct image_info *image_info = &nnpdev->boot_image;
+
+ /* check if the image is already loaded or in progress */
+ mutex_lock(&image_info->mutex);
+ if (image_info->state != IMAGE_NONE) {
+ mutex_unlock(&image_info->mutex);
+ return -EINVAL;
+ }
+
+ /* initialize image load request */
+ image_info->state = IMAGE_REQUESTED;
+ mutex_unlock(&image_info->mutex);
+ INIT_WORK(&image_info->work, load_image_handler);
+
+ /* schedule work to load the image */
+ schedule_work(&image_info->work);
+
+ return 0;
+}
+
+/**
+ * nnpdev_unload_boot_image() - unmaps boot image for device
+ * @nnpdev: the device
+ *
+ * This function is called when the device no longer need the boot image
+ * in memory. either because it was already copied to the device or when
+ * the device is removed during the image load request is in progress.
+ * The function unmaps the device from the host resource.
+ *
+ * Return: error code or zero.
+ */
+int nnpdev_unload_boot_image(struct nnp_device *nnpdev)
+{
+ struct image_info *image_info = &nnpdev->boot_image;
+ int ret = 0;
+
+ mutex_lock(&image_info->mutex);
+ switch (image_info->state) {
+ case IMAGE_NONE:
+ ret = -EINVAL;
+ goto done;
+ case IMAGE_REQUESTED:
+ image_info->state = IMAGE_NONE;
+ mutex_unlock(&image_info->mutex);
+ cancel_work_sync(&image_info->work);
+ return 0;
+ case IMAGE_LOAD_FAILED:
+ case IMAGE_AVAILABLE:
+ break;
+ }
+
+ if (image_info->hostres) {
+ nnp_hostres_unmap_device(image_info->hostres_map);
+ nnp_hostres_put(image_info->hostres);
+ image_info->hostres = NULL;
+ }
+
+ image_info->state = IMAGE_NONE;
+
+done:
+ mutex_unlock(&image_info->mutex);
+ return ret;
+}
diff --git a/drivers/misc/intel-nnpi/bootimage.h b/drivers/misc/intel-nnpi/bootimage.h
new file mode 100644
index 0000000..c5efe46
--- /dev/null
+++ b/drivers/misc/intel-nnpi/bootimage.h
@@ -0,0 +1,43 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright (C) 2019-2021 Intel Corporation */
+
+#ifndef _NNPDRV_BOOTIMAGE_H
+#define _NNPDRV_BOOTIMAGE_H
+
+#include <linux/mutex.h>
+#include <linux/workqueue.h>
+
+struct nnp_device;
+
+enum image_state {
+ IMAGE_NONE = 0,
+ IMAGE_REQUESTED,
+ IMAGE_LOAD_FAILED,
+ IMAGE_AVAILABLE
+};
+
+/**
+ * struct image_info - describes a boot image object
+ * @work: handle for placing the image load in a workqueue
+ * @state: state indicating whether it is loaded or load failed
+ * @mutex: protects accesses to @state and @hostres
+ * @load_fail_err: zero or error code if @state is IMAGE_LOAD_FAILED.
+ * @hostres: host resource object allocated for the image content
+ * @hostres_map: mapping object of host resource to device
+ *
+ * This structure describe a request to load boot image from disk,
+ * there is one such structure for each device.
+ */
+struct image_info {
+ struct work_struct work;
+ enum image_state state;
+ struct mutex mutex;
+ struct host_resource *hostres;
+ struct nnpdev_mapping *hostres_map;
+};
+
+void nnpdev_boot_image_init(struct image_info *boot_image);
+int nnpdev_load_boot_image(struct nnp_device *nnpdev);
+int nnpdev_unload_boot_image(struct nnp_device *nnpdev);
+
+#endif /* _NNPDRV_BOOTIMAGE_H */
diff --git a/drivers/misc/intel-nnpi/device.c b/drivers/misc/intel-nnpi/device.c
index a3c6a1d..f4fc975 100644
--- a/drivers/misc/intel-nnpi/device.c
+++ b/drivers/misc/intel-nnpi/device.c
@@ -3,13 +3,18 @@
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+#include <linux/bitfield.h>
#include <linux/device.h>
#include <linux/idr.h>
#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/printk.h>
+#include "bootimage.h"
#include "device.h"
#include "host_chardev.h"
#include "msg_scheduler.h"
+#include "nnp_boot_defs.h"
static DEFINE_IDA(dev_ida);
@@ -18,6 +23,178 @@ bool nnpdev_no_devices(void)
return ida_is_empty(&dev_ida);
}
+static void send_sysinfo_request_to_bios(struct nnp_device *nnpdev)
+{
+ u64 cmd[3];
+
+ cmd[0] = FIELD_PREP(NNP_H2C_BIOS_SYS_INFO_REQ_QW0_OP_MASK,
+ NNP_IPC_H2C_OP_BIOS_PROTOCOL);
+ cmd[0] |= FIELD_PREP(NNP_H2C_BIOS_SYS_INFO_REQ_QW0_TYPE_MASK,
+ NNP_IPC_H2C_TYPE_SYSTEM_INFO_REQ);
+ cmd[0] |= FIELD_PREP(NNP_H2C_BIOS_SYS_INFO_REQ_QW0_SIZE_MASK,
+ 2 * sizeof(u64));
+
+ cmd[1] = (u64)nnpdev->bios_system_info_dma_addr;
+
+ cmd[2] = FIELD_PREP(NNP_H2C_BIOS_SYS_INFO_REQ_QW2_SIZE_MASK,
+ NNP_PAGE_SIZE);
+
+ nnpdev->ops->cmdq_flush(nnpdev);
+
+ nnpdev->ops->cmdq_write_mesg(nnpdev, cmd, 3);
+}
+
+/**
+ * build_bios_version_string() - builds printable string of bios version string
+ * @nnpdev: pointer to device structure
+ *
+ * Initializes nnpdev->bios_version_str with printable string of bios version
+ * from bios_system_info page.
+ */
+static void build_bios_version_string(struct nnp_device *nnpdev)
+{
+ unsigned int i;
+ __le16 *v;
+
+ if (!nnpdev->bios_system_info)
+ return;
+
+ /*
+ * The bios version string in the bios's system info page
+ * holds __le16 for each character in the version string.
+ * (see struct nnp_c2h_bios_version)
+ * Here we convert it to string of chars by taking only the
+ * LSB from each 16-bit character
+ */
+ v = (__le16 *)&nnpdev->bios_system_info->bios_ver;
+
+ /* check that bios version string is corrected null terminated */
+ if (nnpdev->bios_system_info->bios_ver.null_terminator != 0)
+ return;
+
+ for (i = 0; i < NNP_BIOS_VERSION_LEN - 1 && v[i] != 0; ++i)
+ nnpdev->bios_version_str[i] = v[i];
+
+ nnpdev->bios_version_str[i] = '\0';
+}
+
+static int unload_boot_image(struct nnp_device *nnpdev)
+{
+ nnpdev->boot_image_loaded = false;
+ return nnpdev_unload_boot_image(nnpdev);
+}
+
+/**
+ * nnpdev_set_boot_state() - sets new device state.
+ * @nnpdev: pointer to device structure
+ * @mask: mask of device state bits defined in device.h
+ *
+ * This function sets new device status and handles the state machine of
+ * device boot flow.
+ * It is being called when various device notifications are received or
+ * some error conditions are detected.
+ *
+ * The following flow describes the communication flow with the NNP-I card's
+ * BIOS during the device boot flow, this function gets called when device
+ * state changes when progressing in this flow:
+ * 1) The device report its boot state through the "card doorbell" register,
+ * that signals an interrupt to the host and the "pci" layer in the driver
+ * calls the nnpdev_card_doorbell_value_changed function.
+ * 2) When the device signals that it is "Ready to boot", the host driver
+ * sends it through the "command queue" an address of page in host memory.
+ * 3) The card BIOS fills the page of memory with card system info and change
+ * the doorbell value to "sysinfo ready"
+ * 4) The host driver then initiate the boot image loading.
+ * 5) When boot image is ready in memory, the host driver send a
+ * "Boot image ready" message and the card BIOS starts booting and changes
+ * the doorbell value to indicate success or failure.
+ * 6) When receiving indication about success/failure the host driver signals
+ * that the device no longer needs the boot image in memory.
+ * When all devices no longer need the image it will be removed.
+ */
+void nnpdev_set_boot_state(struct nnp_device *nnpdev, u32 mask)
+{
+ u32 state, prev_state;
+ bool becomes_ready = false;
+ int ret;
+
+ /*
+ * Save previous state and modify current state
+ * with the changed state mask
+ */
+ spin_lock(&nnpdev->lock);
+ prev_state = nnpdev->state;
+ if ((mask & NNP_DEVICE_CARD_BOOT_STATE_MASK) != 0) {
+ /*
+ * When boot state changes previous boot states are reset.
+ * also, device error conditions is cleared.
+ */
+ nnpdev->state &= ~(NNP_DEVICE_CARD_BOOT_STATE_MASK);
+ nnpdev->state &= ~(NNP_DEVICE_ERROR_MASK);
+ }
+ nnpdev->state |= mask;
+ state = nnpdev->state;
+ spin_unlock(&nnpdev->lock);
+
+ dev_dbg(nnpdev->dev,
+ "device state change 0x%x --> 0x%x\n", prev_state, state);
+
+ /* Unload boot image if boot started or failed */
+ if (nnpdev->boot_image_loaded &&
+ (((state & NNP_DEVICE_BOOT_STARTED) &&
+ !(prev_state & NNP_DEVICE_BOOT_STARTED)) ||
+ (state & NNP_DEVICE_BOOT_FAILED))) {
+ ret = unload_boot_image(nnpdev);
+ /* This should never fail */
+ if (ret)
+ dev_dbg(nnpdev->dev,
+ "Unexpected error while unloading boot image. rc=%d\n",
+ ret);
+ }
+
+ /* if in error state - no need to check rest of the states */
+ if (state & NNP_DEVICE_ERROR_MASK)
+ return;
+
+ if ((state & NNP_DEVICE_BOOT_BIOS_READY) &&
+ !(prev_state & NNP_DEVICE_BOOT_BIOS_READY)) {
+ becomes_ready = true;
+ nnpdev->is_recovery_bios = false;
+ }
+
+ if ((state & NNP_DEVICE_BOOT_RECOVERY_BIOS_READY) &&
+ !(prev_state & NNP_DEVICE_BOOT_RECOVERY_BIOS_READY)) {
+ becomes_ready = true;
+ nnpdev->is_recovery_bios = true;
+ }
+
+ if (becomes_ready ||
+ mask == NNP_DEVICE_BOOT_BIOS_READY ||
+ mask == NNP_DEVICE_BOOT_RECOVERY_BIOS_READY) {
+ if (!becomes_ready)
+ dev_dbg(nnpdev->dev, "Re-sending sysinfo page to bios!!\n");
+
+ /* Send request to fill system_info buffer */
+ send_sysinfo_request_to_bios(nnpdev);
+ return;
+ }
+
+ /* Handle boot image request */
+ if ((state & NNP_DEVICE_BOOT_SYSINFO_READY) &&
+ !(prev_state & NNP_DEVICE_BOOT_SYSINFO_READY) &&
+ !nnpdev->boot_image_loaded) {
+ build_bios_version_string(nnpdev);
+ nnpdev->bios_system_info_valid = true;
+ nnpdev->boot_image_loaded = true;
+ ret = nnpdev_load_boot_image(nnpdev);
+
+ if (ret)
+ dev_err(nnpdev->dev,
+ "Unexpected error while loading boot image. rc=%d\n",
+ ret);
+ }
+}
+
/**
* nnpdev_init() - initialize NNP-I device structure.
* @nnpdev: device to be initialized
@@ -64,8 +241,33 @@ int nnpdev_init(struct nnp_device *nnpdev, struct device *dev,
goto err_msg_sched;
}
+ nnpdev->wq = create_singlethread_workqueue("nnpdev_wq");
+ if (!nnpdev->wq) {
+ ret = -ENOMEM;
+ goto err_cmdq;
+ }
+
+ /* setup memory for bios system info */
+ nnpdev->bios_system_info =
+ dma_alloc_coherent(nnpdev->dev, NNP_PAGE_SIZE,
+ &nnpdev->bios_system_info_dma_addr, GFP_KERNEL);
+ if (!nnpdev->bios_system_info) {
+ ret = -ENOMEM;
+ goto err_wq;
+ }
+
+ /* set host driver state to "Not ready" */
+ nnpdev->ops->set_host_doorbell_value(nnpdev, 0);
+
+ spin_lock_init(&nnpdev->lock);
+ nnpdev_boot_image_init(&nnpdev->boot_image);
+
return 0;
+err_wq:
+ destroy_workqueue(nnpdev->wq);
+err_cmdq:
+ nnp_msched_queue_destroy(nnpdev->cmdq);
err_msg_sched:
nnp_msched_destroy(nnpdev->cmdq_sched);
err_ida:
@@ -74,6 +276,71 @@ int nnpdev_init(struct nnp_device *nnpdev, struct device *dev,
}
EXPORT_SYMBOL(nnpdev_init);
+struct doorbell_work {
+ struct work_struct work;
+ struct nnp_device *nnpdev;
+ u32 val;
+};
+
+static void doorbell_changed_handler(struct work_struct *work)
+{
+ struct doorbell_work *req = container_of(work, struct doorbell_work,
+ work);
+ u32 boot_state, state = 0;
+ u32 error_state;
+ u32 doorbell_val = req->val;
+ struct nnp_device *nnpdev = req->nnpdev;
+
+ nnpdev->card_doorbell_val = doorbell_val;
+
+ error_state = FIELD_GET(NNP_CARD_ERROR_MASK, doorbell_val);
+ boot_state = FIELD_GET(NNP_CARD_BOOT_STATE_MASK, doorbell_val);
+
+ if (error_state) {
+ state = NNP_DEVICE_BOOT_FAILED;
+
+ switch (error_state) {
+ case NNP_CARD_ERROR_NOT_CAPSULE:
+ state |= NNP_DEVICE_CAPSULE_EXPECTED;
+ break;
+ case NNP_CARD_ERROR_CORRUPTED_IMAGE:
+ state |= NNP_DEVICE_CORRUPTED_BOOT_IMAGE;
+ break;
+ case NNP_CARD_ERROR_CAPSULE_FAILED:
+ state |= NNP_DEVICE_CAPSULE_FAILED;
+ break;
+ default:
+ break;
+ }
+ } else if (boot_state != nnpdev->curr_boot_state) {
+ nnpdev->curr_boot_state = boot_state;
+ switch (boot_state) {
+ case NNP_CARD_BOOT_STATE_BIOS_READY:
+ state = NNP_DEVICE_BOOT_BIOS_READY;
+ break;
+ case NNP_CARD_BOOT_STATE_RECOVERY_BIOS_READY:
+ state = NNP_DEVICE_BOOT_RECOVERY_BIOS_READY;
+ break;
+ case NNP_CARD_BOOT_STATE_BIOS_SYSINFO_READY:
+ state = NNP_DEVICE_BOOT_SYSINFO_READY;
+ break;
+ case NNP_CARD_BOOT_STATE_BOOT_STARTED:
+ state = NNP_DEVICE_BOOT_STARTED;
+ break;
+ case NNP_CARD_BOOT_STATE_BIOS_FLASH_STARTED:
+ state = NNP_DEVICE_BIOS_UPDATE_STARTED;
+ break;
+ default:
+ break;
+ }
+ }
+
+ if (state)
+ nnpdev_set_boot_state(nnpdev, state);
+
+ kfree(req);
+}
+
/**
* nnpdev_card_doorbell_value_changed() - card doorbell changed notification
* @nnpdev: The nnp device
@@ -85,7 +352,18 @@ int nnpdev_init(struct nnp_device *nnpdev, struct device *dev,
void nnpdev_card_doorbell_value_changed(struct nnp_device *nnpdev,
u32 doorbell_val)
{
+ struct doorbell_work *req;
+
dev_dbg(nnpdev->dev, "Got card doorbell value 0x%x\n", doorbell_val);
+
+ req = kzalloc(sizeof(*req), GFP_KERNEL);
+ if (!req)
+ return;
+
+ req->nnpdev = nnpdev;
+ req->val = doorbell_val;
+ INIT_WORK(&req->work, doorbell_changed_handler);
+ queue_work(nnpdev->wq, &req->work);
}
EXPORT_SYMBOL(nnpdev_card_doorbell_value_changed);
@@ -100,6 +378,18 @@ void nnpdev_destroy(struct nnp_device *nnpdev)
{
dev_dbg(nnpdev->dev, "Destroying NNP-I device\n");
+ /*
+ * If device is removed while boot image load is in-flight,
+ * stop the image load and flag it is not needed.
+ */
+ if (nnpdev->boot_image_loaded)
+ unload_boot_image(nnpdev);
+
+ destroy_workqueue(nnpdev->wq);
+
+ dma_free_coherent(nnpdev->dev, NNP_PAGE_SIZE, nnpdev->bios_system_info,
+ nnpdev->bios_system_info_dma_addr);
+
nnp_msched_destroy(nnpdev->cmdq_sched);
ida_simple_remove(&dev_ida, nnpdev->id);
}
@@ -121,3 +411,4 @@ static void __exit nnp_cleanup(void)
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Intel(R) NNPI Framework");
MODULE_AUTHOR("Intel Corporation");
+MODULE_FIRMWARE(NNP_FIRMWARE_NAME);
diff --git a/drivers/misc/intel-nnpi/device.h b/drivers/misc/intel-nnpi/device.h
index 562bbc4..3745f5c 100644
--- a/drivers/misc/intel-nnpi/device.h
+++ b/drivers/misc/intel-nnpi/device.h
@@ -4,8 +4,50 @@
#ifndef _NNPDRV_DEVICE_H
#define _NNPDRV_DEVICE_H
+#include <linux/spinlock.h>
+#include <linux/workqueue.h>
+
+#include "bootimage.h"
+#include "ipc_protocol.h"
+#include "msg_scheduler.h"
+
#define NNP_MAX_DEVS 256
+#define NNP_FIRMWARE_NAME "intel/nnpi/disk.img"
+
+/* device state bits */
+#define NNP_DEVICE_BOOT_BIOS_READY BIT(1)
+#define NNP_DEVICE_BOOT_RECOVERY_BIOS_READY BIT(2)
+#define NNP_DEVICE_BOOT_SYSINFO_READY BIT(3)
+#define NNP_DEVICE_BOOT_STARTED BIT(4)
+#define NNP_DEVICE_BIOS_UPDATE_READY BIT(5)
+#define NNP_DEVICE_BIOS_UPDATE_STARTED BIT(6)
+#define NNP_DEVICE_BIOS_UPDATE_DONE BIT(7)
+#define NNP_DEVICE_CARD_DRIVER_READY BIT(8)
+#define NNP_DEVICE_CARD_READY BIT(9)
+#define NNP_DEVICE_CARD_ENABLED BIT(10)
+
+#define NNP_DEVICE_CARD_BOOT_STATE_MASK GENMASK(9, 1)
+
+#define NNP_DEVICE_ACTIVE_MASK (NNP_DEVICE_CARD_READY | \
+ NNP_DEVICE_CARD_ENABLED)
+
+#define NNP_DEVICE_FAILED_VERSION BIT(16)
+#define NNP_DEVICE_BOOT_FAILED BIT(17)
+#define NNP_DEVICE_HOST_DRIVER_ERROR BIT(18)
+#define NNP_DEVICE_KERNEL_CRASH BIT(20)
+#define NNP_DEVICE_PCI_ERROR BIT(21)
+#define NNP_DEVICE_CARD_IN_RESET BIT(22)
+#define NNP_DEVICE_FATAL_MCE_ERROR BIT(23)
+#define NNP_DEVICE_FATAL_DRAM_ECC_ERROR BIT(24)
+#define NNP_DEVICE_FATAL_ICE_ERROR BIT(25)
+#define NNP_DEVICE_HANG BIT(26)
+#define NNP_DEVICE_PROTOCOL_ERROR BIT(27)
+#define NNP_DEVICE_CAPSULE_EXPECTED BIT(28)
+#define NNP_DEVICE_CAPSULE_FAILED BIT(29)
+#define NNP_DEVICE_CORRUPTED_BOOT_IMAGE BIT(30)
+#define NNP_DEVICE_ERROR_MASK GENMASK(31, 16)
+
/**
* struct nnp_device - structure for NNP-I device info
* @ops: device operations implemented by the underlying device driver
@@ -15,6 +57,18 @@
* submissions to the device's command queue.
* @cmdq: input queue to @cmdq_sched used to schedule driver internal commands
* to be sent to the device.
+ * @wq: singlethread workqueue for processing device's response messages.
+ * @lock: protects accesses to @state
+ * @is_recovery_bios: true if device has booted from the recovery bios flash
+ * @boot_image_loaded: true if boot image load has started
+ * @bios_system_info_dma_addr: dma page allocated for bios system info.
+ * @bios_system_info: virtual pointer to bios system info page
+ * @bios_version_str: the device's started bios version string
+ * @bios_system_info_valid: true if @bios_system_info has been filled and valid
+ * @state: current device boot state mask (see device state bits above)
+ * @curr_boot_state: last boot state field received from device doorbell reg
+ * @card_doorbell_val: last received device doorbell register value.
+ * @boot_image: boot image object used to boot the card
*/
struct nnp_device {
const struct nnp_device_ops *ops;
@@ -23,6 +77,21 @@ struct nnp_device {
struct nnp_msched *cmdq_sched;
struct nnp_msched_queue *cmdq;
+
+ struct workqueue_struct *wq;
+ spinlock_t lock;
+ bool is_recovery_bios;
+ bool boot_image_loaded;
+
+ dma_addr_t bios_system_info_dma_addr;
+ struct nnp_c2h_system_info *bios_system_info;
+ char bios_version_str[NNP_BIOS_VERSION_LEN];
+ bool bios_system_info_valid;
+
+ u32 state;
+ u32 curr_boot_state;
+ u32 card_doorbell_val;
+ struct image_info boot_image;
};
/**
@@ -30,10 +99,12 @@ struct nnp_device {
* @cmdq_flush: empties the device command queue, discarding all queued
* commands.
* @cmdq_write_mesg: inserts a command message to the card's command queue.
+ * @set_host_doorbell_value: change the host doorbell value on device.
*/
struct nnp_device_ops {
int (*cmdq_flush)(struct nnp_device *hw_dev);
int (*cmdq_write_mesg)(struct nnp_device *nnpdev, u64 *msg, u32 size);
+ int (*set_host_doorbell_value)(struct nnp_device *nnpdev, u32 value);
};
bool nnpdev_no_devices(void);
@@ -47,4 +118,10 @@ int nnpdev_init(struct nnp_device *nnpdev, struct device *dev,
void nnpdev_destroy(struct nnp_device *nnpdev);
void nnpdev_card_doorbell_value_changed(struct nnp_device *nnpdev,
u32 doorbell_val);
+
+/*
+ * Framework internal functions (not exported)
+ */
+void nnpdev_set_boot_state(struct nnp_device *nnpdev, u32 mask);
+
#endif
diff --git a/drivers/misc/intel-nnpi/nnp_pcie.c b/drivers/misc/intel-nnpi/nnp_pcie.c
index 7aa9074..3840a53 100644
--- a/drivers/misc/intel-nnpi/nnp_pcie.c
+++ b/drivers/misc/intel-nnpi/nnp_pcie.c
@@ -375,9 +375,25 @@ static int nnp_cmdq_flush(struct nnp_device *nnpdev)
return 0;
}
+static int nnp_set_host_doorbell_value(struct nnp_device *nnpdev, u32 value)
+{
+ struct nnp_pci *nnp_pci = container_of(nnpdev, struct nnp_pci, nnpdev);
+
+ /*
+ * The SELF_RESET bit is set only by the h/w layer,
+ * do not allow higher layer to set it
+ */
+ value &= ~NNP_HOST_DRV_REQUEST_SELF_RESET_MASK;
+
+ nnp_mmio_write(nnp_pci, ELBI_PCI_HOST_DOORBELL_VALUE, value);
+
+ return 0;
+}
+
static struct nnp_device_ops nnp_device_ops = {
.cmdq_flush = nnp_cmdq_flush,
.cmdq_write_mesg = nnp_cmdq_write_mesg,
+ .set_host_doorbell_value = nnp_set_host_doorbell_value,
};
static void set_host_boot_state(struct nnp_pci *nnp_pci, int boot_state)
--
1.8.3.1
---------------------------------------------------------------------
Intel Israel (74) Limited
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
The nnpdev_process_messages function handles all messages coming
from a NNP-I device. Based on an opcode field attached to each message,
it calls the correct response processing handler. The function is
called from the NNP-I device driver, from a threaded interrupt handler, when
responses arrive in the HW response queue.
Signed-off-by: Guy Zadicario <[email protected]>
Reviewed-by: Alexander Shishkin <[email protected]>
---
drivers/misc/intel-nnpi/device.c | 136 +++++++++++++++++++++++++++++++++++++
drivers/misc/intel-nnpi/device.h | 10 +++
drivers/misc/intel-nnpi/nnp_pcie.c | 3 +
3 files changed, 149 insertions(+)
diff --git a/drivers/misc/intel-nnpi/device.c b/drivers/misc/intel-nnpi/device.c
index f4fc975..2e6ab82 100644
--- a/drivers/misc/intel-nnpi/device.c
+++ b/drivers/misc/intel-nnpi/device.c
@@ -23,6 +23,142 @@ bool nnpdev_no_devices(void)
return ida_is_empty(&dev_ida);
}
+/**
+ * handle_bios_protocol() - process response coming from card's BIOS.
+ * @nnpdev: The nnp device
+ * @msgbuf: pointer to response message content
+ * @avail_qwords: number of 64-bit units available in @msgbuf
+ *
+ * IPC protocol with card's BIOS may have different response sizes.
+ * @avail_qwords is the number of 64-bit units available in @msgbuf buffer.
+ * If the actual response size is larger then available data in the buffer,
+ * the function returns 0 to indicate that this is a partial response. Otherwise
+ * the actual response size is returned (in units of qwords).
+ *
+ * Return: 0 if @msgbuf contains a partial response otherwise the number of
+ * qwords of the response in @msgbuf.
+ */
+static int handle_bios_protocol(struct nnp_device *nnpdev, const u64 *msgbuf,
+ int avail_qwords)
+{
+ int msg_size, msg_qwords;
+
+ msg_size = FIELD_GET(NNP_C2H_BIOS_PROTOCOL_TYPE_MASK, msgbuf[0]);
+
+ /* The +1 is because size field does not include header */
+ msg_qwords = DIV_ROUND_UP(msg_size, 8) + 1;
+
+ if (msg_qwords > avail_qwords)
+ return 0;
+
+ return msg_qwords;
+}
+
+typedef int (*response_handler)(struct nnp_device *nnpdev, const u64 *msgbuf,
+ int avail_qwords);
+
+static response_handler resp_handlers[NNP_IPC_C2H_OPCODE_LAST + 1] = {
+ [NNP_IPC_C2H_OP_BIOS_PROTOCOL] = handle_bios_protocol
+};
+
+/**
+ * nnpdev_process_messages() - process response messages from nnpi device
+ * @nnpdev: The nnp device
+ * @hw_msg: pointer to response message content
+ * @hw_nof_msg: number of 64-bit units available in hw_msg buffer.
+ *
+ * This function is called from the PCIe device driver when response messages
+ * are arrived in the HWQ. It is called in sequence, should not be re-entrant.
+ * The function may not block !
+ */
+void nnpdev_process_messages(struct nnp_device *nnpdev, u64 *hw_msg,
+ unsigned int hw_nof_msg)
+{
+ int j = 0;
+ int msg_size;
+ u64 *msg;
+ unsigned int nof_msg;
+ bool fatal_protocol_error = false;
+
+ /* ignore any response if protocol error detected */
+ if ((nnpdev->state & NNP_DEVICE_PROTOCOL_ERROR) != 0)
+ return;
+
+ /*
+ * If we have pending messages from previous round
+ * copy the new messages to the pending list and process
+ * the pending list.
+ * otherwise process the messages received from HW directly
+ */
+ msg = hw_msg;
+ nof_msg = hw_nof_msg;
+ if (nnpdev->response_num_msgs > 0) {
+ /*
+ * Check to prevent response buffer overrun.
+ * This should never happen since the buffer is twice
+ * the size of the HW response queue. This check is
+ * for safety and debug purposes.
+ */
+ if (hw_nof_msg + nnpdev->response_num_msgs >=
+ NNP_DEVICE_RESPONSE_BUFFER_LEN) {
+ dev_dbg(nnpdev->dev,
+ "device response buffer would overrun: %d + %d !!\n",
+ nnpdev->response_num_msgs, hw_nof_msg);
+ return;
+ }
+
+ memcpy(&nnpdev->response_buf[nnpdev->response_num_msgs], hw_msg,
+ hw_nof_msg * sizeof(u64));
+ msg = nnpdev->response_buf;
+ nof_msg = nnpdev->response_num_msgs + hw_nof_msg;
+ }
+
+ /*
+ * loop for each message
+ */
+ do {
+ int op_code = FIELD_GET(NNP_C2H_OP_MASK, msg[j]);
+ response_handler handler;
+
+ /* opcodes above OP_BIOS_PROTOCOL are not yet supported */
+ if (op_code > NNP_IPC_C2H_OP_BIOS_PROTOCOL) {
+ fatal_protocol_error = true;
+ break;
+ }
+
+ /* dispatch the message request */
+ handler = resp_handlers[op_code];
+ if (!handler) {
+ /* Should not happen! */
+ dev_dbg(nnpdev->dev,
+ "Unknown response opcode received %d (0x%llx)\n",
+ op_code, msg[j]);
+ fatal_protocol_error = true;
+ break;
+ }
+
+ msg_size = (*handler)(nnpdev, &msg[j], (nof_msg - j));
+
+ j += msg_size;
+ } while (j < nof_msg || !msg_size);
+
+ if (fatal_protocol_error)
+ nnpdev->state |= NNP_DEVICE_PROTOCOL_ERROR;
+
+ /*
+ * If unprocessed messages left, copy it to the pending messages buffer
+ * for the next time this function will be called.
+ */
+ if (j < nof_msg) {
+ memcpy(&nnpdev->response_buf[0], &msg[j],
+ (nof_msg - j) * sizeof(u64));
+ nnpdev->response_num_msgs = nof_msg - j;
+ } else {
+ nnpdev->response_num_msgs = 0;
+ }
+}
+EXPORT_SYMBOL(nnpdev_process_messages);
+
static void send_sysinfo_request_to_bios(struct nnp_device *nnpdev)
{
u64 cmd[3];
diff --git a/drivers/misc/intel-nnpi/device.h b/drivers/misc/intel-nnpi/device.h
index 3745f5c..b83f67a 100644
--- a/drivers/misc/intel-nnpi/device.h
+++ b/drivers/misc/intel-nnpi/device.h
@@ -48,6 +48,9 @@
#define NNP_DEVICE_CORRUPTED_BOOT_IMAGE BIT(30)
#define NNP_DEVICE_ERROR_MASK GENMASK(31, 16)
+#define NNP_DEVICE_RESPONSE_FIFO_LEN 16
+#define NNP_DEVICE_RESPONSE_BUFFER_LEN (NNP_DEVICE_RESPONSE_FIFO_LEN * 2)
+
/**
* struct nnp_device - structure for NNP-I device info
* @ops: device operations implemented by the underlying device driver
@@ -61,6 +64,8 @@
* @lock: protects accesses to @state
* @is_recovery_bios: true if device has booted from the recovery bios flash
* @boot_image_loaded: true if boot image load has started
+ * @response_buf: buffer of device response messages arrived from "pci" layer.
+ * @response_num_msgs: number of qwords available in @response_buf
* @bios_system_info_dma_addr: dma page allocated for bios system info.
* @bios_system_info: virtual pointer to bios system info page
* @bios_version_str: the device's started bios version string
@@ -83,6 +88,9 @@ struct nnp_device {
bool is_recovery_bios;
bool boot_image_loaded;
+ u64 response_buf[NNP_DEVICE_RESPONSE_BUFFER_LEN];
+ unsigned int response_num_msgs;
+
dma_addr_t bios_system_info_dma_addr;
struct nnp_c2h_system_info *bios_system_info;
char bios_version_str[NNP_BIOS_VERSION_LEN];
@@ -118,6 +126,8 @@ int nnpdev_init(struct nnp_device *nnpdev, struct device *dev,
void nnpdev_destroy(struct nnp_device *nnpdev);
void nnpdev_card_doorbell_value_changed(struct nnp_device *nnpdev,
u32 doorbell_val);
+void nnpdev_process_messages(struct nnp_device *nnpdev, u64 *hw_msg,
+ unsigned int hw_nof_msg);
/*
* Framework internal functions (not exported)
diff --git a/drivers/misc/intel-nnpi/nnp_pcie.c b/drivers/misc/intel-nnpi/nnp_pcie.c
index 3840a53..4c9b4c6 100644
--- a/drivers/misc/intel-nnpi/nnp_pcie.c
+++ b/drivers/misc/intel-nnpi/nnp_pcie.c
@@ -150,6 +150,9 @@ static void nnp_process_commands(struct nnp_pci *nnp_pci)
response_pci_control |= FIELD_PREP(RESPQ_READ_PTR_MASK, read_pointer);
nnp_mmio_write(nnp_pci, ELBI_RESPONSE_PCI_CONTROL,
response_pci_control);
+
+ nnpdev_process_messages(&nnp_pci->nnpdev, nnp_pci->response_buf,
+ avail_slots);
}
static void mask_all_interrupts(struct nnp_pci *nnp_pci)
--
1.8.3.1
---------------------------------------------------------------------
Intel Israel (74) Limited
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
Establish bi-directional communication channels between user-mode
processes and NNP-I devices. Each command channel object ("channel")
holds a queue of messages from a single user-mode connection to a
single NNP-I device, as well as a ring-buffer to hold response messages
from this NNP-I device back to the user-mode process.
Messages to the NNP-I device are put by the channel into a command
queue. Response messages coming back from the device are being routed
to the intended channel's ring-buffer, where they are consumed by this
channel's user-mode connection. Routing of messages to and from the
device is done based on a channel's 10-bit unique id, which is included
in the messages.
The interface for consuming responses from the ring-buffer and writing
command messages into the msg_scheduler's command queue will be added
in a future patch. This patch only adds the channel creation code and
response message routing to the targeted channel.
When creating a "command channel", the user should give an open file
descriptor to the /dev/nnpi_host device. This file descriptor
associates the channel with a particular "nnp_user" object. The channel
can only reference host resources created by that "user".
Signed-off-by: Guy Zadicario <[email protected]>
Reviewed-by: Alexander Shishkin <[email protected]>
---
drivers/misc/intel-nnpi/Makefile | 2 +-
drivers/misc/intel-nnpi/cmd_chan.c | 314 +++++++++++++++++++++
drivers/misc/intel-nnpi/cmd_chan.h | 72 +++++
drivers/misc/intel-nnpi/device.c | 113 +++++++-
drivers/misc/intel-nnpi/device.h | 17 +-
drivers/misc/intel-nnpi/ipc_include/ipc_protocol.h | 3 +
6 files changed, 516 insertions(+), 5 deletions(-)
create mode 100644 drivers/misc/intel-nnpi/cmd_chan.c
create mode 100644 drivers/misc/intel-nnpi/cmd_chan.h
diff --git a/drivers/misc/intel-nnpi/Makefile b/drivers/misc/intel-nnpi/Makefile
index e46c89f..b3bab2a 100644
--- a/drivers/misc/intel-nnpi/Makefile
+++ b/drivers/misc/intel-nnpi/Makefile
@@ -6,7 +6,7 @@
obj-$(CONFIG_INTEL_NNPI) := intel_nnpi.o intel_nnpi_pcie.o
intel_nnpi-y := device.o msg_scheduler.o hostres.o host_chardev.o nnp_user.o \
- bootimage.o
+ bootimage.o cmd_chan.o
intel_nnpi_pcie-y := nnp_pcie.o
diff --git a/drivers/misc/intel-nnpi/cmd_chan.c b/drivers/misc/intel-nnpi/cmd_chan.c
new file mode 100644
index 0000000..b5518e0
--- /dev/null
+++ b/drivers/misc/intel-nnpi/cmd_chan.c
@@ -0,0 +1,314 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (C) 2019-2021 Intel Corporation */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/anon_inodes.h>
+#include <linux/dev_printk.h>
+#include <linux/file.h>
+#include <linux/minmax.h>
+#include <linux/slab.h>
+
+#include "cmd_chan.h"
+#include "host_chardev.h"
+#include "ipc_protocol.h"
+#include "nnp_user.h"
+
+#define RESPQ_INIT_BUF_SIZE SZ_2K /* must be power of 2 */
+#define RESPQ_MAX_BUF_SIZE SZ_1M /* must be power of 2 */
+
+static inline int respq_free_bytes(struct nnp_chan *chan)
+{
+ return CIRC_SPACE(chan->respq.head, chan->respq.tail, chan->respq_size);
+}
+
+static inline void respq_push(struct nnp_chan *chan, void *buf, int count)
+{
+ char *dst = chan->respq.buf + chan->respq.head;
+ int t = CIRC_SPACE_TO_END(chan->respq.head, chan->respq.tail,
+ chan->respq_size);
+
+ if (t >= count) {
+ memcpy(dst, buf, count);
+ } else {
+ memcpy(dst, buf, t);
+ memcpy(chan->respq.buf, (u8 *)buf + t, count - t);
+ }
+ chan->respq.head = (chan->respq.head + count) & (chan->respq_size - 1);
+}
+
+static inline void respq_pop(struct nnp_chan *chan, void *buf, int count)
+{
+ char *src = chan->respq.buf + chan->respq.tail;
+ int t = CIRC_CNT_TO_END(chan->respq.head, chan->respq.tail,
+ chan->respq_size);
+
+ if (t >= count) {
+ memcpy(buf, src, count);
+ } else {
+ memcpy(buf, src, t);
+ memcpy((u8 *)buf + t, chan->respq.buf, count - t);
+ }
+ chan->respq.tail = (chan->respq.tail + count) & (chan->respq_size - 1);
+}
+
+/**
+ * nnpdev_chan_create() - creates a command channel object
+ * @nnpdev: the device
+ * @host_fd: opened file descriptor to "/dev/nnpi_host"
+ * @min_id: minimum range for allocating ipc channel id for that channel
+ * @max_id: maximum range for allocating ipc channel id for that channel
+ * @get_device_events: true if this channel needs to receive device-level
+ * responses (not originated to specific channel).
+ *
+ * This function create a "command channel" and assign it a unique id within
+ * the range [@min_id..@max_id]. channels in id range [0, 255] are assumed to be
+ * used for inference related operations and have slightly special semantics.
+ *
+ * Return: pointer to created channel or error.
+ */
+struct nnp_chan *nnpdev_chan_create(struct nnp_device *nnpdev, int host_fd,
+ unsigned int min_id, unsigned int max_id,
+ bool get_device_events)
+{
+ struct nnp_chan *cmd_chan;
+ int chan_id;
+ int ret;
+ unsigned int max_proto_id = BIT(NNP_IPC_CHANNEL_BITS) - 1;
+
+ if (min_id > max_proto_id)
+ return ERR_PTR(-EINVAL);
+ if (max_id > max_proto_id)
+ max_id = max_proto_id;
+ if (max_id < min_id)
+ return ERR_PTR(-EINVAL);
+
+ ret = ida_simple_get(&nnpdev->cmd_chan_ida, min_id, max_id + 1,
+ GFP_KERNEL);
+ if (ret < 0)
+ return ERR_PTR(ret);
+ chan_id = ret;
+
+ cmd_chan = kzalloc(sizeof(*cmd_chan), GFP_KERNEL);
+ if (!cmd_chan) {
+ ret = -ENOMEM;
+ goto err_ida;
+ }
+
+ cmd_chan->respq_buf = kmalloc(RESPQ_INIT_BUF_SIZE, GFP_KERNEL);
+ if (!cmd_chan->respq_buf) {
+ ret = -ENOMEM;
+ goto err_alloc;
+ }
+ cmd_chan->respq_size = RESPQ_INIT_BUF_SIZE;
+ cmd_chan->respq.buf = cmd_chan->respq_buf;
+ spin_lock_init(&cmd_chan->respq_lock);
+
+ cmd_chan->host_file = nnp_host_file_get(host_fd);
+ if (!cmd_chan->host_file) {
+ ret = -EINVAL;
+ goto err_respq;
+ }
+
+ cmd_chan->cmdq = nnp_msched_queue_create(nnpdev->cmdq_sched);
+ if (!cmd_chan->cmdq) {
+ ret = -ENOMEM;
+ goto err_file_get;
+ }
+
+ kref_init(&cmd_chan->ref);
+ cmd_chan->chan_id = chan_id;
+ cmd_chan->nnpdev = nnpdev;
+ cmd_chan->get_device_events = get_device_events;
+
+ cmd_chan->nnp_user = cmd_chan->host_file->private_data;
+ nnp_user_get(cmd_chan->nnp_user);
+
+ init_waitqueue_head(&cmd_chan->resp_waitq);
+ mutex_init(&cmd_chan->dev_mutex);
+
+ /*
+ * Add channel to the channel hash
+ */
+ spin_lock(&nnpdev->lock);
+ hash_add(nnpdev->cmd_chan_hash, &cmd_chan->hash_node, cmd_chan->chan_id);
+
+ spin_unlock(&nnpdev->lock);
+
+ return cmd_chan;
+
+err_file_get:
+ fput(cmd_chan->host_file);
+err_respq:
+ kfree(cmd_chan->respq_buf);
+err_alloc:
+ kfree(cmd_chan);
+err_ida:
+ ida_simple_remove(&nnpdev->cmd_chan_ida, chan_id);
+ return ERR_PTR(ret);
+}
+
+static void nnp_chan_release(struct kref *kref)
+{
+ struct nnp_chan *cmd_chan = container_of(kref, struct nnp_chan, ref);
+
+ nnp_chan_disconnect(cmd_chan);
+
+ nnp_user_put(cmd_chan->nnp_user);
+
+ kfree(cmd_chan->respq_buf);
+ kfree(cmd_chan);
+}
+
+void nnp_chan_get(struct nnp_chan *cmd_chan)
+{
+ kref_get(&cmd_chan->ref);
+}
+
+void nnp_chan_put(struct nnp_chan *cmd_chan)
+{
+ kref_put(&cmd_chan->ref, nnp_chan_release);
+}
+
+/**
+ * nnp_chan_disconnect() - disconnect the channel from the NNP-I device object
+ * @cmd_chan: the command channel object
+ *
+ * This function is called when the channel is released or the NNP-I device is
+ * being removed. It disconnect the channel from the nnp_device object.
+ * A disconnected channel can no longer become connected again and cannot
+ * be used to communicate with any device.
+ */
+void nnp_chan_disconnect(struct nnp_chan *cmd_chan)
+{
+ struct nnp_device *nnpdev;
+
+ mutex_lock(&cmd_chan->dev_mutex);
+ if (!cmd_chan->nnpdev) {
+ mutex_unlock(&cmd_chan->dev_mutex);
+ return;
+ }
+
+ nnpdev = cmd_chan->nnpdev;
+ cmd_chan->nnpdev = NULL;
+ spin_lock(&nnpdev->lock);
+ hash_del(&cmd_chan->hash_node);
+ spin_unlock(&nnpdev->lock);
+ mutex_unlock(&cmd_chan->dev_mutex);
+
+ nnp_msched_queue_sync(cmd_chan->cmdq);
+ nnp_msched_queue_destroy(cmd_chan->cmdq);
+
+ ida_simple_remove(&nnpdev->cmd_chan_ida, cmd_chan->chan_id);
+}
+
+static int resize_respq(struct nnp_chan *cmd_chan)
+{
+ unsigned int avail_size;
+ unsigned int new_size;
+ char *new_buf;
+
+ new_size = min_t(unsigned int, cmd_chan->respq_size * 2, RESPQ_MAX_BUF_SIZE);
+
+ /* do not try to resize if already in maximum size */
+ if (new_size == cmd_chan->respq_size)
+ return -ENOMEM;
+
+ new_buf = kmalloc(new_size, GFP_KERNEL);
+ if (!new_buf)
+ return -ENOMEM;
+
+ /* copy data from old to new ring buffer */
+ spin_lock(&cmd_chan->respq_lock);
+ avail_size = CIRC_CNT(cmd_chan->respq.head, cmd_chan->respq.tail,
+ cmd_chan->respq_size);
+ if (avail_size > 0)
+ respq_pop(cmd_chan, new_buf, avail_size);
+ kfree(cmd_chan->respq_buf);
+ cmd_chan->respq_buf = new_buf;
+ cmd_chan->respq_size = new_size;
+ cmd_chan->respq.buf = cmd_chan->respq_buf;
+ cmd_chan->respq.tail = 0;
+ cmd_chan->respq.head = avail_size;
+ spin_unlock(&cmd_chan->respq_lock);
+ dev_dbg(cmd_chan->nnpdev->dev, "channel respq resized to %d\n", new_size);
+
+ return 0;
+}
+
+/**
+ * try_add_response() - adds a response message to respq if enough space exist
+ * @cmd_chan: the command channel object
+ * @hw_msg: response message arrived from device
+ * @size: size in bytes of the response
+ *
+ * Return: zero on success, -ENOSPC if message does not fit
+ */
+static int try_add_response(struct nnp_chan *cmd_chan, u64 *hw_msg, u32 size)
+{
+ spin_lock(&cmd_chan->respq_lock);
+
+ /* Would the response fit in the buffer? */
+ if (respq_free_bytes(cmd_chan) < size + sizeof(size)) {
+ spin_unlock(&cmd_chan->respq_lock);
+ return -ENOSPC;
+ }
+
+ /* push the response message to the ring buffer */
+ respq_push(cmd_chan, &size, sizeof(size));
+ respq_push(cmd_chan, hw_msg, size);
+
+ spin_unlock(&cmd_chan->respq_lock);
+
+ wake_up_all(&cmd_chan->resp_waitq);
+
+ return 0;
+}
+
+/**
+ * nnp_chan_add_response() - adds a response message targeting this channel
+ * @cmd_chan: the command channel object
+ * @hw_msg: response message arrived from device
+ * @size: size in bytes of the response
+ *
+ * This function is being called when a response arrived from the NNP-I card
+ * which targets to a specific command channel object.
+ * The function puts the response message in a ring buffer and will later be
+ * consumed by user space through a call to read(2) on the channel' file
+ * descriptor.
+ *
+ * Return: error code or zero on success.
+ */
+int nnp_chan_add_response(struct nnp_chan *cmd_chan, u64 *hw_msg, u32 size)
+{
+ int ret;
+
+retry:
+ ret = try_add_response(cmd_chan, hw_msg, size);
+ if (ret == -ENOSPC) {
+ /*
+ * This should *rarely* happen in normal system
+ * operation since the ring-buffer is big enough.
+ * We will get here only if the user application sleeps
+ * for a *very* long time without draining the responses.
+ * Try to resize the response buffer when it does
+ * happen, but up to a maximum value.
+ * If resize failed, we have no choice but to lose the
+ * response. Only the application that uses that channel
+ * will get affected.
+ */
+ ret = resize_respq(cmd_chan);
+ if (!ret)
+ goto retry;
+ }
+
+ if (ret) {
+ if (!cmd_chan->resp_lost)
+ dev_err_ratelimited(cmd_chan->nnpdev->dev,
+ "Response queue full for channel %d losing response!\n",
+ cmd_chan->chan_id);
+ cmd_chan->resp_lost++;
+ }
+
+ return ret;
+}
diff --git a/drivers/misc/intel-nnpi/cmd_chan.h b/drivers/misc/intel-nnpi/cmd_chan.h
new file mode 100644
index 0000000..8cb1a5e
--- /dev/null
+++ b/drivers/misc/intel-nnpi/cmd_chan.h
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright (C) 2019-2021 Intel Corporation */
+
+#ifndef NNPDRV_CMD_CHAN_H
+#define NNPDRV_CMD_CHAN_H
+
+#include <linux/circ_buf.h>
+#include <linux/hashtable.h>
+#include <linux/kref.h>
+#include <linux/list.h>
+#include <linux/mutex.h>
+#include <linux/spinlock.h>
+
+#include "device.h"
+
+/**
+ * struct nnp_chan - structure object for user<->device communication
+ * @ref: refcount for this object
+ * @nnpdev: the device this channel is connected to. May be NULL after device
+ * disconnects (on device removal or reset).
+ * @chan_id: the ipc channel id for this channel
+ * @hash_node: node to include this object in list of channels
+ * hash is in (cmd_chan_hash in nnp_device).
+ * @get_device_events: true if device-level events received from card should
+ * be sent over this channel to user.
+ * @cmdq: message queue added to msg_scheduler, for user commands to be sent
+ * to the device.
+ * @host_file: reference to opened "/dev/nnpi_host" object which defines the
+ * nnp_user object this channel connects to.
+ * @nnp_user: the nnp_user this channel belongs to.
+ * the channel can reference host resources created by this
+ * nnp_user object.
+ * @dev_mutex: protects @nnpdev
+ * @resp_waitq: waitqueue used for waiting for response messages be available.
+ * @respq: circular buffer object that receive response messages from device.
+ * @respq_lock: protects @respq
+ * @respq_buf: buffer space allocated for circular response buffer.
+ * @respq_size: current allocated size of circular response buffer.
+ * @resp_lost: number of response messages lost due to response buffer full.
+ */
+struct nnp_chan {
+ struct kref ref;
+ struct nnp_device *nnpdev;
+ u16 chan_id;
+ struct hlist_node hash_node;
+ bool get_device_events;
+
+ struct nnp_msched_queue *cmdq;
+ struct file *host_file;
+ struct nnp_user_info *nnp_user;
+
+ struct mutex dev_mutex;
+ wait_queue_head_t resp_waitq;
+
+ struct circ_buf respq;
+ spinlock_t respq_lock;
+ char *respq_buf;
+ unsigned int respq_size;
+ unsigned int resp_lost;
+};
+
+struct nnp_chan *nnpdev_chan_create(struct nnp_device *nnpdev, int host_fd,
+ unsigned int min_id, unsigned int max_id,
+ bool get_device_events);
+
+void nnp_chan_get(struct nnp_chan *cmd_chan);
+void nnp_chan_put(struct nnp_chan *cmd_chan);
+void nnp_chan_disconnect(struct nnp_chan *cmd_chan);
+
+int nnp_chan_add_response(struct nnp_chan *cmd_chan, u64 *hw_msg, u32 size);
+
+#endif
diff --git a/drivers/misc/intel-nnpi/device.c b/drivers/misc/intel-nnpi/device.c
index 7bcab563..3902876 100644
--- a/drivers/misc/intel-nnpi/device.c
+++ b/drivers/misc/intel-nnpi/device.c
@@ -11,6 +11,7 @@
#include <linux/printk.h>
#include "bootimage.h"
+#include "cmd_chan.h"
#include "device.h"
#include "host_chardev.h"
#include "msg_scheduler.h"
@@ -61,6 +62,9 @@ static void process_query_version_reply(struct work_struct *work)
u32 protocol_version;
u32 card_boot_state;
u32 val;
+ u64 chan_resp_op_size;
+ u64 chan_cmd_op_size;
+ int i;
query_version_work =
container_of(work, struct query_version_work, work);
@@ -70,6 +74,15 @@ static void process_query_version_reply(struct work_struct *work)
card_boot_state = FIELD_GET(NNP_CARD_BOOT_STATE_MASK,
nnpdev->card_doorbell_val);
+ chan_resp_op_size = query_version_work->chan_resp_op_size;
+ chan_cmd_op_size = query_version_work->chan_cmd_op_size;
+ for (i = 0; i < NNP_IPC_NUM_USER_OPS; i++) {
+ nnpdev->ipc_chan_resp_op_size[i] = chan_resp_op_size & 0x3;
+ chan_resp_op_size >>= 2;
+ nnpdev->ipc_chan_cmd_op_size[i] = chan_cmd_op_size & 0x3;
+ chan_cmd_op_size >>= 2;
+ }
+
nnpdev->protocol_version =
query_version_work->protocol_version;
nnpdev->chan_protocol_version =
@@ -167,6 +180,38 @@ static int handle_bios_protocol(struct nnp_device *nnpdev, const u64 *msgbuf,
return msg_qwords;
}
+struct nnp_chan *nnpdev_find_channel(struct nnp_device *nnpdev, u16 chan_id)
+{
+ struct nnp_chan *cmd_chan;
+
+ spin_lock(&nnpdev->lock);
+ hash_for_each_possible(nnpdev->cmd_chan_hash, cmd_chan, hash_node, chan_id)
+ if (cmd_chan->chan_id == chan_id) {
+ nnp_chan_get(cmd_chan);
+ spin_unlock(&nnpdev->lock);
+ return cmd_chan;
+ }
+ spin_unlock(&nnpdev->lock);
+
+ return NULL;
+}
+
+static void disconnect_all_channels(struct nnp_device *nnpdev)
+{
+ struct nnp_chan *cmd_chan;
+ int i;
+
+restart:
+ spin_lock(&nnpdev->lock);
+ hash_for_each(nnpdev->cmd_chan_hash, i, cmd_chan, hash_node) {
+ spin_unlock(&nnpdev->lock);
+ nnp_chan_disconnect(cmd_chan);
+ nnp_chan_put(cmd_chan);
+ goto restart;
+ }
+ spin_unlock(&nnpdev->lock);
+}
+
typedef int (*response_handler)(struct nnp_device *nnpdev, const u64 *msgbuf,
int avail_qwords);
@@ -175,6 +220,50 @@ typedef int (*response_handler)(struct nnp_device *nnpdev, const u64 *msgbuf,
[NNP_IPC_C2H_OP_BIOS_PROTOCOL] = handle_bios_protocol
};
+static int dispatch_chan_message(struct nnp_device *nnpdev, u64 *hw_msg,
+ unsigned int size)
+{
+ int op_code = FIELD_GET(NNP_C2H_CHAN_MSG_OP_MASK, hw_msg[0]);
+ int chan_id = FIELD_GET(NNP_C2H_CHAN_MSG_CHAN_ID_MASK, hw_msg[0]);
+ struct nnp_chan *chan;
+ int msg_size;
+
+ if (op_code < NNP_IPC_MIN_USER_OP ||
+ op_code > NNP_IPC_MAX_USER_OP) {
+ /* Should not happen! */
+ dev_err(nnpdev->dev,
+ "chan response opcode out-of-range received %d (0x%llx)\n",
+ op_code, *hw_msg);
+ return -EINVAL;
+ }
+
+ msg_size = nnpdev->ipc_chan_resp_op_size[op_code - NNP_IPC_MIN_USER_OP];
+ if (msg_size == 0) {
+ /* Should not happen! */
+ dev_err(nnpdev->dev,
+ "Unknown response chan opcode received %d (0x%llx)\n",
+ op_code, *hw_msg);
+ return -EINVAL;
+ }
+
+ /* Check for partial message */
+ if (size < msg_size)
+ return -ENOSPC;
+
+ chan = nnpdev_find_channel(nnpdev, chan_id);
+ if (!chan) {
+ dev_err(nnpdev->dev,
+ "Got response for invalid channel chan_id=%d 0x%llx\n",
+ chan_id, *hw_msg);
+ return msg_size;
+ }
+
+ nnp_chan_add_response(chan, hw_msg, msg_size * 8);
+ nnp_chan_put(chan);
+
+ return msg_size;
+}
+
/**
* nnpdev_process_messages() - process response messages from nnpi device
* @nnpdev: The nnp device
@@ -234,10 +323,18 @@ void nnpdev_process_messages(struct nnp_device *nnpdev, u64 *hw_msg,
int op_code = FIELD_GET(NNP_C2H_OP_MASK, msg[j]);
response_handler handler;
- /* opcodes above OP_BIOS_PROTOCOL are not yet supported */
+ /* opcodes above OP_BIOS_PROTOCOL are routed to a channel */
if (op_code > NNP_IPC_C2H_OP_BIOS_PROTOCOL) {
- fatal_protocol_error = true;
- break;
+ msg_size = dispatch_chan_message(nnpdev, &msg[j],
+ nof_msg - j);
+ if (msg_size < 0) {
+ if (msg_size != -ENOSPC)
+ fatal_protocol_error = true;
+ break;
+ }
+
+ j += msg_size;
+ continue;
}
/* dispatch the message request */
@@ -492,6 +589,11 @@ int nnpdev_init(struct nnp_device *nnpdev, struct device *dev,
nnpdev->ops = ops;
nnpdev->protocol_version = 0;
+ nnpdev->protocol_version = 0;
+
+ ida_init(&nnpdev->cmd_chan_ida);
+ hash_init(nnpdev->cmd_chan_hash);
+
nnpdev->cmdq_sched = nnp_msched_create(nnpdev);
if (!nnpdev->cmdq_sched) {
ret = -ENOMEM;
@@ -668,10 +770,15 @@ void nnpdev_destroy(struct nnp_device *nnpdev)
destroy_workqueue(nnpdev->wq);
+ disconnect_all_channels(nnpdev);
dma_free_coherent(nnpdev->dev, NNP_PAGE_SIZE, nnpdev->bios_system_info,
nnpdev->bios_system_info_dma_addr);
nnp_msched_destroy(nnpdev->cmdq_sched);
+ /*
+ * nnpdev->cmd_chan_ida is empty after disconnect_all_channels,
+ * ida_destroy is not needed
+ */
ida_simple_remove(&dev_ida, nnpdev->id);
}
EXPORT_SYMBOL(nnpdev_destroy);
diff --git a/drivers/misc/intel-nnpi/device.h b/drivers/misc/intel-nnpi/device.h
index 63bc54d..9b6383e 100644
--- a/drivers/misc/intel-nnpi/device.h
+++ b/drivers/misc/intel-nnpi/device.h
@@ -4,6 +4,8 @@
#ifndef _NNPDRV_DEVICE_H
#define _NNPDRV_DEVICE_H
+#include <linux/hashtable.h>
+#include <linux/idr.h>
#include <linux/spinlock.h>
#include <linux/workqueue.h>
@@ -70,11 +72,13 @@ struct query_version_work {
* @cmdq: input queue to @cmdq_sched used to schedule driver internal commands
* to be sent to the device.
* @wq: singlethread workqueue for processing device's response messages.
- * @lock: protects accesses to @state
+ * @lock: protects accesses to @state and @cmd_chan_hash
* @is_recovery_bios: true if device has booted from the recovery bios flash
* @boot_image_loaded: true if boot image load has started
* @response_buf: buffer of device response messages arrived from "pci" layer.
* @response_num_msgs: number of qwords available in @response_buf
+ * @cmd_chan_ida: allocate channel ids to be used in ipc protocol.
+ * @cmd_chan_hash: maps command channel id to its struct pointer.
* @bios_system_info_dma_addr: dma page allocated for bios system info.
* @bios_system_info: virtual pointer to bios system info page
* @bios_version_str: the device's started bios version string
@@ -87,6 +91,9 @@ struct query_version_work {
* @boot_image: boot image object used to boot the card
* @query_version_work: work struct used to schedule processing of version
* reply response message arrived from card.
+ * @ipc_chan_resp_op_size: holds response size for each possible channel
+ * response.
+ * @ipc_chan_cmd_op_size: holds command size for each possible channel command.
*/
struct nnp_device {
const struct nnp_device_ops *ops;
@@ -104,6 +111,9 @@ struct nnp_device {
u64 response_buf[NNP_DEVICE_RESPONSE_BUFFER_LEN];
unsigned int response_num_msgs;
+ struct ida cmd_chan_ida;
+ DECLARE_HASHTABLE(cmd_chan_hash, 6);
+
dma_addr_t bios_system_info_dma_addr;
struct nnp_c2h_system_info *bios_system_info;
char bios_version_str[NNP_BIOS_VERSION_LEN];
@@ -117,6 +127,9 @@ struct nnp_device {
struct image_info boot_image;
struct query_version_work query_version_work;
+
+ u8 ipc_chan_resp_op_size[NNP_IPC_NUM_USER_OPS];
+ u8 ipc_chan_cmd_op_size[NNP_IPC_NUM_USER_OPS];
};
/**
@@ -151,4 +164,6 @@ void nnpdev_process_messages(struct nnp_device *nnpdev, u64 *hw_msg,
*/
void nnpdev_set_boot_state(struct nnp_device *nnpdev, u32 mask);
+struct nnp_chan *nnpdev_find_channel(struct nnp_device *nnpdev, u16 chan_id);
+
#endif
diff --git a/drivers/misc/intel-nnpi/ipc_include/ipc_protocol.h b/drivers/misc/intel-nnpi/ipc_include/ipc_protocol.h
index 59b4a79..037c362 100644
--- a/drivers/misc/intel-nnpi/ipc_include/ipc_protocol.h
+++ b/drivers/misc/intel-nnpi/ipc_include/ipc_protocol.h
@@ -11,6 +11,9 @@
#define IPC_OP_MAX BIT(6)
#define NNP_IPC_OPCODE_MASK GENMASK(5, 0)
+#define NNP_IPC_MIN_USER_OP 32
+#define NNP_IPC_MAX_USER_OP 63
+#define NNP_IPC_NUM_USER_OPS (NNP_IPC_MAX_USER_OP - NNP_IPC_MIN_USER_OP + 1)
#define NNP_MSG_SIZE(msg) (sizeof(msg) / sizeof(__le64))
--
1.8.3.1
---------------------------------------------------------------------
Intel Israel (74) Limited
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
There is no point allowing a user application to create host resources
on a system that does not equipped with any NNP-I devices.
Fail openning the nnpi_host character device when no NNP-I devices are
attached.
It is OK to do that check without any synchronization as a race would not
be an issue.
Signed-off-by: Guy Zadicario <[email protected]>
Reviewed-by: Alexander Shishkin <[email protected]>
---
drivers/misc/intel-nnpi/device.c | 5 +++++
drivers/misc/intel-nnpi/device.h | 2 ++
drivers/misc/intel-nnpi/host_chardev.c | 7 +++++++
3 files changed, 14 insertions(+)
diff --git a/drivers/misc/intel-nnpi/device.c b/drivers/misc/intel-nnpi/device.c
index 0f98398..a3c6a1d 100644
--- a/drivers/misc/intel-nnpi/device.c
+++ b/drivers/misc/intel-nnpi/device.c
@@ -13,6 +13,11 @@
static DEFINE_IDA(dev_ida);
+bool nnpdev_no_devices(void)
+{
+ return ida_is_empty(&dev_ida);
+}
+
/**
* nnpdev_init() - initialize NNP-I device structure.
* @nnpdev: device to be initialized
diff --git a/drivers/misc/intel-nnpi/device.h b/drivers/misc/intel-nnpi/device.h
index 7d7ef60..562bbc4 100644
--- a/drivers/misc/intel-nnpi/device.h
+++ b/drivers/misc/intel-nnpi/device.h
@@ -36,6 +36,8 @@ struct nnp_device_ops {
int (*cmdq_write_mesg)(struct nnp_device *nnpdev, u64 *msg, u32 size);
};
+bool nnpdev_no_devices(void);
+
/*
* Functions exported by the device framework module which are
* called by the lower layer NNP-I device driver module
diff --git a/drivers/misc/intel-nnpi/host_chardev.c b/drivers/misc/intel-nnpi/host_chardev.c
index 6048fda..fad5954 100644
--- a/drivers/misc/intel-nnpi/host_chardev.c
+++ b/drivers/misc/intel-nnpi/host_chardev.c
@@ -217,6 +217,13 @@ static int host_open(struct inode *inode, struct file *f)
if (!is_host_file(f))
return -EINVAL;
+ /*
+ * No point to serve host resource creation while no
+ * NNP-I devices exist in the system.
+ */
+ if (nnpdev_no_devices())
+ return -ENODEV;
+
user_info = kzalloc(sizeof(*user_info), GFP_KERNEL);
if (!user_info)
return -ENOMEM;
--
1.8.3.1
---------------------------------------------------------------------
Intel Israel (74) Limited
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
Expose an anon file descriptor interface to a command channel object
which allows user-space to send commands to the device by writing to
that file as well as consume device response messages by reading the
file.
When the file is released (closed), a channel shut-down sequence
starts. First, a message is sent to the device notifying it that the
channel is closing. Once the response to this message is received from
the device, the command channel object is destroyed.
Signed-off-by: Guy Zadicario <[email protected]>
Reviewed-by: Alexander Shishkin <[email protected]>
---
drivers/misc/intel-nnpi/cmd_chan.c | 362 +++++++++++++++++++++++++++++++++++++
drivers/misc/intel-nnpi/cmd_chan.h | 22 ++-
drivers/misc/intel-nnpi/device.c | 71 ++++++++
3 files changed, 454 insertions(+), 1 deletion(-)
diff --git a/drivers/misc/intel-nnpi/cmd_chan.c b/drivers/misc/intel-nnpi/cmd_chan.c
index b5518e0..89ae604 100644
--- a/drivers/misc/intel-nnpi/cmd_chan.c
+++ b/drivers/misc/intel-nnpi/cmd_chan.c
@@ -4,13 +4,16 @@
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include <linux/anon_inodes.h>
+#include <linux/bitfield.h>
#include <linux/dev_printk.h>
#include <linux/file.h>
#include <linux/minmax.h>
+#include <linux/poll.h>
#include <linux/slab.h>
#include "cmd_chan.h"
#include "host_chardev.h"
+#include "ipc_c2h_events.h"
#include "ipc_protocol.h"
#include "nnp_user.h"
@@ -52,6 +55,258 @@ static inline void respq_pop(struct nnp_chan *chan, void *buf, int count)
chan->respq.tail = (chan->respq.tail + count) & (chan->respq_size - 1);
}
+static inline void respq_unpop(struct nnp_chan *chan, int count)
+{
+ chan->respq.tail = (chan->respq.tail - count) & (chan->respq_size - 1);
+}
+
+enum respq_state {
+ RESPQ_EMPTY = 0,
+ RESPQ_MSG_AVAIL,
+ RESPQ_DISCONNECTED
+};
+
+/**
+ * respq_state() - check if a response message is available to be popped
+ * @chan: the cmd_chan object
+ *
+ * Checks if new response message is available or channel has been destroyed.
+ *
+ * Return:
+ * * RESPQ_EMPTY - response queue is empty
+ * * RESPQ_MSG_AVAIL - a response message is available in the queue
+ * * RESPQ_DISCONNECTED - the channel in a destroyed state
+ */
+static enum respq_state respq_state(struct nnp_chan *chan)
+{
+ bool ret;
+
+ mutex_lock(&chan->dev_mutex);
+ if (chan->state == NNP_CHAN_DESTROYED) {
+ mutex_unlock(&chan->dev_mutex);
+ return RESPQ_DISCONNECTED;
+ }
+
+ spin_lock(&chan->respq_lock);
+ /*
+ * response messages are pushed into the respq ring-buffer by pushing
+ * the size of the message (as u32) followed by message content.
+ * So an entire message is available only if more than sizeof(u32)
+ * bytes are available (there is no message with zero size).
+ */
+ if (CIRC_CNT(chan->respq.head, chan->respq.tail, chan->respq_size) >
+ sizeof(u32))
+ ret = RESPQ_MSG_AVAIL;
+ else
+ ret = RESPQ_EMPTY;
+ spin_unlock(&chan->respq_lock);
+
+ mutex_unlock(&chan->dev_mutex);
+
+ return ret;
+}
+
+static inline int is_cmd_chan_file(struct file *f);
+
+static int cmd_chan_file_release(struct inode *inode, struct file *f)
+{
+ struct nnp_chan *chan = f->private_data;
+ struct file *host_file;
+
+ if (!is_cmd_chan_file(f))
+ return -EINVAL;
+
+ nnp_chan_send_destroy(chan);
+
+ host_file = chan->host_file;
+ nnp_chan_put(chan);
+ fput(host_file);
+
+ return 0;
+}
+
+/**
+ * cmd_chan_file_read() - reads a single response message arrived from device
+ * @f: cmd_chan file descriptor
+ * @buf: buffer to receive the message
+ * @size: size of buf, must be at least 16 qwords (16 * sizeof(u64))
+ * @off: ignored.
+ *
+ * This function will block and wait until interrupted or a response
+ * message from device is available.
+ * When message(s) are available, it reads a single message, copy it to
+ * @buf and returns the message size.
+ * The given @buf and @size must be large enough to receive the largest
+ * possible response which is 16 qwords, otherwise -EINVAL is returned.
+ * The function returns the size of the received message, a return value
+ * of zero means that corrupted message has been detected and no more reads
+ * can be made from this channel.
+ *
+ * Return: if positive, the size in bytes of the read message.
+ * zero, if a corrupted message has been detected.
+ * error code otherwise
+ */
+static ssize_t cmd_chan_file_read(struct file *f, char __user *buf, size_t size,
+ loff_t *off)
+{
+ struct nnp_chan *chan = f->private_data;
+ u64 msg[NNP_DEVICE_RESPONSE_FIFO_LEN];
+ enum respq_state state;
+ u32 msg_size;
+ int ret;
+
+ if (!is_cmd_chan_file(f))
+ return -EINVAL;
+
+ if (size < sizeof(msg))
+ return -EINVAL;
+
+ /*
+ * wait for response message to be available, interrupted or channel
+ * has been destroyed on us.
+ */
+ ret = wait_event_interruptible(chan->resp_waitq,
+ (state = respq_state(chan)) != RESPQ_EMPTY);
+ if (ret < 0)
+ return ret;
+
+ if (state == RESPQ_DISCONNECTED)
+ return -EPIPE;
+
+ spin_lock(&chan->respq_lock);
+ respq_pop(chan, &msg_size, sizeof(msg_size));
+ /*
+ * Check msg_size does not overrun msg size.
+ * This will never happen unless the response ring buffer got
+ * corrupted in some way.
+ * We detect it here for safety and return zero
+ */
+ if (msg_size > sizeof(msg)) {
+ /*
+ * unpop the bad size to let subsequent read attempts
+ * to fail as well.
+ */
+ respq_unpop(chan, sizeof(msg_size));
+ spin_unlock(&chan->respq_lock);
+ return 0;
+ }
+ respq_pop(chan, msg, msg_size);
+ spin_unlock(&chan->respq_lock);
+
+ if (copy_to_user(buf, msg, msg_size))
+ return -EFAULT;
+
+ return (ssize_t)msg_size;
+}
+
+/**
+ * cmd_chan_file_write() - schedule a command message to be sent to the device.
+ * @f: a cmd_chan file descriptor
+ * @buf: the command message content
+ * @size: size in bytes of the message, must be multiple of 8 and not larger
+ * than 3 qwords.
+ * @off: ignored
+ *
+ * This function reads a command message from buffer and puts it in the
+ * channel's message queue to schedule it to be delivered to the device.
+ * The function returns when the message is copied to the message scheduler
+ * queue without waiting for it to be sent out.
+ * A valid command message size must be qword aligned and not larger than
+ * the maximum size the message scheduler support, which is 3 qwords.
+ *
+ * The function also validate the command content and fail if the chan_id
+ * field of the command header does not belong to the same channel of this
+ * file descriptor, or the command opcode is out of range, or the command
+ * size does not fit the size of this opcode.
+ *
+ * Return: the size of the message written or error code.
+ */
+static ssize_t cmd_chan_file_write(struct file *f, const char __user *buf,
+ size_t size, loff_t *off)
+{
+ struct nnp_chan *chan = f->private_data;
+ u64 msg[MSG_SCHED_MAX_MSG_SIZE];
+ unsigned int chan_id, opcode;
+ unsigned int op;
+ int rc = 0;
+
+ if (!is_cmd_chan_file(f))
+ return -EINVAL;
+
+ /*
+ * size must be positive, multiple of 8 bytes and
+ * cannot exceed maximum message size
+ */
+ if (!size || size > sizeof(msg) || (size & 0x7) != 0)
+ return -EINVAL;
+
+ if (copy_from_user(msg, buf, size))
+ return -EFAULT;
+
+ /*
+ * Check chan_id, opcode and message size are valid
+ */
+ opcode = FIELD_GET(NNP_H2C_CHAN_MSG_OP_MASK, msg[0]);
+ chan_id = FIELD_GET(NNP_H2C_CHAN_MSG_CHAN_ID_MASK, msg[0]);
+ if (chan_id != chan->chan_id)
+ return -EINVAL;
+ if (opcode < NNP_IPC_MIN_USER_OP)
+ return -EINVAL;
+ op = opcode - NNP_IPC_MIN_USER_OP;
+
+ mutex_lock(&chan->dev_mutex);
+ if (!chan->nnpdev) {
+ /* The device was removed */
+ mutex_unlock(&chan->dev_mutex);
+ return -EPIPE;
+ }
+ if (size != chan->nnpdev->ipc_chan_cmd_op_size[op] * 8) {
+ mutex_unlock(&chan->dev_mutex);
+ return -EINVAL;
+ }
+
+ if (!is_card_fatal_drv_event(chan_broken(chan)))
+ rc = nnp_msched_queue_add_msg(chan->cmdq, msg, size / 8);
+ mutex_unlock(&chan->dev_mutex);
+
+ if (rc < 0)
+ return rc;
+
+ return size;
+}
+
+static unsigned int cmd_chan_file_poll(struct file *f, struct poll_table_struct *pt)
+{
+ struct nnp_chan *chan = f->private_data;
+ unsigned int mask = POLLOUT | POLLWRNORM;
+ enum respq_state state;
+
+ if (!is_cmd_chan_file(f))
+ return 0;
+
+ poll_wait(f, &chan->resp_waitq, pt);
+ state = respq_state(chan);
+ if (state != RESPQ_EMPTY)
+ mask |= POLLIN | POLLRDNORM;
+ if (state == RESPQ_DISCONNECTED)
+ mask |= POLLHUP;
+
+ return mask;
+}
+
+static const struct file_operations nnp_chan_fops = {
+ .owner = THIS_MODULE,
+ .release = cmd_chan_file_release,
+ .read = cmd_chan_file_read,
+ .write = cmd_chan_file_write,
+ .poll = cmd_chan_file_poll,
+};
+
+static inline int is_cmd_chan_file(struct file *f)
+{
+ return f->f_op == &nnp_chan_fops;
+}
+
/**
* nnpdev_chan_create() - creates a command channel object
* @nnpdev: the device
@@ -119,6 +374,7 @@ struct nnp_chan *nnpdev_chan_create(struct nnp_device *nnpdev, int host_fd,
kref_init(&cmd_chan->ref);
cmd_chan->chan_id = chan_id;
cmd_chan->nnpdev = nnpdev;
+ cmd_chan->fd = -1;
cmd_chan->get_device_events = get_device_events;
cmd_chan->nnp_user = cmd_chan->host_file->private_data;
@@ -154,6 +410,17 @@ static void nnp_chan_release(struct kref *kref)
nnp_chan_disconnect(cmd_chan);
+ /*
+ * If a chan file was created (through nnp_chan_create_file),
+ * the host_file was already put when the file has released, otherwise
+ * we put it here.
+ * This is because we want to release host_file once the channel file
+ * has been closed even though the channel object may continue to exist
+ * until the card will send a respond that it was destroyed.
+ */
+ if (cmd_chan->fd < 0)
+ fput(cmd_chan->host_file);
+
nnp_user_put(cmd_chan->nnp_user);
kfree(cmd_chan->respq_buf);
@@ -170,6 +437,99 @@ void nnp_chan_put(struct nnp_chan *cmd_chan)
kref_put(&cmd_chan->ref, nnp_chan_release);
}
+int nnp_chan_create_file(struct nnp_chan *cmd_chan)
+{
+ /*
+ * get refcount to the channel that will drop when
+ * the file is released.
+ */
+ nnp_chan_get(cmd_chan);
+
+ cmd_chan->fd = anon_inode_getfd("nnpi_chan", &nnp_chan_fops, cmd_chan,
+ O_RDWR | O_CLOEXEC);
+ if (cmd_chan->fd < 0)
+ nnp_chan_put(cmd_chan);
+
+ return cmd_chan->fd;
+}
+
+/**
+ * nnp_chan_set_destroyed() - atomically mark the channel "destroyed"
+ * @chan: the cmd_chan
+ *
+ * This function sets the command channel state to "destroyed" and returns
+ * the previous destroyed state.
+ * This function should be called once the channel has been destructed on the
+ * device and a "channel destroyed" response message arrived.
+ *
+ * Return: true if the channel was already marked destroyed.
+ */
+bool nnp_chan_set_destroyed(struct nnp_chan *chan)
+{
+ bool ret;
+
+ mutex_lock(&chan->dev_mutex);
+ ret = (chan->state == NNP_CHAN_DESTROYED);
+ chan->state = NNP_CHAN_DESTROYED;
+ mutex_unlock(&chan->dev_mutex);
+
+ wake_up_all(&chan->resp_waitq);
+
+ return ret;
+}
+
+/**
+ * nnp_chan_send_destroy() - sends a "destroy channel" command to device
+ * @chan: the cmd_chan to destroy.
+ *
+ * This function sends a command to the device to destroy a command channel,
+ * The channel object remains to exist, it will be dropped only when the device
+ * send back a "channel destroyed" response message.
+ * In case the device is in critical error state, we treat it as not
+ * functional, and the function will immediately drop the channel object without
+ * sending any command and will return with success.
+ *
+ * Return: 0 on success, error value otherwise.
+ */
+int nnp_chan_send_destroy(struct nnp_chan *chan)
+{
+ u64 cmd;
+ int ret = 0;
+ bool do_put = false;
+
+ mutex_lock(&chan->dev_mutex);
+ if (chan->state == NNP_CHAN_DESTROYED || !chan->nnpdev)
+ goto done;
+
+ cmd = FIELD_PREP(NNP_H2C_OP_MASK, NNP_IPC_H2C_OP_CHANNEL_OP);
+ cmd |= FIELD_PREP(NNP_H2C_CHANNEL_OP_CHAN_ID_MASK, chan->chan_id);
+ cmd |= FIELD_PREP(NNP_H2C_CHANNEL_OP_DESTROY_MASK, 1);
+
+ chan->event_msg = 0;
+
+ /*
+ * If card is in critical state (or was during the channel lifetime)
+ * we destroy the channel.
+ * otherwise, we send a destroy command to card and will destroy when
+ * the destroy reply arrives.
+ */
+ if (is_card_fatal_drv_event(chan_broken(chan))) {
+ chan->state = NNP_CHAN_DESTROYED;
+ do_put = true;
+ goto done;
+ }
+
+ ret = nnp_msched_queue_msg(chan->cmdq, cmd);
+
+done:
+ mutex_unlock(&chan->dev_mutex);
+ if (do_put) {
+ wake_up_all(&chan->resp_waitq);
+ nnp_chan_put(chan);
+ }
+ return ret;
+}
+
/**
* nnp_chan_disconnect() - disconnect the channel from the NNP-I device object
* @cmd_chan: the command channel object
@@ -194,8 +554,10 @@ void nnp_chan_disconnect(struct nnp_chan *cmd_chan)
spin_lock(&nnpdev->lock);
hash_del(&cmd_chan->hash_node);
spin_unlock(&nnpdev->lock);
+ cmd_chan->state = NNP_CHAN_DESTROYED;
mutex_unlock(&cmd_chan->dev_mutex);
+ wake_up_all(&cmd_chan->resp_waitq);
nnp_msched_queue_sync(cmd_chan->cmdq);
nnp_msched_queue_destroy(cmd_chan->cmdq);
diff --git a/drivers/misc/intel-nnpi/cmd_chan.h b/drivers/misc/intel-nnpi/cmd_chan.h
index 2be88c6..d60abf4 100644
--- a/drivers/misc/intel-nnpi/cmd_chan.h
+++ b/drivers/misc/intel-nnpi/cmd_chan.h
@@ -16,6 +16,16 @@
#include "ipc_c2h_events.h"
/**
+ * enum nnp_chan_state - indicate special state of a command channel
+ * @NNP_CHAN_NORMAL: channel is in normal state.
+ * @NNP_CHAN_DESTROYED: channel should be treated as no-longer-exist on card.
+ */
+enum nnp_chan_state {
+ NNP_CHAN_NORMAL = 0,
+ NNP_CHAN_DESTROYED,
+};
+
+/**
* struct nnp_chan - structure object for user<->device communication
* @ref: refcount for this object
* @nnpdev: the device this channel is connected to. May be NULL after device
@@ -23,9 +33,11 @@
* @chan_id: the ipc channel id for this channel
* @hash_node: node to include this object in list of channels
* hash is in (cmd_chan_hash in nnp_device).
+ * @event_msg: ipc event response received from device during create channel
* @card_critical_error_msg: last critical event report received from device
* @get_device_events: true if device-level events received from card should
* be sent over this channel to user.
+ * @fd: file descriptor created for the channel (implements read/write)
* @cmdq: message queue added to msg_scheduler, for user commands to be sent
* to the device.
* @host_file: reference to opened "/dev/nnpi_host" object which defines the
@@ -33,7 +45,8 @@
* @nnp_user: the nnp_user this channel belongs to.
* the channel can reference host resources created by this
* nnp_user object.
- * @dev_mutex: protects @nnpdev
+ * @dev_mutex: protects @nnpdev and @state
+ * @state: the current state of this channel.
* @resp_waitq: waitqueue used for waiting for response messages be available.
* @respq: circular buffer object that receive response messages from device.
* @respq_lock: protects @respq
@@ -46,15 +59,18 @@ struct nnp_chan {
struct nnp_device *nnpdev;
u16 chan_id;
struct hlist_node hash_node;
+ u64 event_msg;
u64 card_critical_error_msg;
bool get_device_events;
+ int fd;
struct nnp_msched_queue *cmdq;
struct file *host_file;
struct nnp_user_info *nnp_user;
struct mutex dev_mutex;
wait_queue_head_t resp_waitq;
+ enum nnp_chan_state state;
struct circ_buf respq;
spinlock_t respq_lock;
@@ -73,6 +89,10 @@ struct nnp_chan *nnpdev_chan_create(struct nnp_device *nnpdev, int host_fd,
void nnp_chan_put(struct nnp_chan *cmd_chan);
void nnp_chan_disconnect(struct nnp_chan *cmd_chan);
+int nnp_chan_create_file(struct nnp_chan *cmd_chan);
+int nnp_chan_send_destroy(struct nnp_chan *chan);
+bool nnp_chan_set_destroyed(struct nnp_chan *chan);
+
int nnp_chan_add_response(struct nnp_chan *cmd_chan, u64 *hw_msg, u32 size);
#endif
diff --git a/drivers/misc/intel-nnpi/device.c b/drivers/misc/intel-nnpi/device.c
index 1064edc..ece19c0 100644
--- a/drivers/misc/intel-nnpi/device.c
+++ b/drivers/misc/intel-nnpi/device.c
@@ -246,6 +246,47 @@ static void nnpdev_submit_device_event_to_channels(struct nnp_device *nnpdev,
if (should_wake)
wake_up_all(&nnpdev->waitq);
+
+ /*
+ * On card fatal event, we consider the device dead and there is
+ * no point communicating with it. The user will destroy the channel
+ * and initiate a device reset to fix this.
+ * We disconnect all channels and set each as "destroyed" since the
+ * NNP_IPC_CHANNEL_DESTROYED response, which normally do that, will
+ * never arrive.
+ */
+ if (is_card_fatal_drv_event(event_code))
+ disconnect_all_channels(nnpdev);
+}
+
+static void handle_channel_destroy(struct nnp_device *nnpdev, u64 event_msg)
+{
+ struct nnp_chan *cmd_chan;
+ unsigned int chan_id;
+
+ chan_id = FIELD_GET(NNP_C2H_EVENT_REPORT_OBJ_ID_MASK, event_msg);
+ cmd_chan = nnpdev_find_channel(nnpdev, chan_id);
+ if (!cmd_chan) {
+ dev_err(nnpdev->dev,
+ "Got channel destroyed reply for not existing channel %d\n",
+ chan_id);
+ return;
+ }
+
+ /*
+ * Channel is destroyed on device. Put the main ref of cmd_chan if it
+ * did not already done.
+ * There is one possible case that the channel will be already marked
+ * as destroyed when we get here. This is when we got some card fatal
+ * event, which caused us to flag the channel as destroyed, but later
+ * the "destroy channel" response has arrived from the device
+ * (unexpected).
+ */
+ if (!nnp_chan_set_destroyed(cmd_chan))
+ nnp_chan_put(cmd_chan);
+
+ /* put against the get from find_channel */
+ nnp_chan_put(cmd_chan);
}
/*
@@ -254,6 +295,36 @@ static void nnpdev_submit_device_event_to_channels(struct nnp_device *nnpdev,
*/
static void process_device_event(struct nnp_device *nnpdev, u64 event_msg)
{
+ unsigned int event_code = FIELD_GET(NNP_C2H_EVENT_REPORT_CODE_MASK, event_msg);
+ unsigned int obj_id, event_val;
+
+ if (!is_card_fatal_event(event_code)) {
+ switch (event_code) {
+ case NNP_IPC_DESTROY_CHANNEL_FAILED:
+ obj_id = FIELD_GET(NNP_C2H_EVENT_REPORT_OBJ_ID_MASK, event_msg);
+ event_val = FIELD_GET(NNP_C2H_EVENT_REPORT_VAL_MASK, event_msg);
+ dev_err(nnpdev->dev,
+ "Channel destroyed failed channel %d val %d\n",
+ obj_id, event_val);
+ /*
+ * We should not enter this case never as the card will
+ * send this response only when the driver requested to
+ * destroy a not-exist channel, which means a driver
+ * bug.
+ * To handle the case we continue and destroy the channel
+ * on the host side.
+ */
+ fallthrough;
+ case NNP_IPC_CHANNEL_DESTROYED:
+ handle_channel_destroy(nnpdev, event_msg);
+ break;
+ default:
+ dev_err(nnpdev->dev,
+ "Unknown event received - %u\n", event_code);
+ return;
+ }
+ }
+
/* submit the event to all channels requested to get device events */
nnpdev_submit_device_event_to_channels(nnpdev, event_msg);
}
--
1.8.3.1
---------------------------------------------------------------------
Intel Israel (74) Limited
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
Check that the card booted SW stack is compatible with the driver
running on the host. The IPC protocol between the driver and cards's
SW stack may change from time to time when releasing new versions of
the card boot image.
When the card boots and signals to the host that it is booted and
ready (through doorbell change interrupt), the driver sends
it a "query version" command. In response, the device sends back a
"query version reply" response with information of the IPC protocol
version which it supports. Only when the version check passes, the
device is considered booted and ready for operation.
If the version check fails, the device is put in error state. In order
to recover from this error condition, a device reset is required
(probably preceded by installation of a new, compatile boot image).
Signed-off-by: Guy Zadicario <[email protected]>
Reviewed-by: Alexander Shishkin <[email protected]>
---
drivers/misc/intel-nnpi/device.c | 145 +++++++++++++++++++++++++++++++++++++++
drivers/misc/intel-nnpi/device.h | 17 +++++
2 files changed, 162 insertions(+)
diff --git a/drivers/misc/intel-nnpi/device.c b/drivers/misc/intel-nnpi/device.c
index 2e6ab82..7bcab563 100644
--- a/drivers/misc/intel-nnpi/device.c
+++ b/drivers/misc/intel-nnpi/device.c
@@ -24,6 +24,119 @@ bool nnpdev_no_devices(void)
}
/**
+ * process_query_version_reply() - process a "query_version_reply" response
+ * @work: work struct of the calling work
+ *
+ * This function processes a "query_version_reply" response message from
+ * the card which is sent as reply to query_version command submitted
+ * earlier.
+ * The function checks that the IPC protocol version that is supported by the
+ * device matches the one supported by the driver. If there is no match the
+ * device state is put in error.
+ * There are two IPC protocol versions which are checked:
+ * 'protocol_version': is IPC protocol version of command and response messages
+ * That are built (for commands) and processed by this kernel mode
+ * driver. The protocol is defined in ipc_include/ipc_protocol.h
+ * A mismatch is possible in cases that the device has booted with
+ * a wrong/older version of the card boot image.
+ * 'chan_protocol_version': is IPC protocol of command and responses which are
+ * supported by the device but are built and processed in user-space.
+ * The structure of the commands and responses are mostly opaque to
+ * the kernel mode driver. This separation allows to update the
+ * device boot image and user-space library to support new sets
+ * of commands without changing the kernel driver.
+ * The restriction for such commands and responses is that the lowest
+ * 16-bits of the command/response are defined to include the
+ * command/response opcode and the channel id.
+ * The kernel driver should also know for each possible command and
+ * response opcode the size of the message. This info is received
+ * from the device within this "query_version_reply" response
+ * encoded in the chan_resp_op_size and chan_cmd_op_size fields
+ * of the response.
+ */
+static void process_query_version_reply(struct work_struct *work)
+{
+ struct query_version_work *query_version_work;
+ struct nnp_device *nnpdev;
+ u32 protocol_version;
+ u32 card_boot_state;
+ u32 val;
+
+ query_version_work =
+ container_of(work, struct query_version_work, work);
+ nnpdev = container_of(query_version_work, struct nnp_device,
+ query_version_work);
+ protocol_version = NNP_IPC_PROTOCOL_VERSION;
+ card_boot_state = FIELD_GET(NNP_CARD_BOOT_STATE_MASK,
+ nnpdev->card_doorbell_val);
+
+ nnpdev->protocol_version =
+ query_version_work->protocol_version;
+ nnpdev->chan_protocol_version =
+ query_version_work->chan_protocol_version;
+
+ /*
+ * NOTE: The card firmware and host driver protocol version must
+ * exactly match in the major and minor version components.
+ * There is no backwards compatibility on the protocol!
+ * When a device is put in a protocol version error state, the
+ * user must install a matching device firmware and reset the device
+ * in order to allow the device to function.
+ */
+ if (NNP_VERSION_MAJOR(query_version_work->protocol_version) !=
+ NNP_VERSION_MAJOR(protocol_version) ||
+ NNP_VERSION_MINOR(query_version_work->protocol_version) !=
+ NNP_VERSION_MINOR(protocol_version) ||
+ query_version_work->chan_resp_op_size == 0) {
+ nnpdev_set_boot_state(nnpdev, NNP_DEVICE_FAILED_VERSION);
+ /* set host driver state in doorbell register */
+ val = FIELD_PREP(NNP_HOST_DRV_STATE_MASK,
+ NNP_HOST_DRV_STATE_VERSION_ERROR);
+ nnpdev->ops->set_host_doorbell_value(nnpdev, val);
+ } else if (card_boot_state == NNP_CARD_BOOT_STATE_DRV_READY) {
+ nnpdev_set_boot_state(nnpdev, NNP_DEVICE_CARD_DRIVER_READY);
+ } else if (card_boot_state == NNP_CARD_BOOT_STATE_CARD_READY) {
+ /* Card driver finished initialization */
+ nnpdev_set_boot_state(nnpdev,
+ NNP_DEVICE_CARD_DRIVER_READY |
+ NNP_DEVICE_CARD_READY |
+ NNP_DEVICE_CARD_ENABLED);
+ }
+
+ query_version_work->running = false;
+}
+
+static int handle_query_version_reply3(struct nnp_device *nnpdev,
+ const u64 *msgbuf, int avail_qwords)
+{
+ int msg_qwords = 3; /* QUERY_VERSION_REPLY3 response len is 3 qwords */
+
+ if (avail_qwords < msg_qwords)
+ return 0;
+
+ /*
+ * This should not happen, but if it does, just ignore the message
+ * There is no fear in race condition on "running" flag as only
+ * single version reply message should be processed after each
+ * device reset.
+ */
+ if (nnpdev->query_version_work.running)
+ return msg_qwords;
+
+ nnpdev->query_version_work.running = true;
+ nnpdev->query_version_work.protocol_version =
+ FIELD_GET(NNP_C2H_VERSION_REPLY_QW0_PROT_VER_MASK, msgbuf[0]);
+ nnpdev->query_version_work.chan_protocol_version =
+ FIELD_GET(NNP_C2H_VERSION_REPLY_QW0_CHAN_VER_MASK, msgbuf[0]);
+ nnpdev->query_version_work.chan_resp_op_size = msgbuf[1];
+ nnpdev->query_version_work.chan_cmd_op_size = msgbuf[2];
+
+ queue_work(nnpdev->wq, &nnpdev->query_version_work.work);
+
+ return msg_qwords;
+}
+
+/**
* handle_bios_protocol() - process response coming from card's BIOS.
* @nnpdev: The nnp device
* @msgbuf: pointer to response message content
@@ -58,6 +171,7 @@ typedef int (*response_handler)(struct nnp_device *nnpdev, const u64 *msgbuf,
int avail_qwords);
static response_handler resp_handlers[NNP_IPC_C2H_OPCODE_LAST + 1] = {
+ [NNP_IPC_C2H_OP_QUERY_VERSION_REPLY3] = handle_query_version_reply3,
[NNP_IPC_C2H_OP_BIOS_PROTOCOL] = handle_bios_protocol
};
@@ -329,6 +443,18 @@ void nnpdev_set_boot_state(struct nnp_device *nnpdev, u32 mask)
"Unexpected error while loading boot image. rc=%d\n",
ret);
}
+
+ /* Handle transition to active state */
+ if (((state & NNP_DEVICE_CARD_DRIVER_READY) ||
+ (state & NNP_DEVICE_CARD_READY)) &&
+ !(prev_state & NNP_DEVICE_CARD_DRIVER_READY) &&
+ !(prev_state & NNP_DEVICE_CARD_READY)) {
+ u32 val;
+
+ /* set host driver state to "Driver ready" */
+ val = FIELD_PREP(NNP_HOST_DRV_STATE_MASK, NNP_HOST_DRV_STATE_READY);
+ nnpdev->ops->set_host_doorbell_value(nnpdev, val);
+ }
}
/**
@@ -364,6 +490,7 @@ int nnpdev_init(struct nnp_device *nnpdev, struct device *dev,
*/
nnpdev->dev = dev;
nnpdev->ops = ops;
+ nnpdev->protocol_version = 0;
nnpdev->cmdq_sched = nnp_msched_create(nnpdev);
if (!nnpdev->cmdq_sched) {
@@ -397,6 +524,7 @@ int nnpdev_init(struct nnp_device *nnpdev, struct device *dev,
spin_lock_init(&nnpdev->lock);
nnpdev_boot_image_init(&nnpdev->boot_image);
+ INIT_WORK(&nnpdev->query_version_work.work, process_query_version_reply);
return 0;
@@ -426,6 +554,7 @@ static void doorbell_changed_handler(struct work_struct *work)
u32 error_state;
u32 doorbell_val = req->val;
struct nnp_device *nnpdev = req->nnpdev;
+ u64 query_cmd;
nnpdev->card_doorbell_val = doorbell_val;
@@ -466,6 +595,22 @@ static void doorbell_changed_handler(struct work_struct *work)
case NNP_CARD_BOOT_STATE_BIOS_FLASH_STARTED:
state = NNP_DEVICE_BIOS_UPDATE_STARTED;
break;
+ case NNP_CARD_BOOT_STATE_DRV_READY:
+ case NNP_CARD_BOOT_STATE_CARD_READY:
+ /* card is up - send "query_version" command */
+ query_cmd = FIELD_PREP(NNP_H2C_OP_MASK,
+ NNP_IPC_H2C_OP_QUERY_VERSION);
+ if (nnp_msched_queue_msg(nnpdev->cmdq, query_cmd) ||
+ nnp_msched_queue_sync(nnpdev->cmdq))
+ dev_err(nnpdev->dev, "Query version msg error\n");
+ break;
+
+ case NNP_CARD_BOOT_STATE_NOT_READY:
+ /* card is down reset the device boot and error state */
+ spin_lock(&nnpdev->lock);
+ nnpdev->state = 0;
+ spin_unlock(&nnpdev->lock);
+ break;
default:
break;
}
diff --git a/drivers/misc/intel-nnpi/device.h b/drivers/misc/intel-nnpi/device.h
index b83f67a..63bc54d 100644
--- a/drivers/misc/intel-nnpi/device.h
+++ b/drivers/misc/intel-nnpi/device.h
@@ -51,6 +51,15 @@
#define NNP_DEVICE_RESPONSE_FIFO_LEN 16
#define NNP_DEVICE_RESPONSE_BUFFER_LEN (NNP_DEVICE_RESPONSE_FIFO_LEN * 2)
+struct query_version_work {
+ struct work_struct work;
+ u64 chan_resp_op_size;
+ u64 chan_cmd_op_size;
+ u16 protocol_version;
+ u16 chan_protocol_version;
+ bool running;
+};
+
/**
* struct nnp_device - structure for NNP-I device info
* @ops: device operations implemented by the underlying device driver
@@ -71,9 +80,13 @@
* @bios_version_str: the device's started bios version string
* @bios_system_info_valid: true if @bios_system_info has been filled and valid
* @state: current device boot state mask (see device state bits above)
+ * @protocol_version: version of host->card IPC protocol
+ * @chan_protocol_version: version of user-space->card IPC protocol
* @curr_boot_state: last boot state field received from device doorbell reg
* @card_doorbell_val: last received device doorbell register value.
* @boot_image: boot image object used to boot the card
+ * @query_version_work: work struct used to schedule processing of version
+ * reply response message arrived from card.
*/
struct nnp_device {
const struct nnp_device_ops *ops;
@@ -98,8 +111,12 @@ struct nnp_device {
u32 state;
u32 curr_boot_state;
+ unsigned short protocol_version;
+ unsigned short chan_protocol_version;
u32 card_doorbell_val;
struct image_info boot_image;
+
+ struct query_version_work query_version_work;
};
/**
--
1.8.3.1
---------------------------------------------------------------------
Intel Israel (74) Limited
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
Route both types of messages coming from the NNP-I card - event
report messages, which are handled by the driver, and command-response
messages, which should be routed to a specific command channel object
("channel").
Event report messages are device-level messages which are not
associated with a specific channel. They are typically initiated by the
NNP-I card, to indicate an error, status change, or any event which
is not a response to a message sent from a spcific channel. These event
report messages are handled by the driver.
In contrast, command-response messages are associated with a specific
channel, and are typically sent from the NNP-I card in response to a
message sent from a channel to the card. These command-response
messages are added to the intended channel ring-buffer for consumption
by this channel.
The list of messages, of both types, coming from the card, is defined
in ipc_include/ipc_c2h_events.h included in this patch.
Signed-off-by: Guy Zadicario <[email protected]>
Reviewed-by: Alexander Shishkin <[email protected]>
---
drivers/misc/intel-nnpi/cmd_chan.h | 6 +
drivers/misc/intel-nnpi/device.c | 104 +++++++++++
drivers/misc/intel-nnpi/device.h | 2 +
.../misc/intel-nnpi/ipc_include/ipc_c2h_events.h | 198 +++++++++++++++++++++
4 files changed, 310 insertions(+)
create mode 100644 drivers/misc/intel-nnpi/ipc_include/ipc_c2h_events.h
diff --git a/drivers/misc/intel-nnpi/cmd_chan.h b/drivers/misc/intel-nnpi/cmd_chan.h
index 8cb1a5e..2be88c6 100644
--- a/drivers/misc/intel-nnpi/cmd_chan.h
+++ b/drivers/misc/intel-nnpi/cmd_chan.h
@@ -4,6 +4,7 @@
#ifndef NNPDRV_CMD_CHAN_H
#define NNPDRV_CMD_CHAN_H
+#include <linux/bitfield.h>
#include <linux/circ_buf.h>
#include <linux/hashtable.h>
#include <linux/kref.h>
@@ -12,6 +13,7 @@
#include <linux/spinlock.h>
#include "device.h"
+#include "ipc_c2h_events.h"
/**
* struct nnp_chan - structure object for user<->device communication
@@ -21,6 +23,7 @@
* @chan_id: the ipc channel id for this channel
* @hash_node: node to include this object in list of channels
* hash is in (cmd_chan_hash in nnp_device).
+ * @card_critical_error_msg: last critical event report received from device
* @get_device_events: true if device-level events received from card should
* be sent over this channel to user.
* @cmdq: message queue added to msg_scheduler, for user commands to be sent
@@ -43,6 +46,7 @@ struct nnp_chan {
struct nnp_device *nnpdev;
u16 chan_id;
struct hlist_node hash_node;
+ u64 card_critical_error_msg;
bool get_device_events;
struct nnp_msched_queue *cmdq;
@@ -59,6 +63,8 @@ struct nnp_chan {
unsigned int resp_lost;
};
+#define chan_broken(chan) FIELD_GET(NNP_C2H_EVENT_REPORT_CODE_MASK, (chan)->card_critical_error_msg)
+
struct nnp_chan *nnpdev_chan_create(struct nnp_device *nnpdev, int host_fd,
unsigned int min_id, unsigned int max_id,
bool get_device_events);
diff --git a/drivers/misc/intel-nnpi/device.c b/drivers/misc/intel-nnpi/device.c
index 3902876..1064edc 100644
--- a/drivers/misc/intel-nnpi/device.c
+++ b/drivers/misc/intel-nnpi/device.c
@@ -14,6 +14,7 @@
#include "cmd_chan.h"
#include "device.h"
#include "host_chardev.h"
+#include "ipc_c2h_events.h"
#include "msg_scheduler.h"
#include "nnp_boot_defs.h"
@@ -212,11 +213,113 @@ static void disconnect_all_channels(struct nnp_device *nnpdev)
spin_unlock(&nnpdev->lock);
}
+static void nnpdev_submit_device_event_to_channels(struct nnp_device *nnpdev,
+ u64 event_msg)
+{
+ struct nnp_chan *cmd_chan;
+ int i;
+ unsigned int event_code;
+ bool should_wake = false;
+ bool is_card_fatal;
+
+ event_code = FIELD_GET(NNP_C2H_EVENT_REPORT_CODE_MASK, event_msg);
+ is_card_fatal = is_card_fatal_event(event_code);
+
+ spin_lock(&nnpdev->lock);
+ hash_for_each(nnpdev->cmd_chan_hash, i, cmd_chan, hash_node) {
+ /*
+ * Update channel's card critical error,
+ * but do not override it if a more sever "fatal_drv" error
+ * event is already set.
+ */
+ if (is_card_fatal &&
+ !is_card_fatal_drv_event(chan_broken(cmd_chan))) {
+ cmd_chan->card_critical_error_msg = event_msg;
+ should_wake = true;
+ }
+
+ /* Send the event message to the channel (if needed) */
+ if (is_card_fatal || cmd_chan->get_device_events)
+ nnp_chan_add_response(cmd_chan, &event_msg, sizeof(event_msg));
+ }
+ spin_unlock(&nnpdev->lock);
+
+ if (should_wake)
+ wake_up_all(&nnpdev->waitq);
+}
+
+/*
+ * this function handle device-level event report message.
+ * which is usually affect the entire device and not a single channel
+ */
+static void process_device_event(struct nnp_device *nnpdev, u64 event_msg)
+{
+ /* submit the event to all channels requested to get device events */
+ nnpdev_submit_device_event_to_channels(nnpdev, event_msg);
+}
+
+struct event_report_work {
+ struct work_struct work;
+ struct nnp_device *nnpdev;
+ u64 event_msg;
+};
+
+static void device_event_report_handler(struct work_struct *work)
+{
+ struct event_report_work *req =
+ container_of(work, struct event_report_work, work);
+
+ process_device_event(req->nnpdev, req->event_msg);
+
+ kfree(req);
+}
+
+static int handle_event_report(struct nnp_device *nnpdev, const u64 *msgbuf,
+ int avail_qwords)
+{
+ int msg_qwords = 1; /* EVENT_REPORT response len is 1 qword */
+ struct event_report_work *req;
+ u64 event_msg;
+
+ if (avail_qwords < msg_qwords)
+ return 0;
+
+ event_msg = msgbuf[0];
+ if (FIELD_GET(NNP_C2H_EVENT_REPORT_CHAN_VALID_MASK, event_msg)) {
+ struct nnp_chan *cmd_chan;
+ unsigned int chan_id;
+
+ chan_id = FIELD_GET(NNP_C2H_EVENT_REPORT_CHAN_ID_MASK, event_msg);
+ cmd_chan = nnpdev_find_channel(nnpdev, chan_id);
+ if (cmd_chan) {
+ nnp_chan_add_response(cmd_chan, &event_msg, sizeof(event_msg));
+ nnp_chan_put(cmd_chan);
+ } else {
+ dev_dbg(nnpdev->dev,
+ "Got Event Report for non existing channel id %d\n",
+ chan_id);
+ }
+ return msg_qwords;
+ }
+
+ req = kzalloc(sizeof(*req), GFP_NOWAIT);
+ if (!req)
+ return msg_qwords;
+
+ req->event_msg = event_msg;
+ req->nnpdev = nnpdev;
+ INIT_WORK(&req->work, device_event_report_handler);
+ queue_work(nnpdev->wq, &req->work);
+
+ return msg_qwords;
+}
+
typedef int (*response_handler)(struct nnp_device *nnpdev, const u64 *msgbuf,
int avail_qwords);
static response_handler resp_handlers[NNP_IPC_C2H_OPCODE_LAST + 1] = {
[NNP_IPC_C2H_OP_QUERY_VERSION_REPLY3] = handle_query_version_reply3,
+ [NNP_IPC_C2H_OP_EVENT_REPORT] = handle_event_report,
[NNP_IPC_C2H_OP_BIOS_PROTOCOL] = handle_bios_protocol
};
@@ -593,6 +696,7 @@ int nnpdev_init(struct nnp_device *nnpdev, struct device *dev,
ida_init(&nnpdev->cmd_chan_ida);
hash_init(nnpdev->cmd_chan_hash);
+ init_waitqueue_head(&nnpdev->waitq);
nnpdev->cmdq_sched = nnp_msched_create(nnpdev);
if (!nnpdev->cmdq_sched) {
diff --git a/drivers/misc/intel-nnpi/device.h b/drivers/misc/intel-nnpi/device.h
index 9b6383e..c37f1da 100644
--- a/drivers/misc/intel-nnpi/device.h
+++ b/drivers/misc/intel-nnpi/device.h
@@ -79,6 +79,7 @@ struct query_version_work {
* @response_num_msgs: number of qwords available in @response_buf
* @cmd_chan_ida: allocate channel ids to be used in ipc protocol.
* @cmd_chan_hash: maps command channel id to its struct pointer.
+ * @waitq: used to wait for device response messages
* @bios_system_info_dma_addr: dma page allocated for bios system info.
* @bios_system_info: virtual pointer to bios system info page
* @bios_version_str: the device's started bios version string
@@ -113,6 +114,7 @@ struct nnp_device {
struct ida cmd_chan_ida;
DECLARE_HASHTABLE(cmd_chan_hash, 6);
+ wait_queue_head_t waitq;
dma_addr_t bios_system_info_dma_addr;
struct nnp_c2h_system_info *bios_system_info;
diff --git a/drivers/misc/intel-nnpi/ipc_include/ipc_c2h_events.h b/drivers/misc/intel-nnpi/ipc_include/ipc_c2h_events.h
new file mode 100644
index 0000000..5ca1b8e
--- /dev/null
+++ b/drivers/misc/intel-nnpi/ipc_include/ipc_c2h_events.h
@@ -0,0 +1,198 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright (C) 2019-2021 Intel Corporation */
+
+#ifndef _NNP_IPC_C2H_EVENTS_H
+#define _NNP_IPC_C2H_EVENTS_H
+
+/**
+ * The following describes the possible values for a c2h_event_report message
+ * sent from card to host to report on some error or other events.
+ *
+ * The c2h_event_report message has the following fields available to describe
+ * the event:
+ * event_code - 7 bits value describing the type of event
+ * event_val - 8 bits value - interpretation depends on event_code
+ * chan_id - the protocol id of the channel in which the event was
+ * occurred.
+ * obj_id - 16 bits, interpretation depends on event_code, usually used
+ * to hold an inference object protocol ID.
+ * obj_id_2 - 16 bits, in case obj_id is not enough to describe the object
+ * In this file we define the possible values for the above fields and document
+ * each field meaning for each possible event_code.
+ */
+
+/**
+ * Event codes ranges
+ *
+ * error codes are grouped into the following ranges:
+ * 0 - 3 ==> non error events generated by daemon/runtime
+ * 4 - 47 ==> non error events generated by card kernel driver
+ * 48 - 51 ==> non-critical error events generated by daemon/runtime
+ * 52 - 95 ==> non-critical error events generatd by kernel driver
+ * 96 - 103 ==> context-critical error events generated by daemon/runtime
+ * 104 - 111 ==> context-critical error events generated by kernel driver
+ * 112 - 119 ==> card-critical error events generated by daemon/runtime
+ * 120 - 127 ==> card-critical error events generated by kernel driver
+ *
+ * context-critical error event is one that puts the infer context in an
+ * un-recoverable error state.
+ * card-critical error event is one that make the card not useful for inference
+ * request until it is reset.
+ */
+#define EVENT_NON_ERR_START 0
+#define EVENT_NON_ERR_DRV_START 4
+#define EVENT_ERR_START 48
+#define EVENT_ERR_DRV_START 52
+#define EVENT_CONTEXT_FATAL_START 96
+#define EVENT_CONTEXT_FATAL_DRV_START 104
+#define EVENT_CARD_FATAL_START 112
+#define EVENT_CARD_FATAL_DRV_START 120
+
+#define is_context_fatal_event(e) ({ \
+ int e_ = (e); \
+ (e_ >= EVENT_CONTEXT_FATAL_START && e_ < EVENT_CARD_FATAL_START); \
+})
+
+#define is_card_fatal_event(e) ((e) >= EVENT_CARD_FATAL_START)
+#define is_card_fatal_drv_event(e) ((e) >= EVENT_CARD_FATAL_DRV_START)
+
+#define NNP_IPC_RUNTIME_DONE (EVENT_NON_ERR_START + 1)
+/* MAX offset for EVENT_NON_ERR_START is 3 */
+
+/* non-error event codes */
+#define NNP_IPC_CREATE_CONTEXT_SUCCESS (EVENT_NON_ERR_DRV_START + 0)
+#define NNP_IPC_CREATE_DEVRES_SUCCESS (EVENT_NON_ERR_DRV_START + 1)
+#define NNP_IPC_CREATE_COPY_SUCCESS (EVENT_NON_ERR_DRV_START + 2)
+#define NNP_IPC_EXECUTE_COPY_SUCCESS (EVENT_NON_ERR_DRV_START + 3)
+#define NNP_IPC_DEVRES_DESTROYED (EVENT_NON_ERR_DRV_START + 4)
+#define NNP_IPC_COPY_DESTROYED (EVENT_NON_ERR_DRV_START + 5)
+#define NNP_IPC_CONTEXT_DESTROYED (EVENT_NON_ERR_DRV_START + 6)
+#define NNP_IPC_CREATE_DEVNET_SUCCESS (EVENT_NON_ERR_DRV_START + 7)
+#define NNP_IPC_DEVNET_DESTROYED (EVENT_NON_ERR_DRV_START + 8)
+#define NNP_IPC_CREATE_INFREQ_SUCCESS (EVENT_NON_ERR_DRV_START + 9)
+#define NNP_IPC_INFREQ_DESTROYED (EVENT_NON_ERR_DRV_START + 10)
+#define NNP_IPC_RECOVER_CONTEXT_SUCCESS (EVENT_NON_ERR_DRV_START + 11)
+#define NNP_IPC_THERMAL_TRIP_EVENT (EVENT_NON_ERR_DRV_START + 12)
+#define NNP_IPC_DEVNET_ADD_RES_SUCCESS (EVENT_NON_ERR_DRV_START + 13)
+#define NNP_IPC_DEVICE_STATE_CHANGED (EVENT_NON_ERR_DRV_START + 14)
+#define NNP_IPC_DEVNET_RESOURCES_RESERVATION_SUCCESS \
+ (EVENT_NON_ERR_DRV_START + 15)
+#define NNP_IPC_DEVNET_RESOURCES_RELEASE_SUCCESS (EVENT_NON_ERR_DRV_START + 16)
+#define NNP_IPC_CREATE_CHANNEL_SUCCESS (EVENT_NON_ERR_DRV_START + 17)
+#define NNP_IPC_CHANNEL_DESTROYED (EVENT_NON_ERR_DRV_START + 18)
+#define NNP_IPC_CHANNEL_SET_RB_SUCCESS (EVENT_NON_ERR_DRV_START + 19)
+#define NNP_IPC_CHANNEL_MAP_HOSTRES_SUCCESS (EVENT_NON_ERR_DRV_START + 20)
+#define NNP_IPC_CHANNEL_UNMAP_HOSTRES_SUCCESS (EVENT_NON_ERR_DRV_START + 21)
+#define NNP_IPC_ABORT_REQUEST (EVENT_NON_ERR_DRV_START + 22)
+#define NNP_IPC_GET_FIFO (EVENT_NON_ERR_DRV_START + 23)
+#define NNP_IPC_CREATE_CMD_SUCCESS (EVENT_NON_ERR_DRV_START + 24)
+#define NNP_IPC_CMD_DESTROYED (EVENT_NON_ERR_DRV_START + 25)
+#define NNP_IPC_EXECUTE_CMD_COMPLETE (EVENT_NON_ERR_DRV_START + 26)
+#define NNP_IPC_DEVNET_SET_PROPERTY_SUCCESS (EVENT_NON_ERR_DRV_START + 27)
+#define NNP_IPC_EXECUTE_CPYLST_SUCCESS (EVENT_NON_ERR_DRV_START + 28)
+#define NNP_IPC_GET_CR_FIFO_REPLY (EVENT_NON_ERR_DRV_START + 29)
+#define NNP_IPC_P2P_PEERS_CONNECTED (EVENT_NON_ERR_DRV_START + 30)
+#define NNP_IPC_P2P_PEER_DEV_UPDATED (EVENT_NON_ERR_DRV_START + 31)
+#define NNP_IPC_EXECUTE_COPY_SUBRES_SUCCESS (EVENT_NON_ERR_DRV_START + 32)
+/* MAX offset for EVENT_NON_ERR_DRV_START is 43 */
+
+/* non-critical error event codes */
+#define NNP_IPC_CREATE_CONTEXT_FAILED (EVENT_ERR_DRV_START + 0)
+#define NNP_IPC_CREATE_DEVRES_FAILED (EVENT_ERR_DRV_START + 1)
+#define NNP_IPC_CREATE_COPY_FAILED (EVENT_ERR_DRV_START + 2)
+#define NNP_IPC_DESTROY_CONTEXT_FAILED (EVENT_ERR_DRV_START + 3)
+#define NNP_IPC_DESTROY_DEVRES_FAILED (EVENT_ERR_DRV_START + 4)
+#define NNP_IPC_DESTROY_COPY_FAILED (EVENT_ERR_DRV_START + 5)
+#define NNP_IPC_CREATE_SYNC_FAILED (EVENT_ERR_DRV_START + 6)
+#define NNP_IPC_ERROR_SUB_RESOURCE_LOAD_FAILED (EVENT_ERR_DRV_START + 7)
+#define NNP_IPC_CREATE_DEVNET_FAILED (EVENT_ERR_DRV_START + 8)
+#define NNP_IPC_DESTROY_DEVNET_FAILED (EVENT_ERR_DRV_START + 9)
+#define NNP_IPC_CREATE_INFREQ_FAILED (EVENT_ERR_DRV_START + 10)
+#define NNP_IPC_DESTROY_INFREQ_FAILED (EVENT_ERR_DRV_START + 11)
+#define NNP_IPC_RECOVER_CONTEXT_FAILED (EVENT_ERR_DRV_START + 12)
+#define NNP_IPC_ERROR_MCE_CORRECTABLE (EVENT_ERR_DRV_START + 13)
+#define NNP_IPC_ERROR_MCE_UNCORRECTABLE (EVENT_ERR_DRV_START + 14)
+#define NNP_IPC_DEVNET_ADD_RES_FAILED (EVENT_ERR_DRV_START + 15)
+#define NNP_IPC_DEVNET_RESOURCES_RESERVATION_FAILED (EVENT_ERR_DRV_START + 16)
+#define NNP_IPC_DEVNET_RESOURCES_RELEASE_FAILED (EVENT_ERR_DRV_START + 17)
+#define NNP_IPC_CREATE_CHANNEL_FAILED (EVENT_ERR_DRV_START + 18)
+#define NNP_IPC_DESTROY_CHANNEL_FAILED (EVENT_ERR_DRV_START + 19)
+#define NNP_IPC_CHANNEL_SET_RB_FAILED (EVENT_ERR_DRV_START + 20)
+#define NNP_IPC_CREATE_CMD_FAILED (EVENT_ERR_DRV_START + 21)
+#define NNP_IPC_DESTROY_CMD_FAILED (EVENT_ERR_DRV_START + 22)
+#define NNP_IPC_CHANNEL_MAP_HOSTRES_FAILED (EVENT_ERR_DRV_START + 23)
+#define NNP_IPC_CHANNEL_UNMAP_HOSTRES_FAILED (EVENT_ERR_DRV_START + 24)
+#define NNP_IPC_DEVNET_SET_PROPERTY_FAILED (EVENT_ERR_DRV_START + 25)
+#define NNP_IPC_ERROR_DRAM_ECC_CORRECTABLE (EVENT_ERR_DRV_START + 26)
+#define NNP_IPC_EXECUTE_COPY_FAILED (EVENT_ERR_DRV_START + 27)
+#define NNP_IPC_SCHEDULE_INFREQ_FAILED (EVENT_ERR_DRV_START + 28)
+#define NNP_IPC_EXECUTE_CPYLST_FAILED (EVENT_ERR_DRV_START + 29)
+#define NNP_IPC_EXECUTE_COPY_SUBRES_FAILED (EVENT_ERR_DRV_START + 30)
+#define NNP_IPC_EC_FAILED_TO_RELEASE_CREDIT (EVENT_ERR_DRV_START + 31)
+#define NNP_IPC_DMA_HANG_DETECTED (EVENT_ERR_DRV_START + 32)
+/* MAX offset for EVENT_ERR_DRV_START is 43 */
+
+/* context critical error event codes */
+#define NNP_IPC_ERROR_RUNTIME_LAUNCH (EVENT_CONTEXT_FATAL_START + 0)
+#define NNP_IPC_ERROR_RUNTIME_DIED (EVENT_CONTEXT_FATAL_START + 1)
+/* MAX offset for EVENT_CONTEXT_FATAL_START is 7 */
+
+#define NNP_IPC_CONTEXT_EXEC_ERROR (EVENT_CONTEXT_FATAL_DRV_START + 0)
+#define NNP_IPC_CTX_DRAM_ECC_UNCORRECTABLE (EVENT_CONTEXT_FATAL_DRV_START + 1)
+/* MAX offset for EVENT_CONTEXT_FATAL_DRV_START is 7 */
+
+/* card critical error event codes */
+#define NNP_IPC_ERROR_OS_CRASHED (EVENT_CARD_FATAL_START + 0)
+#define NNP_IPC_ERROR_DRAM_ECC_UNCORRECTABLE_FATAL (EVENT_CARD_FATAL_START + 1)
+#define NNP_IPC_ERROR_FATAL_ICE_ERROR (EVENT_CARD_FATAL_START + 2)
+/* MAX offset for EVENT_CARD_FATAL_START is 7 */
+
+/* card critical and driver fatal*/
+#define NNP_IPC_ERROR_PCI_ERROR (EVENT_CARD_FATAL_DRV_START + 0)
+#define NNP_IPC_ERROR_MCE_UNCORRECTABLE_FATAL (EVENT_CARD_FATAL_DRV_START + 1)
+#define NNP_IPC_ERROR_CARD_RESET (EVENT_CARD_FATAL_DRV_START + 2)
+#define NNP_IPC_ERROR_CHANNEL_KILLED (EVENT_CARD_FATAL_DRV_START + 3)
+#define NNP_IPC_ERROR_PROTOCOL_ERROR (EVENT_CARD_FATAL_DRV_START + 4)
+#define NNP_IPC_FATAL_DMA_HANG_DETECTED (EVENT_CARD_FATAL_DRV_START + 5)
+/* MAX offset for EVENT_CARD_FATAL_DRV_START is 7 */
+
+enum event_val {
+ NNP_IPC_NO_ERROR = 0,
+ NNP_IPC_NO_SUCH_CONTEXT = 1,
+ NNP_IPC_NO_SUCH_DEVRES = 2,
+ NNP_IPC_NO_SUCH_COPY = 3,
+ NNP_IPC_NO_SUCH_NET = 4,
+ NNP_IPC_NO_SUCH_INFREQ = 5,
+ NNP_IPC_ALREADY_EXIST = 6,
+ NNP_IPC_NO_DAEMON = 7,
+ NNP_IPC_NO_MEMORY = 8,
+ NNP_IPC_RUNTIME_FAILED = 9,
+ NNP_IPC_RUNTIME_LAUNCH_FAILED = 10,
+ NNP_IPC_DMA_ERROR = 11,
+ NNP_IPC_RUNTIME_NOT_SUPPORTED = 12,
+ NNP_IPC_RUNTIME_INVALID_EXECUTABLE_NETWORK_BINARY = 13,
+ NNP_IPC_RUNTIME_INFER_MISSING_RESOURCE = 14,
+ NNP_IPC_RUNTIME_INFER_EXEC_ERROR = 15,
+ NNP_IPC_RUNTIME_INFER_SCHEDULE_ERROR = 16,
+ NNP_IPC_CONTEXT_BROKEN = 17,
+ NNP_IPC_DEVNET_RESERVE_INSUFFICIENT_RESOURCES = 18,
+ NNP_IPC_TIMEOUT_EXCEEDED = 19,
+ NNP_IPC_ECC_ALLOC_FAILED = 20,
+ NNP_IPC_NO_SUCH_CHANNEL = 21,
+ NNP_IPC_NO_SUCH_CMD = 22,
+ NNP_IPC_NO_SUCH_HOSTRES = 23,
+ NNP_IPC_DEVNET_EDIT_BUSY = 24,
+ NNP_IPC_DEVNET_EDIT_ERROR = 25,
+ NNP_IPC_NOT_SUPPORTED = 26,
+ NNP_IPC_ICEDRV_INFER_EXEC_ERROR = 27,
+ NNP_IPC_ICEDRV_INFER_EXEC_ERROR_NEED_RESET = 28,
+ NNP_IPC_ICEDRV_INFER_EXEC_ERROR_NEED_CARD_RESET = 29,
+ NNP_IPC_NO_EXEC_ERRORS = 30,
+ NNP_IPC_IO_ERROR = 31,
+ NNP_IPC_INPUT_IS_DIRTY = 32,
+
+ /* Non failure events */
+ NNP_IPC_CMDLIST_FINISHED = 128,
+};
+
+#endif
--
1.8.3.1
---------------------------------------------------------------------
Intel Israel (74) Limited
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
Expose a character device for each NNP-I device (/dev/nnpi%d) with
IOCTL interface. Using this character device, user-space can create a
command channel object, through which it can send and receive messages
to and from the device.
Signed-off-by: Guy Zadicario <[email protected]>
Reviewed-by: Alexander Shishkin <[email protected]>
---
drivers/misc/intel-nnpi/Makefile | 2 +-
drivers/misc/intel-nnpi/cmd_chan.c | 11 +
drivers/misc/intel-nnpi/cmd_chan.h | 1 +
drivers/misc/intel-nnpi/device.c | 50 ++++-
drivers/misc/intel-nnpi/device.h | 11 +
drivers/misc/intel-nnpi/device_chardev.c | 348 +++++++++++++++++++++++++++++++
drivers/misc/intel-nnpi/device_chardev.h | 14 ++
include/uapi/misc/intel_nnpi.h | 43 ++++
8 files changed, 478 insertions(+), 2 deletions(-)
create mode 100644 drivers/misc/intel-nnpi/device_chardev.c
create mode 100644 drivers/misc/intel-nnpi/device_chardev.h
diff --git a/drivers/misc/intel-nnpi/Makefile b/drivers/misc/intel-nnpi/Makefile
index b3bab2a..a294cf0c 100644
--- a/drivers/misc/intel-nnpi/Makefile
+++ b/drivers/misc/intel-nnpi/Makefile
@@ -6,7 +6,7 @@
obj-$(CONFIG_INTEL_NNPI) := intel_nnpi.o intel_nnpi_pcie.o
intel_nnpi-y := device.o msg_scheduler.o hostres.o host_chardev.o nnp_user.o \
- bootimage.o cmd_chan.o
+ bootimage.o cmd_chan.o device_chardev.o
intel_nnpi_pcie-y := nnp_pcie.o
diff --git a/drivers/misc/intel-nnpi/cmd_chan.c b/drivers/misc/intel-nnpi/cmd_chan.c
index 89ae604..0ad281a 100644
--- a/drivers/misc/intel-nnpi/cmd_chan.c
+++ b/drivers/misc/intel-nnpi/cmd_chan.c
@@ -557,6 +557,17 @@ void nnp_chan_disconnect(struct nnp_chan *cmd_chan)
cmd_chan->state = NNP_CHAN_DESTROYED;
mutex_unlock(&cmd_chan->dev_mutex);
+ /*
+ * If the channel is not in critical state,
+ * put it in critical state and wake any user
+ * which might wait for the device.
+ */
+ if (!chan_drv_fatal(cmd_chan)) {
+ cmd_chan->card_critical_error_msg = FIELD_PREP(NNP_C2H_EVENT_REPORT_CODE_MASK,
+ NNP_IPC_ERROR_CHANNEL_KILLED);
+ wake_up_all(&nnpdev->waitq);
+ }
+
wake_up_all(&cmd_chan->resp_waitq);
nnp_msched_queue_sync(cmd_chan->cmdq);
nnp_msched_queue_destroy(cmd_chan->cmdq);
diff --git a/drivers/misc/intel-nnpi/cmd_chan.h b/drivers/misc/intel-nnpi/cmd_chan.h
index d60abf4..3eb5c1c 100644
--- a/drivers/misc/intel-nnpi/cmd_chan.h
+++ b/drivers/misc/intel-nnpi/cmd_chan.h
@@ -80,6 +80,7 @@ struct nnp_chan {
};
#define chan_broken(chan) FIELD_GET(NNP_C2H_EVENT_REPORT_CODE_MASK, (chan)->card_critical_error_msg)
+#define chan_drv_fatal(chan) (is_card_fatal_drv_event(chan_broken(chan)))
struct nnp_chan *nnpdev_chan_create(struct nnp_device *nnpdev, int host_fd,
unsigned int min_id, unsigned int max_id,
diff --git a/drivers/misc/intel-nnpi/device.c b/drivers/misc/intel-nnpi/device.c
index ece19c0..17746d2 100644
--- a/drivers/misc/intel-nnpi/device.c
+++ b/drivers/misc/intel-nnpi/device.c
@@ -13,6 +13,7 @@
#include "bootimage.h"
#include "cmd_chan.h"
#include "device.h"
+#include "device_chardev.h"
#include "host_chardev.h"
#include "ipc_c2h_events.h"
#include "msg_scheduler.h"
@@ -259,6 +260,22 @@ static void nnpdev_submit_device_event_to_channels(struct nnp_device *nnpdev,
disconnect_all_channels(nnpdev);
}
+static void handle_channel_create_response(struct nnp_device *nnpdev, u64 event_msg)
+{
+ struct nnp_chan *cmd_chan;
+ unsigned int chan_id;
+
+ chan_id = FIELD_GET(NNP_C2H_EVENT_REPORT_OBJ_ID_MASK, event_msg);
+
+ cmd_chan = nnpdev_find_channel(nnpdev, chan_id);
+ if (!cmd_chan)
+ return;
+
+ cmd_chan->event_msg = event_msg;
+ nnp_chan_put(cmd_chan);
+ wake_up_all(&nnpdev->waitq);
+}
+
static void handle_channel_destroy(struct nnp_device *nnpdev, u64 event_msg)
{
struct nnp_chan *cmd_chan;
@@ -300,6 +317,10 @@ static void process_device_event(struct nnp_device *nnpdev, u64 event_msg)
if (!is_card_fatal_event(event_code)) {
switch (event_code) {
+ case NNP_IPC_CREATE_CHANNEL_SUCCESS:
+ case NNP_IPC_CREATE_CHANNEL_FAILED:
+ handle_channel_create_response(nnpdev, event_msg);
+ break;
case NNP_IPC_DESTROY_CHANNEL_FAILED:
obj_id = FIELD_GET(NNP_C2H_EVENT_REPORT_OBJ_ID_MASK, event_msg);
event_val = FIELD_GET(NNP_C2H_EVENT_REPORT_VAL_MASK, event_msg);
@@ -796,6 +817,11 @@ int nnpdev_init(struct nnp_device *nnpdev, struct device *dev,
goto err_wq;
}
+ /* Create the character device interface to this device */
+ ret = nnpdev_cdev_create(nnpdev);
+ if (ret)
+ goto err_sys_info;
+
/* set host driver state to "Not ready" */
nnpdev->ops->set_host_doorbell_value(nnpdev, 0);
@@ -805,6 +831,9 @@ int nnpdev_init(struct nnp_device *nnpdev, struct device *dev,
return 0;
+err_sys_info:
+ dma_free_coherent(nnpdev->dev, NNP_PAGE_SIZE, nnpdev->bios_system_info,
+ nnpdev->bios_system_info_dma_addr);
err_wq:
destroy_workqueue(nnpdev->wq);
err_cmdq:
@@ -946,6 +975,10 @@ void nnpdev_destroy(struct nnp_device *nnpdev)
destroy_workqueue(nnpdev->wq);
disconnect_all_channels(nnpdev);
+
+ /* destroy character device */
+ nnpdev_cdev_destroy(nnpdev);
+
dma_free_coherent(nnpdev->dev, NNP_PAGE_SIZE, nnpdev->bios_system_info,
nnpdev->bios_system_info_dma_addr);
@@ -960,13 +993,28 @@ void nnpdev_destroy(struct nnp_device *nnpdev)
static int __init nnp_init(void)
{
- return nnp_init_host_interface();
+ int ret;
+
+ ret = nnp_init_host_interface();
+ if (ret)
+ return ret;
+
+ ret = nnpdev_cdev_class_init();
+ if (ret)
+ goto err_class;
+
+ return 0;
+
+err_class:
+ nnp_release_host_interface();
+ return ret;
}
subsys_initcall(nnp_init);
static void __exit nnp_cleanup(void)
{
nnp_release_host_interface();
+ nnpdev_cdev_class_cleanup();
/* dev_ida is already empty here - no point calling ida_destroy */
}
module_exit(nnp_cleanup);
diff --git a/drivers/misc/intel-nnpi/device.h b/drivers/misc/intel-nnpi/device.h
index c37f1da..e7e66d6 100644
--- a/drivers/misc/intel-nnpi/device.h
+++ b/drivers/misc/intel-nnpi/device.h
@@ -4,8 +4,10 @@
#ifndef _NNPDRV_DEVICE_H
#define _NNPDRV_DEVICE_H
+#include <linux/cdev.h>
#include <linux/hashtable.h>
#include <linux/idr.h>
+#include <linux/list.h>
#include <linux/spinlock.h>
#include <linux/workqueue.h>
@@ -53,6 +55,9 @@
#define NNP_DEVICE_RESPONSE_FIFO_LEN 16
#define NNP_DEVICE_RESPONSE_BUFFER_LEN (NNP_DEVICE_RESPONSE_FIFO_LEN * 2)
+#define NNP_MAX_CHANNEL_ID 1023 /* has 10 bits in ipc protocol */
+#define NNP_MAX_INF_CONTEXT_CHANNEL_ID 255 /* [0, 255] are reserved for inference contexts */
+
struct query_version_work {
struct work_struct work;
u64 chan_resp_op_size;
@@ -92,6 +97,9 @@ struct query_version_work {
* @boot_image: boot image object used to boot the card
* @query_version_work: work struct used to schedule processing of version
* reply response message arrived from card.
+ * @cdev: cdev object of NNP-I device char dev.
+ * @chardev: character device for this device
+ * @cdev_clients: list of opened struct file to the chardev of this device.
* @ipc_chan_resp_op_size: holds response size for each possible channel
* response.
* @ipc_chan_cmd_op_size: holds command size for each possible channel command.
@@ -130,6 +138,9 @@ struct nnp_device {
struct query_version_work query_version_work;
+ struct cdev cdev;
+ struct device *chardev;
+ struct list_head cdev_clients;
u8 ipc_chan_resp_op_size[NNP_IPC_NUM_USER_OPS];
u8 ipc_chan_cmd_op_size[NNP_IPC_NUM_USER_OPS];
};
diff --git a/drivers/misc/intel-nnpi/device_chardev.c b/drivers/misc/intel-nnpi/device_chardev.c
new file mode 100644
index 0000000..e4bb168
--- /dev/null
+++ b/drivers/misc/intel-nnpi/device_chardev.c
@@ -0,0 +1,348 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (C) 2019-2021 Intel Corporation */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/bitfield.h>
+#include <linux/cdev.h>
+#include <linux/device.h>
+#include <linux/kref.h>
+#include <linux/list.h>
+#include <linux/printk.h>
+#include <linux/slab.h>
+
+#include <uapi/misc/intel_nnpi.h>
+
+#include "cmd_chan.h"
+#include "device_chardev.h"
+#include "ipc_c2h_events.h"
+
+static dev_t devnum;
+static struct class *class;
+
+/**
+ * struct device_client - structure for opened device char device file
+ * @node: list node to include this struct in a list of clients
+ * (nnpdev->cdev_clients).
+ * @nnpdev: the NNP-I device associated with the opened chardev
+ * @mutex: protects @nnpdev
+ *
+ * NOTE: @nnpdev may become NULL if the underlying NNP-I device has removed.
+ * Any ioctl request on the char device in this state will fail with
+ * -ENODEV
+ */
+struct device_client {
+ struct list_head node;
+ struct nnp_device *nnpdev;
+ struct mutex mutex;
+};
+
+/* protects nnpdev->cdev_clients list (for all nnp devices) */
+static DEFINE_MUTEX(clients_mutex);
+
+#define NNPDRV_DEVICE_DEV_NAME "nnpi"
+
+static inline bool is_nnp_device_file(struct file *f);
+
+static int nnp_device_open(struct inode *inode, struct file *f)
+{
+ struct device_client *client;
+ struct nnp_device *nnpdev;
+
+ if (!is_nnp_device_file(f))
+ return -EINVAL;
+
+ if (!inode->i_cdev)
+ return -EINVAL;
+
+ client = kzalloc(sizeof(*client), GFP_KERNEL);
+ if (!client)
+ return -ENOMEM;
+
+ nnpdev = container_of(inode->i_cdev, struct nnp_device, cdev);
+ client->nnpdev = nnpdev;
+ mutex_init(&client->mutex);
+ f->private_data = client;
+
+ mutex_lock(&clients_mutex);
+ list_add_tail(&client->node, &nnpdev->cdev_clients);
+ mutex_unlock(&clients_mutex);
+
+ return 0;
+}
+
+static void disconnect_client_locked(struct device_client *client)
+{
+ lockdep_assert_held(&clients_mutex);
+
+ mutex_lock(&client->mutex);
+ if (!client->nnpdev) {
+ mutex_unlock(&client->mutex);
+ return;
+ }
+ client->nnpdev = NULL;
+ list_del(&client->node);
+ mutex_unlock(&client->mutex);
+}
+
+static int nnp_device_release(struct inode *inode, struct file *f)
+{
+ struct device_client *client = f->private_data;
+
+ if (!is_nnp_device_file(f))
+ return -EINVAL;
+
+ mutex_lock(&clients_mutex);
+ disconnect_client_locked(client);
+ mutex_unlock(&clients_mutex);
+ kfree(client);
+ f->private_data = NULL;
+
+ return 0;
+}
+
+static int event_val_to_nnp_error(enum event_val event_val)
+{
+ switch (event_val) {
+ case NNP_IPC_NO_ERROR:
+ return 0;
+ case NNP_IPC_NO_MEMORY:
+ return -ENOMEM;
+ default:
+ return -EFAULT;
+ }
+}
+
+static int send_create_chan_req(struct nnp_device *nnpdev, struct nnp_chan *chan)
+{
+ unsigned int event_code, event_val;
+ u64 cmd;
+ int ret;
+
+ cmd = FIELD_PREP(NNP_H2C_OP_MASK, NNP_IPC_H2C_OP_CHANNEL_OP);
+ cmd |= FIELD_PREP(NNP_H2C_CHANNEL_OP_CHAN_ID_MASK, chan->chan_id);
+ cmd |= FIELD_PREP(NNP_H2C_CHANNEL_OP_UID_MASK, 0);
+ cmd |= FIELD_PREP(NNP_H2C_CHANNEL_OP_PRIV_MASK, 1);
+
+ ret = nnp_msched_queue_msg(nnpdev->cmdq, cmd);
+ if (ret < 0)
+ return NNPER_DEVICE_ERROR;
+
+ /*
+ * wait until card has respond to the create request or fatal
+ * card error has been detected.
+ */
+ wait_event(nnpdev->waitq, chan->event_msg || chan_drv_fatal(chan));
+ if (!chan->event_msg)
+ return NNPER_DEVICE_ERROR;
+
+ event_code = FIELD_GET(NNP_C2H_EVENT_REPORT_CODE_MASK, chan->event_msg);
+ event_val = FIELD_GET(NNP_C2H_EVENT_REPORT_VAL_MASK, chan->event_msg);
+ if (event_code == NNP_IPC_CREATE_CHANNEL_FAILED)
+ return event_val_to_nnp_error(event_val);
+
+ return 0;
+}
+
+static long create_channel(struct device_client *cinfo, void __user *arg,
+ unsigned int size)
+{
+ struct nnp_device *nnpdev = cinfo->nnpdev;
+ struct ioctl_nnpi_create_channel req;
+ unsigned int io_size = sizeof(req);
+ struct nnp_chan *chan;
+ long ret = 0;
+ u32 error_mask;
+
+ /* only single size structure is currently supported */
+ if (size != io_size)
+ return -EINVAL;
+
+ if (copy_from_user(&req, arg, io_size))
+ return -EFAULT;
+
+ /* o_errno must be cleared on entry */
+ if (req.o_errno)
+ return -EINVAL;
+
+ if (req.i_max_id < req.i_min_id ||
+ req.i_max_id > NNP_MAX_CHANNEL_ID)
+ return -EINVAL;
+
+ /*
+ * Do not allow create command channel if device is in
+ * error state.
+ * However allow new non infer context channels in case
+ * of fatal ICE error in order to allow retrieve debug
+ * information.
+ */
+ error_mask = NNP_DEVICE_ERROR_MASK;
+ if (req.i_max_id > NNP_MAX_INF_CONTEXT_CHANNEL_ID)
+ error_mask &= ~(NNP_DEVICE_FATAL_ICE_ERROR);
+
+ if ((nnpdev->state & error_mask) ||
+ !(nnpdev->state & NNP_DEVICE_CARD_DRIVER_READY) ||
+ (req.i_max_id <= NNP_MAX_INF_CONTEXT_CHANNEL_ID &&
+ (nnpdev->state & NNP_DEVICE_ACTIVE_MASK) !=
+ NNP_DEVICE_ACTIVE_MASK)) {
+ req.o_errno = NNPER_DEVICE_NOT_READY;
+ goto done;
+ }
+
+ /* Validate channel protocol version */
+ if (NNP_VERSION_MAJOR(req.i_protocol_version) !=
+ NNP_VERSION_MAJOR(nnpdev->chan_protocol_version) ||
+ NNP_VERSION_MINOR(req.i_protocol_version) !=
+ NNP_VERSION_MINOR(nnpdev->chan_protocol_version)) {
+ req.o_errno = NNPER_VERSIONS_MISMATCH;
+ goto done;
+ }
+
+ /* create the channel object */
+ chan = nnpdev_chan_create(nnpdev, req.i_host_fd, req.i_min_id, req.i_max_id,
+ req.i_get_device_events);
+ if (IS_ERR(chan)) {
+ ret = PTR_ERR(chan);
+ goto done;
+ }
+
+ /* create the channel on card */
+ req.o_errno = send_create_chan_req(nnpdev, chan);
+ if (req.o_errno)
+ goto err_destroy;
+
+ req.o_channel_id = chan->chan_id;
+
+ /* Attach file descriptor to the channel object */
+ req.o_fd = nnp_chan_create_file(chan);
+
+ /* remove channel object if failed */
+ if (req.o_fd < 0) {
+ /* the channel already created on card - send a destroy request */
+ nnp_chan_send_destroy(chan);
+ ret = req.o_fd;
+ }
+
+ goto done;
+
+err_destroy:
+ /* the channel was not created on card - destroy it now */
+ if (!nnp_chan_set_destroyed(chan))
+ nnp_chan_put(chan);
+done:
+ if (!ret && copy_to_user(arg, &req, io_size))
+ return -EFAULT;
+
+ return ret;
+}
+
+static long nnp_device_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
+{
+ struct device_client *client = f->private_data;
+ unsigned int ioc_nr, size;
+ long ret;
+
+ if (!is_nnp_device_file(f))
+ return -ENOTTY;
+
+ if (_IOC_TYPE(cmd) != 'D')
+ return -EINVAL;
+
+ mutex_lock(&client->mutex);
+ if (!client->nnpdev) {
+ mutex_unlock(&client->mutex);
+ return -ENODEV;
+ }
+
+ ioc_nr = _IOC_NR(cmd);
+ size = _IOC_SIZE(cmd);
+
+ switch (ioc_nr) {
+ case _IOC_NR(IOCTL_NNPI_DEVICE_CREATE_CHANNEL):
+ ret = create_channel(client, (void __user *)arg, size);
+ break;
+ default:
+ ret = -EINVAL;
+ break;
+ }
+
+ mutex_unlock(&client->mutex);
+
+ return ret;
+}
+
+static const struct file_operations nnp_device_fops = {
+ .owner = THIS_MODULE,
+ .open = nnp_device_open,
+ .release = nnp_device_release,
+ .unlocked_ioctl = nnp_device_ioctl,
+ .compat_ioctl = nnp_device_ioctl,
+};
+
+static inline bool is_nnp_device_file(struct file *f)
+{
+ return f->f_op == &nnp_device_fops;
+}
+
+int nnpdev_cdev_create(struct nnp_device *nnpdev)
+{
+ int ret;
+
+ INIT_LIST_HEAD(&nnpdev->cdev_clients);
+
+ cdev_init(&nnpdev->cdev, &nnp_device_fops);
+ nnpdev->cdev.owner = THIS_MODULE;
+ ret = cdev_add(&nnpdev->cdev, MKDEV(MAJOR(devnum), nnpdev->id), 1);
+ if (ret)
+ return ret;
+
+ nnpdev->chardev = device_create(class, NULL, MKDEV(MAJOR(devnum), nnpdev->id),
+ nnpdev, NNPI_DEVICE_DEV_FMT, nnpdev->id);
+ if (IS_ERR(nnpdev->chardev)) {
+ cdev_del(&nnpdev->cdev);
+ return PTR_ERR(nnpdev->chardev);
+ }
+
+ return 0;
+}
+
+void nnpdev_cdev_destroy(struct nnp_device *nnpdev)
+{
+ struct device_client *client, *tmp;
+
+ device_destroy(class, MKDEV(MAJOR(devnum), nnpdev->id));
+
+ /* disconnect all chardev clients from the device */
+ mutex_lock(&clients_mutex);
+ list_for_each_entry_safe(client, tmp, &nnpdev->cdev_clients, node)
+ disconnect_client_locked(client);
+ mutex_unlock(&clients_mutex);
+
+ cdev_del(&nnpdev->cdev);
+}
+
+int nnpdev_cdev_class_init(void)
+{
+ int ret;
+
+ ret = alloc_chrdev_region(&devnum, 0, NNP_MAX_DEVS,
+ NNPDRV_DEVICE_DEV_NAME);
+ if (ret < 0)
+ return ret;
+
+ class = class_create(THIS_MODULE, NNPDRV_DEVICE_DEV_NAME);
+ if (IS_ERR(class)) {
+ ret = PTR_ERR(class);
+ unregister_chrdev_region(devnum, NNP_MAX_DEVS);
+ return ret;
+ }
+
+ return 0;
+}
+
+void nnpdev_cdev_class_cleanup(void)
+{
+ class_destroy(class);
+ unregister_chrdev_region(devnum, NNP_MAX_DEVS);
+}
+
diff --git a/drivers/misc/intel-nnpi/device_chardev.h b/drivers/misc/intel-nnpi/device_chardev.h
new file mode 100644
index 0000000..0db919d
--- /dev/null
+++ b/drivers/misc/intel-nnpi/device_chardev.h
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright (C) 2019-2021 Intel Corporation */
+
+#ifndef _NNPDRV_DEVICE_CHARDEV_H
+#define _NNPDRV_DEVICE_CHARDEV_H
+
+#include "device.h"
+
+int nnpdev_cdev_create(struct nnp_device *nnpdev);
+void nnpdev_cdev_destroy(struct nnp_device *nnpdev);
+int nnpdev_cdev_class_init(void);
+void nnpdev_cdev_class_cleanup(void);
+
+#endif
diff --git a/include/uapi/misc/intel_nnpi.h b/include/uapi/misc/intel_nnpi.h
index 5114aea..620a5d4 100644
--- a/include/uapi/misc/intel_nnpi.h
+++ b/include/uapi/misc/intel_nnpi.h
@@ -134,6 +134,49 @@ struct nnpdrv_ioctl_destroy_hostres {
__u32 o_errno;
};
+/*
+ * ioctls for /dev/nnpi%d device
+ */
+#define NNPI_DEVICE_DEV_FMT "nnpi%u"
+
+/**
+ * IOCTL_NNPI_DEVICE_CREATE_CHANNEL:
+ *
+ * A request to create a new communication "channel" with an NNP-I device.
+ * This channel can be used to send command and receive responses from the
+ * device.
+ */
+#define IOCTL_NNPI_DEVICE_CREATE_CHANNEL \
+ _IOWR('D', 0, struct ioctl_nnpi_create_channel)
+
+/**
+ * struct ioctl_nnpi_create_channel - IOCTL_NNPI_DEVICE_CREATE_CHANNEL payload
+ * @i_host_fd: opened file descriptor to /dev/nnpi_host
+ * @i_min_id: minimum range for channel id allocation
+ * @i_max_id: maximum range for channel id allocation
+ * @i_get_device_events: if true, device-level event responses will be
+ * delivered to be read from the channel.
+ * @i_protocol_version: The NNP_IPC_CHAN_PROTOCOL_VERSION the user-space has
+ * compiled with.
+ * @o_fd: returns file-descriptor through which commands/responses can be
+ * write/read.
+ * @o_errno: On input, must be set to 0.
+ * On output, 0 on success, one of the NNPERR_* error codes on error.
+ * @o_channel_id: returns the unique id of the channel
+ *
+ * Argument structure for IOCTL_NNPI_DEVICE_CREATE_CHANNEL ioctl.
+ */
+struct ioctl_nnpi_create_channel {
+ __s32 i_host_fd;
+ __u32 i_min_id;
+ __u32 i_max_id;
+ __s32 i_get_device_events;
+ __u32 i_protocol_version;
+ __s32 o_fd;
+ __u32 o_errno;
+ __u16 o_channel_id;
+};
+
/****************************************************************
* Error code values - errors returned in o_errno fields of
* above structures.
--
1.8.3.1
---------------------------------------------------------------------
Intel Israel (74) Limited
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
Provide an IOCTL interface for mapping and unmapping host resources to a
channel, through the device's /dev/nnpi%d char device. The mapping gets a
uniqueue ID and the page list of the host resource is transferred to the
device. Later, commands to the device can reference the resource by the
channel ID and map ID.
There is a special interface to map host resources which serve as
host-to-card and card-to-host ring buffers. These ring buffers can be
referenced later by the ring-buffer direction and index, rather than by
a map ID.
Signed-off-by: Guy Zadicario <[email protected]>
Reviewed-by: Alexander Shishkin <[email protected]>
---
drivers/misc/intel-nnpi/cmd_chan.c | 103 ++++++++
drivers/misc/intel-nnpi/cmd_chan.h | 37 ++-
drivers/misc/intel-nnpi/device.c | 58 +++-
drivers/misc/intel-nnpi/device_chardev.c | 441 +++++++++++++++++++++++++++++++
include/uapi/misc/intel_nnpi.h | 111 ++++++++
5 files changed, 748 insertions(+), 2 deletions(-)
diff --git a/drivers/misc/intel-nnpi/cmd_chan.c b/drivers/misc/intel-nnpi/cmd_chan.c
index 0ad281a..d98d02b 100644
--- a/drivers/misc/intel-nnpi/cmd_chan.c
+++ b/drivers/misc/intel-nnpi/cmd_chan.c
@@ -383,6 +383,9 @@ struct nnp_chan *nnpdev_chan_create(struct nnp_device *nnpdev, int host_fd,
init_waitqueue_head(&cmd_chan->resp_waitq);
mutex_init(&cmd_chan->dev_mutex);
+ ida_init(&cmd_chan->hostres_map_ida);
+ hash_init(cmd_chan->hostres_hash);
+
/*
* Add channel to the channel hash
*/
@@ -411,6 +414,11 @@ static void nnp_chan_release(struct kref *kref)
nnp_chan_disconnect(cmd_chan);
/*
+ * cmd_chan->hostres_map_ida is empty at this point,
+ * calling ida_destroy is not needed
+ */
+
+ /*
* If a chan file was created (through nnp_chan_create_file),
* the host_file was already put when the file has released, otherwise
* we put it here.
@@ -542,6 +550,9 @@ int nnp_chan_send_destroy(struct nnp_chan *chan)
void nnp_chan_disconnect(struct nnp_chan *cmd_chan)
{
struct nnp_device *nnpdev;
+ struct chan_hostres_map *hostres_map;
+ struct hlist_node *tmp;
+ int i;
mutex_lock(&cmd_chan->dev_mutex);
if (!cmd_chan->nnpdev) {
@@ -573,6 +584,32 @@ void nnp_chan_disconnect(struct nnp_chan *cmd_chan)
nnp_msched_queue_destroy(cmd_chan->cmdq);
ida_simple_remove(&nnpdev->cmd_chan_ida, cmd_chan->chan_id);
+
+ /*
+ * Unmap ring buffers
+ */
+ for (i = 0; i < NNP_IPC_MAX_CHANNEL_RB; i++) {
+ if (cmd_chan->h2c_rb_hostres_map[i]) {
+ nnp_hostres_unmap_device(cmd_chan->h2c_rb_hostres_map[i]);
+ cmd_chan->h2c_rb_hostres_map[i] = NULL;
+ }
+ if (cmd_chan->c2h_rb_hostres_map[i]) {
+ nnp_hostres_unmap_device(cmd_chan->c2h_rb_hostres_map[i]);
+ cmd_chan->c2h_rb_hostres_map[i] = NULL;
+ }
+ }
+
+ /*
+ * Destroy all host resource maps
+ */
+ mutex_lock(&cmd_chan->dev_mutex);
+ hash_for_each_safe(cmd_chan->hostres_hash, i, tmp, hostres_map, hash_node) {
+ hash_del(&hostres_map->hash_node);
+ ida_simple_remove(&cmd_chan->hostres_map_ida, hostres_map->id);
+ nnp_hostres_unmap_device(hostres_map->hostres_map);
+ kfree(hostres_map);
+ }
+ mutex_unlock(&cmd_chan->dev_mutex);
}
static int resize_respq(struct nnp_chan *cmd_chan)
@@ -685,3 +722,69 @@ int nnp_chan_add_response(struct nnp_chan *cmd_chan, u64 *hw_msg, u32 size)
return ret;
}
+
+int nnp_chan_set_ringbuf(struct nnp_chan *chan, bool h2c, unsigned int id,
+ struct nnpdev_mapping *hostres_map)
+{
+ if (id >= NNP_IPC_MAX_CHANNEL_RB)
+ return -EINVAL;
+
+ mutex_lock(&chan->dev_mutex);
+ if (h2c) {
+ if (chan->h2c_rb_hostres_map[id])
+ nnp_hostres_unmap_device(chan->h2c_rb_hostres_map[id]);
+ chan->h2c_rb_hostres_map[id] = hostres_map;
+ } else {
+ if (chan->c2h_rb_hostres_map[id])
+ nnp_hostres_unmap_device(chan->c2h_rb_hostres_map[id]);
+ chan->c2h_rb_hostres_map[id] = hostres_map;
+ }
+ mutex_unlock(&chan->dev_mutex);
+
+ return 0;
+}
+
+static struct chan_hostres_map *find_map(struct nnp_chan *chan, unsigned int map_id)
+{
+ struct chan_hostres_map *hostres_map;
+
+ lockdep_assert_held(&chan->dev_mutex);
+
+ hash_for_each_possible(chan->hostres_hash, hostres_map, hash_node,
+ map_id)
+ if (hostres_map->id == map_id)
+ return hostres_map;
+
+ return NULL;
+}
+
+struct chan_hostres_map *nnp_chan_find_map(struct nnp_chan *chan, unsigned int map_id)
+{
+ struct chan_hostres_map *map;
+
+ mutex_lock(&chan->dev_mutex);
+ map = find_map(chan, map_id);
+ mutex_unlock(&chan->dev_mutex);
+
+ return map;
+}
+
+int nnp_chan_unmap_hostres(struct nnp_chan *chan, unsigned int map_id)
+{
+ struct chan_hostres_map *hostres_map;
+
+ mutex_lock(&chan->dev_mutex);
+ hostres_map = find_map(chan, map_id);
+ if (!hostres_map) {
+ mutex_unlock(&chan->dev_mutex);
+ return -ENXIO;
+ }
+ hash_del(&hostres_map->hash_node);
+ ida_simple_remove(&chan->hostres_map_ida, hostres_map->id);
+ nnp_hostres_unmap_device(hostres_map->hostres_map);
+ mutex_unlock(&chan->dev_mutex);
+
+ kfree(hostres_map);
+
+ return 0;
+}
diff --git a/drivers/misc/intel-nnpi/cmd_chan.h b/drivers/misc/intel-nnpi/cmd_chan.h
index 3eb5c1c..2913595 100644
--- a/drivers/misc/intel-nnpi/cmd_chan.h
+++ b/drivers/misc/intel-nnpi/cmd_chan.h
@@ -19,10 +19,13 @@
* enum nnp_chan_state - indicate special state of a command channel
* @NNP_CHAN_NORMAL: channel is in normal state.
* @NNP_CHAN_DESTROYED: channel should be treated as no-longer-exist on card.
+ * @NNP_CHAN_RB_OP_IN_FLIGHT: a ring-buffer create or destroy command has been
+ * sent to device and response did not yet arrived.
*/
enum nnp_chan_state {
NNP_CHAN_NORMAL = 0,
NNP_CHAN_DESTROYED,
+ NNP_CHAN_RB_OP_IN_FLIGHT,
};
/**
@@ -45,14 +48,20 @@ enum nnp_chan_state {
* @nnp_user: the nnp_user this channel belongs to.
* the channel can reference host resources created by this
* nnp_user object.
- * @dev_mutex: protects @nnpdev and @state
+ * @dev_mutex: protects @nnpdev, @state, @hostres_hash and @hostres_map_ida
* @state: the current state of this channel.
+ * @hostres_map_ida: generate ipc ids for hostres mapping
+ * @hostres_hash: hash table to store all host resource mapping, key is ipc id
* @resp_waitq: waitqueue used for waiting for response messages be available.
* @respq: circular buffer object that receive response messages from device.
* @respq_lock: protects @respq
* @respq_buf: buffer space allocated for circular response buffer.
* @respq_size: current allocated size of circular response buffer.
* @resp_lost: number of response messages lost due to response buffer full.
+ * @h2c_rb_hostres_map: host resource mapping used for each host-to-card ring buffer
+ * There may be up to 2 such ring buffers, both can be NULL.
+ * @c2h_rb_hostres_map: host resource mapping used for each card-to-host ring buffer
+ * There may be up to 2 such ring buffers, both can be NULL.
*/
struct nnp_chan {
struct kref ref;
@@ -72,11 +81,32 @@ struct nnp_chan {
wait_queue_head_t resp_waitq;
enum nnp_chan_state state;
+ struct ida hostres_map_ida;
+ DECLARE_HASHTABLE(hostres_hash, 6);
+
struct circ_buf respq;
spinlock_t respq_lock;
char *respq_buf;
unsigned int respq_size;
unsigned int resp_lost;
+
+ struct nnpdev_mapping *h2c_rb_hostres_map[NNP_IPC_MAX_CHANNEL_RB];
+ struct nnpdev_mapping *c2h_rb_hostres_map[NNP_IPC_MAX_CHANNEL_RB];
+};
+
+/**
+ * struct chan_hostres_map - holds host resource mapping to channel
+ *
+ * @id: ipc map id of the mapping
+ * @hash_node: node to include this mapping in @hostres_hash of nnpdrv_cmd_chan
+ * @hostres_map: the host resource mapping object
+ * @event_msg: device response to the map create request
+ */
+struct chan_hostres_map {
+ unsigned int id;
+ struct hlist_node hash_node;
+ struct nnpdev_mapping *hostres_map;
+ u64 event_msg;
};
#define chan_broken(chan) FIELD_GET(NNP_C2H_EVENT_REPORT_CODE_MASK, (chan)->card_critical_error_msg)
@@ -96,4 +126,9 @@ struct nnp_chan *nnpdev_chan_create(struct nnp_device *nnpdev, int host_fd,
int nnp_chan_add_response(struct nnp_chan *cmd_chan, u64 *hw_msg, u32 size);
+int nnp_chan_set_ringbuf(struct nnp_chan *chan, bool h2c, unsigned int id,
+ struct nnpdev_mapping *hostres_map);
+
+struct chan_hostres_map *nnp_chan_find_map(struct nnp_chan *chan, unsigned int map_id);
+int nnp_chan_unmap_hostres(struct nnp_chan *chan, unsigned int map_id);
#endif
diff --git a/drivers/misc/intel-nnpi/device.c b/drivers/misc/intel-nnpi/device.c
index 17746d2..69ff555 100644
--- a/drivers/misc/intel-nnpi/device.c
+++ b/drivers/misc/intel-nnpi/device.c
@@ -276,6 +276,45 @@ static void handle_channel_create_response(struct nnp_device *nnpdev, u64 event_
wake_up_all(&nnpdev->waitq);
}
+static void handle_channel_map_hostres(struct nnp_device *nnpdev, u64 event_msg)
+{
+ struct chan_hostres_map *hostres_map;
+ unsigned int chan_id, map_id;
+ struct nnp_chan *cmd_chan;
+
+ chan_id = FIELD_GET(NNP_C2H_EVENT_REPORT_OBJ_ID_MASK, event_msg);
+ cmd_chan = nnpdev_find_channel(nnpdev, chan_id);
+ if (!cmd_chan)
+ return;
+
+ map_id = FIELD_GET(NNP_C2H_EVENT_REPORT_OBJ_ID2_MASK, event_msg);
+ hostres_map = nnp_chan_find_map(cmd_chan, map_id);
+ if (!hostres_map)
+ goto put_chan;
+
+ hostres_map->event_msg = event_msg;
+ wake_up_all(&nnpdev->waitq);
+
+put_chan:
+ nnp_chan_put(cmd_chan);
+}
+
+static void handle_channel_unmap_hostres(struct nnp_device *nnpdev, u64 event_msg)
+{
+ unsigned int chan_id, map_id;
+ struct nnp_chan *cmd_chan;
+
+ chan_id = FIELD_GET(NNP_C2H_EVENT_REPORT_OBJ_ID_MASK, event_msg);
+ cmd_chan = nnpdev_find_channel(nnpdev, chan_id);
+ if (!cmd_chan)
+ return;
+
+ map_id = FIELD_GET(NNP_C2H_EVENT_REPORT_OBJ_ID2_MASK, event_msg);
+ nnp_chan_unmap_hostres(cmd_chan, map_id);
+
+ nnp_chan_put(cmd_chan);
+}
+
static void handle_channel_destroy(struct nnp_device *nnpdev, u64 event_msg)
{
struct nnp_chan *cmd_chan;
@@ -313,14 +352,20 @@ static void handle_channel_destroy(struct nnp_device *nnpdev, u64 event_msg)
static void process_device_event(struct nnp_device *nnpdev, u64 event_msg)
{
unsigned int event_code = FIELD_GET(NNP_C2H_EVENT_REPORT_CODE_MASK, event_msg);
- unsigned int obj_id, event_val;
+ unsigned int obj_id, event_val, obj_id_2;
if (!is_card_fatal_event(event_code)) {
switch (event_code) {
case NNP_IPC_CREATE_CHANNEL_SUCCESS:
case NNP_IPC_CREATE_CHANNEL_FAILED:
+ case NNP_IPC_CHANNEL_SET_RB_SUCCESS:
+ case NNP_IPC_CHANNEL_SET_RB_FAILED:
handle_channel_create_response(nnpdev, event_msg);
break;
+ case NNP_IPC_CHANNEL_MAP_HOSTRES_SUCCESS:
+ case NNP_IPC_CHANNEL_MAP_HOSTRES_FAILED:
+ handle_channel_map_hostres(nnpdev, event_msg);
+ break;
case NNP_IPC_DESTROY_CHANNEL_FAILED:
obj_id = FIELD_GET(NNP_C2H_EVENT_REPORT_OBJ_ID_MASK, event_msg);
event_val = FIELD_GET(NNP_C2H_EVENT_REPORT_VAL_MASK, event_msg);
@@ -339,6 +384,17 @@ static void process_device_event(struct nnp_device *nnpdev, u64 event_msg)
case NNP_IPC_CHANNEL_DESTROYED:
handle_channel_destroy(nnpdev, event_msg);
break;
+ case NNP_IPC_CHANNEL_UNMAP_HOSTRES_FAILED:
+ obj_id = FIELD_GET(NNP_C2H_EVENT_REPORT_OBJ_ID_MASK, event_msg);
+ obj_id_2 = FIELD_GET(NNP_C2H_EVENT_REPORT_OBJ_ID2_MASK, event_msg);
+ event_val = FIELD_GET(NNP_C2H_EVENT_REPORT_VAL_MASK, event_msg);
+ dev_dbg(nnpdev->dev,
+ "Channel hostres unmap failed on device channel %d map %d val %d\n",
+ obj_id, obj_id_2, event_val);
+ fallthrough;
+ case NNP_IPC_CHANNEL_UNMAP_HOSTRES_SUCCESS:
+ handle_channel_unmap_hostres(nnpdev, event_msg);
+ break;
default:
dev_err(nnpdev->dev,
"Unknown event received - %u\n", event_code);
diff --git a/drivers/misc/intel-nnpi/device_chardev.c b/drivers/misc/intel-nnpi/device_chardev.c
index e4bb168..623352d 100644
--- a/drivers/misc/intel-nnpi/device_chardev.c
+++ b/drivers/misc/intel-nnpi/device_chardev.c
@@ -7,6 +7,7 @@
#include <linux/cdev.h>
#include <linux/device.h>
#include <linux/kref.h>
+#include <linux/dma-map-ops.h>
#include <linux/list.h>
#include <linux/printk.h>
#include <linux/slab.h>
@@ -15,6 +16,7 @@
#include "cmd_chan.h"
#include "device_chardev.h"
+#include "nnp_user.h"
#include "ipc_c2h_events.h"
static dev_t devnum;
@@ -236,6 +238,431 @@ static long create_channel(struct device_client *cinfo, void __user *arg,
return ret;
}
+/**
+ * send_rb_op() - sends CHANNEL_RB_OP command and wait for reply
+ * @chan: the command channel
+ * @rb_op_cmd: the command to send
+ * @o_errno: returns zero or error code from device
+ *
+ * The function sends a "ring buffer operation" command to the device
+ * to either create or destroy a ring buffer object.
+ * This is a synchronous operation, the function will wait until a response
+ * from the device has arrived.
+ * If some other synchronous ring buffer operation is already in progress on
+ * the same channel, the function will fail.
+ *
+ * Return:
+ * * -EBUSY: Ring-buffer create/destroy operation is already in-flight.
+ * * -EPIPE: The channel is in critical error state or sending the command
+ * has failed.
+ * * 0: The command has sent successfully, the operation status is updated
+ * in o_errno, if o_errno is zero, then the create/destoy operation has
+ * succeeded, otherwise it indicates an error code received from
+ * device.
+ */
+static int send_rb_op(struct nnp_chan *chan, u64 rb_op_cmd, __u32 *o_errno)
+{
+ struct nnp_device *nnpdev = chan->nnpdev;
+ unsigned int event_code, event_val;
+ int ret = -EPIPE;
+
+ *o_errno = 0;
+
+ mutex_lock(&chan->dev_mutex);
+ if (chan->state == NNP_CHAN_RB_OP_IN_FLIGHT) {
+ mutex_unlock(&chan->dev_mutex);
+ return -EBUSY;
+ } else if (chan->state == NNP_CHAN_DESTROYED) {
+ mutex_unlock(&chan->dev_mutex);
+ *o_errno = NNPER_DEVICE_ERROR;
+ return 0;
+ }
+ chan->state = NNP_CHAN_RB_OP_IN_FLIGHT;
+ mutex_unlock(&chan->dev_mutex);
+
+ chan->event_msg = 0;
+
+ /* send the command to card */
+ if (!chan_drv_fatal(chan))
+ ret = nnp_msched_queue_msg(nnpdev->cmdq, rb_op_cmd);
+
+ if (ret < 0)
+ goto done;
+
+ /* wait until card respond or card critical error is detected */
+ wait_event(nnpdev->waitq, chan->event_msg || chan_drv_fatal(chan));
+ if (!chan->event_msg) {
+ *o_errno = NNPER_DEVICE_ERROR;
+ ret = 0;
+ goto done;
+ }
+
+ event_code = FIELD_GET(NNP_C2H_EVENT_REPORT_CODE_MASK, chan->event_msg);
+ if (event_code == NNP_IPC_CHANNEL_SET_RB_FAILED) {
+ event_val = FIELD_GET(NNP_C2H_EVENT_REPORT_VAL_MASK, chan->event_msg);
+ *o_errno = event_val_to_nnp_error(event_val);
+ ret = 0;
+ }
+
+done:
+ mutex_lock(&chan->dev_mutex);
+ if (chan->state == NNP_CHAN_RB_OP_IN_FLIGHT)
+ chan->state = NNP_CHAN_NORMAL;
+ mutex_unlock(&chan->dev_mutex);
+ return ret;
+}
+
+static long create_channel_data_ringbuf(struct device_client *cinfo,
+ void __user *arg, unsigned int size)
+{
+ struct nnp_device *nnpdev = cinfo->nnpdev;
+ struct ioctl_nnpi_create_channel_data_ringbuf req;
+ struct user_hostres *hostres_entry = NULL;
+ struct nnp_user_info *nnp_user = NULL;
+ struct nnpdev_mapping *hostres_map;
+ unsigned int io_size = sizeof(req);
+ struct host_resource *hostres;
+ struct nnp_chan *chan = NULL;
+ unsigned long dma_pfn;
+ dma_addr_t page_list;
+ u64 rb_op_cmd;
+ int ret = 0;
+
+ /* only single size structure is currently supported */
+ if (size != io_size)
+ return -EINVAL;
+
+ if (copy_from_user(&req, arg, io_size))
+ return -EFAULT;
+
+ if (req.i_id > NNP_IPC_MAX_CHANNEL_RB - 1)
+ return -EINVAL;
+
+ /* o_errno must be cleared on entry */
+ if (req.o_errno)
+ return -EINVAL;
+
+ chan = nnpdev_find_channel(nnpdev, req.i_channel_id);
+ if (!chan) {
+ req.o_errno = NNPER_NO_SUCH_CHANNEL;
+ goto done;
+ }
+
+ nnp_user = chan->nnp_user;
+ mutex_lock(&nnp_user->mutex);
+ hostres_entry = idr_find(&nnp_user->idr, req.i_hostres_handle);
+ if (!hostres_entry) {
+ req.o_errno = NNPER_NO_SUCH_RESOURCE;
+ goto unlock_user;
+ }
+
+ hostres = hostres_entry->hostres;
+
+ /* check the resource fit the direction */
+ if ((req.i_h2c && !nnp_hostres_is_input(hostres)) ||
+ (!req.i_h2c && !nnp_hostres_is_output(hostres))) {
+ req.o_errno = NNPER_INCOMPATIBLE_RESOURCES;
+ goto unlock_user;
+ }
+
+ hostres_map = nnp_hostres_map_device(hostres, nnpdev, false, &page_list, NULL);
+ if (IS_ERR(hostres_map)) {
+ ret = -EFAULT;
+ goto unlock_user;
+ }
+
+ /*
+ * Its OK to release the mutex here and let other
+ * thread destroy the hostres handle as we already
+ * mapped it (which ref counted)
+ */
+ mutex_unlock(&nnp_user->mutex);
+
+ dma_pfn = NNP_IPC_DMA_ADDR_TO_PFN(page_list);
+ rb_op_cmd = FIELD_PREP(NNP_H2C_OP_MASK, NNP_IPC_H2C_OP_CHANNEL_RB_OP);
+ rb_op_cmd |= FIELD_PREP(NNP_H2C_CHANNEL_RB_OP_CHAN_ID_MASK, chan->chan_id);
+ rb_op_cmd |= FIELD_PREP(NNP_H2C_CHANNEL_RB_OP_ID_MASK, req.i_id);
+ rb_op_cmd |= FIELD_PREP(NNP_H2C_CHANNEL_RB_OP_HOST_PFN_MASK, dma_pfn);
+ if (req.i_h2c)
+ rb_op_cmd |= FIELD_PREP(NNP_H2C_CHANNEL_RB_OP_H2C_MASK, 1);
+
+ ret = send_rb_op(chan, rb_op_cmd, &req.o_errno);
+ if (!ret && !req.o_errno)
+ ret = nnp_chan_set_ringbuf(chan, req.i_h2c, req.i_id, hostres_map);
+
+ if (ret || req.o_errno)
+ nnp_hostres_unmap_device(hostres_map);
+
+ goto put_chan;
+
+unlock_user:
+ mutex_unlock(&nnp_user->mutex);
+put_chan:
+ nnp_chan_put(chan);
+done:
+ if (!ret && copy_to_user(arg, &req, io_size))
+ return -EFAULT;
+
+ return ret;
+}
+
+static long destroy_channel_data_ringbuf(struct device_client *cinfo,
+ void __user *arg, unsigned int size)
+{
+ struct ioctl_nnpi_destroy_channel_data_ringbuf req;
+ struct nnp_device *nnpdev = cinfo->nnpdev;
+ unsigned int io_size = sizeof(req);
+ struct nnp_chan *chan;
+ u64 rb_op_cmd;
+ int ret = 0;
+
+ /* only single size structure is currently supported */
+ if (size != io_size)
+ return -EINVAL;
+
+ if (copy_from_user(&req, arg, io_size))
+ return -EFAULT;
+
+ /* we have one bit in ipc protocol for ringbuf id for each direction */
+ if (req.i_id > 1)
+ return -EINVAL;
+
+ /* o_errno must be cleared on entry */
+ if (req.o_errno)
+ return -EINVAL;
+
+ chan = nnpdev_find_channel(nnpdev, req.i_channel_id);
+ if (!chan) {
+ req.o_errno = NNPER_NO_SUCH_CHANNEL;
+ goto done;
+ }
+
+ rb_op_cmd = FIELD_PREP(NNP_H2C_OP_MASK, NNP_IPC_H2C_OP_CHANNEL_RB_OP);
+ rb_op_cmd |= FIELD_PREP(NNP_H2C_CHANNEL_RB_OP_CHAN_ID_MASK, chan->chan_id);
+ rb_op_cmd |= FIELD_PREP(NNP_H2C_CHANNEL_RB_OP_ID_MASK, req.i_id);
+ rb_op_cmd |= FIELD_PREP(NNP_H2C_CHANNEL_RB_OP_DESTROY_MASK, 1);
+ if (req.i_h2c)
+ rb_op_cmd |= FIELD_PREP(NNP_H2C_CHANNEL_RB_OP_H2C_MASK, 1);
+
+ ret = send_rb_op(chan, rb_op_cmd, &req.o_errno);
+ if (ret || req.o_errno)
+ goto put_chan;
+
+ ret = nnp_chan_set_ringbuf(chan, req.i_h2c, req.i_id, NULL);
+
+put_chan:
+ nnp_chan_put(chan);
+done:
+ if (!ret && copy_to_user(arg, &req, io_size))
+ return -EFAULT;
+
+ return ret;
+}
+
+static int send_map_hostres_req(struct nnp_device *nnpdev, struct nnp_chan *chan,
+ struct chan_hostres_map *hostres_map, dma_addr_t page_list)
+{
+ unsigned int event_code, event_val;
+ unsigned long dma_pfn;
+ u64 cmd[2];
+
+ dma_pfn = NNP_IPC_DMA_ADDR_TO_PFN(page_list);
+ cmd[0] = FIELD_PREP(NNP_H2C_OP_MASK, NNP_IPC_H2C_OP_CHANNEL_HOSTRES_OP);
+ cmd[0] |= FIELD_PREP(NNP_H2C_CHANNEL_HOSTRES_QW0_CHAN_ID_MASK,
+ chan->chan_id);
+ cmd[0] |= FIELD_PREP(NNP_H2C_CHANNEL_HOSTRES_QW0_ID_MASK, hostres_map->id);
+ cmd[1] = FIELD_PREP(NNP_H2C_CHANNEL_HOSTRES_QW1_HOST_PFN_MASK, dma_pfn);
+
+ /* do not send the map command if the device in a fatal error state */
+ if (chan_drv_fatal(chan))
+ return NNPER_DEVICE_ERROR;
+
+ /* send the hostres map command to card */
+ if (nnp_msched_queue_msg(chan->cmdq, cmd) < 0)
+ return NNPER_DEVICE_ERROR;
+
+ /* wait until card respond or card critical error is detected */
+ wait_event(nnpdev->waitq, hostres_map->event_msg || chan_drv_fatal(chan));
+
+ if (!hostres_map->event_msg)
+ return NNPER_DEVICE_ERROR;
+
+ event_code = FIELD_GET(NNP_C2H_EVENT_REPORT_CODE_MASK, hostres_map->event_msg);
+ if (event_code == NNP_IPC_CHANNEL_MAP_HOSTRES_FAILED) {
+ event_val = FIELD_GET(NNP_C2H_EVENT_REPORT_VAL_MASK, hostres_map->event_msg);
+ return event_val_to_nnp_error(event_val);
+ }
+
+ return 0;
+}
+
+static int do_map_hostres(struct nnp_device *nnpdev, struct nnp_chan *chan,
+ unsigned long hostres_handle)
+{
+ struct chan_hostres_map *hostres_map = NULL;
+ struct user_hostres *hostres_entry = NULL;
+ struct nnp_user_info *nnp_user;
+ struct host_resource *hostres;
+ dma_addr_t page_list;
+ int map_id;
+ int err;
+
+ nnp_user = chan->nnp_user;
+ mutex_lock(&nnp_user->mutex);
+ hostres_entry = idr_find(&nnp_user->idr, hostres_handle);
+ if (!hostres_entry) {
+ err = -NNPER_NO_SUCH_RESOURCE;
+ goto unlock_user;
+ }
+ hostres = hostres_entry->hostres;
+
+ hostres_map = kzalloc(sizeof(*hostres_map), GFP_KERNEL);
+ if (!hostres_map) {
+ err = -ENOMEM;
+ goto unlock_user;
+ }
+
+ mutex_lock(&chan->dev_mutex);
+ map_id = ida_simple_get(&chan->hostres_map_ida, 0, U16_MAX, GFP_KERNEL);
+ if (map_id < 0) {
+ err = -ENOMEM;
+ goto err_map;
+ }
+
+ hostres_map->hostres_map = nnp_hostres_map_device(hostres, nnpdev,
+ false, &page_list, NULL);
+ if (IS_ERR(hostres_map->hostres_map)) {
+ err = -EFAULT;
+ goto err_ida;
+ }
+
+ hostres_map->event_msg = 0;
+ hostres_map->id = map_id;
+
+ hash_add(chan->hostres_hash, &hostres_map->hash_node, hostres_map->id);
+ mutex_unlock(&chan->dev_mutex);
+ mutex_unlock(&nnp_user->mutex);
+
+ err = send_map_hostres_req(nnpdev, chan, hostres_map, page_list);
+ if (err) {
+ nnp_chan_unmap_hostres(chan, hostres_map->id);
+ return err;
+ }
+
+ return map_id;
+
+err_ida:
+ ida_simple_remove(&chan->hostres_map_ida, map_id);
+err_map:
+ mutex_unlock(&chan->dev_mutex);
+ kfree(hostres_map);
+unlock_user:
+ mutex_unlock(&nnp_user->mutex);
+ return err;
+}
+
+static long map_hostres(struct device_client *cinfo, void __user *arg,
+ unsigned int size)
+{
+ struct nnp_device *nnpdev = cinfo->nnpdev;
+ struct ioctl_nnpi_channel_map_hostres req;
+ unsigned int io_size = sizeof(req);
+ const struct dma_map_ops *ops;
+ struct nnp_chan *chan = NULL;
+ int ret;
+
+ /* only single size structure is currently supported */
+ if (size != io_size)
+ return -EINVAL;
+
+ if (copy_from_user(&req, arg, io_size))
+ return -EFAULT;
+
+ chan = nnpdev_find_channel(nnpdev, req.i_channel_id);
+ if (!chan) {
+ req.o_errno = NNPER_NO_SUCH_CHANNEL;
+ goto done;
+ }
+
+ ret = do_map_hostres(nnpdev, chan, req.i_hostres_handle);
+ if (ret < 0) {
+ req.o_errno = -ret;
+ goto put_chan;
+ }
+
+ req.o_errno = 0;
+ req.o_map_id = ret;
+
+ ops = get_dma_ops(nnpdev->dev);
+ if (ops)
+ req.o_sync_needed = ops->sync_sg_for_cpu ? 1 : 0;
+ else
+ req.o_sync_needed =
+ !dev_is_dma_coherent(nnpdev->dev);
+
+ goto put_chan;
+
+put_chan:
+ nnp_chan_put(chan);
+done:
+ if (copy_to_user(arg, &req, io_size))
+ return -EFAULT;
+
+ return 0;
+}
+
+static long unmap_hostres(struct device_client *cinfo, void __user *arg,
+ unsigned int size)
+{
+ struct ioctl_nnpi_channel_unmap_hostres req;
+ struct nnp_device *nnpdev = cinfo->nnpdev;
+ struct chan_hostres_map *hostres_map;
+ unsigned int io_size = sizeof(req);
+ struct nnp_chan *chan = NULL;
+ u64 cmd[2];
+ long ret = 0;
+
+ /* only single size structure is currently supported */
+ if (size != io_size)
+ return -EINVAL;
+
+ if (copy_from_user(&req, arg, io_size))
+ return -EFAULT;
+
+ /* o_errno must be cleared on entry */
+ if (req.o_errno)
+ return -EINVAL;
+
+ chan = nnpdev_find_channel(nnpdev, req.i_channel_id);
+ if (!chan) {
+ req.o_errno = NNPER_NO_SUCH_CHANNEL;
+ goto done;
+ }
+
+ hostres_map = nnp_chan_find_map(chan, req.i_map_id);
+ if (!hostres_map) {
+ req.o_errno = NNPER_NO_SUCH_HOSTRES_MAP;
+ goto put_chan;
+ }
+
+ cmd[0] = FIELD_PREP(NNP_H2C_OP_MASK, NNP_IPC_H2C_OP_CHANNEL_HOSTRES_OP);
+ cmd[0] |= FIELD_PREP(NNP_H2C_CHANNEL_HOSTRES_QW0_CHAN_ID_MASK,
+ chan->chan_id);
+ cmd[0] |= FIELD_PREP(NNP_H2C_CHANNEL_HOSTRES_QW0_ID_MASK, req.i_map_id);
+ cmd[0] |= FIELD_PREP(NNP_H2C_CHANNEL_HOSTRES_QW0_UNMAP_MASK, 1);
+ cmd[1] = 0;
+
+ ret = nnp_msched_queue_msg(chan->cmdq, cmd);
+
+put_chan:
+ nnp_chan_put(chan);
+done:
+ if (!ret && copy_to_user(arg, &req, io_size))
+ return -EFAULT;
+
+ return ret;
+}
+
static long nnp_device_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
{
struct device_client *client = f->private_data;
@@ -261,6 +688,20 @@ static long nnp_device_ioctl(struct file *f, unsigned int cmd, unsigned long arg
case _IOC_NR(IOCTL_NNPI_DEVICE_CREATE_CHANNEL):
ret = create_channel(client, (void __user *)arg, size);
break;
+ case _IOC_NR(IOCTL_NNPI_DEVICE_CREATE_CHANNEL_RB):
+ ret = create_channel_data_ringbuf(client, (void __user *)arg,
+ size);
+ break;
+ case _IOC_NR(IOCTL_NNPI_DEVICE_DESTROY_CHANNEL_RB):
+ ret = destroy_channel_data_ringbuf(client, (void __user *)arg,
+ size);
+ break;
+ case _IOC_NR(IOCTL_NNPI_DEVICE_CHANNEL_MAP_HOSTRES):
+ ret = map_hostres(client, (void __user *)arg, size);
+ break;
+ case _IOC_NR(IOCTL_NNPI_DEVICE_CHANNEL_UNMAP_HOSTRES):
+ ret = unmap_hostres(client, (void __user *)arg, size);
+ break;
default:
ret = -EINVAL;
break;
diff --git a/include/uapi/misc/intel_nnpi.h b/include/uapi/misc/intel_nnpi.h
index 620a5d4..8d53af6 100644
--- a/include/uapi/misc/intel_nnpi.h
+++ b/include/uapi/misc/intel_nnpi.h
@@ -150,6 +150,43 @@ struct nnpdrv_ioctl_destroy_hostres {
_IOWR('D', 0, struct ioctl_nnpi_create_channel)
/**
+ * IOCTL_NNPI_DEVICE_CREATE_CHANNEL_RB:
+ *
+ * A request to create a data ring buffer for a command channel object.
+ * This is used to transfer data together with command to the device.
+ * A device command may include a data size fields which indicate how much data
+ * has pushed into that ring-buffer object.
+ */
+#define IOCTL_NNPI_DEVICE_CREATE_CHANNEL_RB \
+ _IOWR('D', 1, struct ioctl_nnpi_create_channel_data_ringbuf)
+
+/**
+ * IOCTL_NNPI_DEVICE_DESTROY_CHANNEL_RB:
+ *
+ * A request to destoy a data ring buffer allocated for a command channel.
+ */
+#define IOCTL_NNPI_DEVICE_DESTROY_CHANNEL_RB \
+ _IOWR('D', 2, struct ioctl_nnpi_destroy_channel_data_ringbuf)
+
+/**
+ * IOCTL_NNPI_DEVICE_CHANNEL_MAP_HOSTRES:
+ *
+ * A request to map a host resource to a command channel object.
+ * Device commands can include "map id" of this mapping for referencing
+ * a host resource.
+ */
+#define IOCTL_NNPI_DEVICE_CHANNEL_MAP_HOSTRES \
+ _IOWR('D', 3, struct ioctl_nnpi_channel_map_hostres)
+
+/**
+ * IOCTL_NNPI_DEVICE_CHANNEL_UNMAP_HOSTRES:
+ *
+ * A request to unmap a host resource previously mapped to a command channel.
+ */
+#define IOCTL_NNPI_DEVICE_CHANNEL_UNMAP_HOSTRES \
+ _IOWR('D', 4, struct ioctl_nnpi_channel_unmap_hostres)
+
+/**
* struct ioctl_nnpi_create_channel - IOCTL_NNPI_DEVICE_CREATE_CHANNEL payload
* @i_host_fd: opened file descriptor to /dev/nnpi_host
* @i_min_id: minimum range for channel id allocation
@@ -177,6 +214,80 @@ struct ioctl_nnpi_create_channel {
__u16 o_channel_id;
};
+/**
+ * struct ioctl_nnpi_create_channel_data_ringbuf
+ * @i_hostres_handle: handle of a host resource which will be used to hold
+ * the ring-buffer content.
+ * @i_channel_id: command channel id.
+ * @i_id: id of the ring buffer object (can be 0 or 1).
+ * @i_h2c: non-zero if this ring-buffer is for command submission use,
+ * otherwise it is for responses.
+ * @o_errno: On input, must be set to 0.
+ * On output, 0 on success, one of the NNPERR_* error codes on error.
+ *
+ * this is the payload for IOCTL_NNPI_DEVICE_CREATE_CHANNEL_RB ioctl
+ */
+struct ioctl_nnpi_create_channel_data_ringbuf {
+ __s32 i_hostres_handle;
+ __u32 i_channel_id;
+ __u32 i_id;
+ __u32 i_h2c;
+ __u32 o_errno;
+};
+
+/**
+ * struct ioctl_nnpi_destroy_channel_data_ringbuf
+ * @i_channel_id: command channel id.
+ * @i_id: id of the ring buffer object (can be 0 or 1).
+ * @i_h2c: true if this ring-buffer is for command submission use,
+ * otherwise it is for responses.
+ * @o_errno: On input, must be set to 0.
+ * On output, 0 on success, one of the NNPERR_* error codes on error.
+ *
+ * this is the payload for IOCTL_NNPI_DEVICE_DESTROY_CHANNEL_RB ioctl
+ */
+struct ioctl_nnpi_destroy_channel_data_ringbuf {
+ __u32 i_channel_id;
+ __u32 i_id;
+ __u32 i_h2c;
+ __u32 o_errno;
+};
+
+/**
+ * struct ioctl_nnpi_channel_map_hostres
+ * @i_hostres_handle: handle of a host resource to be mapped
+ * @i_channel_id: command channel id.
+ * @o_map_id: returns unique id of the mapping
+ * @o_sync_needed: returns non-zero if LOCK/UNLOCK_HOST_RESOURCE ioctls
+ * needs to be used before/after accessing the resource from cpu.
+ * @o_errno: On input, must be set to 0.
+ * On output, 0 on success, one of the NNPERR_* error codes on error.
+ *
+ * this is the payload for IOCTL_NNPI_DEVICE_CHANNEL_MAP_HOSTRES ioctl
+ */
+struct ioctl_nnpi_channel_map_hostres {
+ __s32 i_hostres_handle;
+ __u32 i_channel_id;
+ __u32 o_map_id;
+ __u32 o_sync_needed;
+ __u32 o_errno;
+};
+
+/**
+ * ioctl_nnpi_channel_unmap_hostres
+ * @i_channel_id: command channel id.
+ * @i_map_id: mapping id
+ * @o_errno: On input, must be set to 0.
+ * On output, 0 on success, one of the NNPERR_* error codes on error.
+ *
+ * This is the payload for IOCTL_NNPI_DEVICE_CHANNEL_UNMAP_HOSTRES ioctl
+ */
+struct ioctl_nnpi_channel_unmap_hostres {
+ __u32 i_channel_id;
+ __u32 i_map_id;
+ __u32 o_errno;
+};
+
/****************************************************************
* Error code values - errors returned in o_errno fields of
* above structures.
--
1.8.3.1
---------------------------------------------------------------------
Intel Israel (74) Limited
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
On Wed, May 12, 2021 at 10:10:31AM +0300, Guy Zadicario wrote:
> Hi,
>
> The following series is a driver for a new PCIe device from Intel named NNP-I
> (Nirvana Neural Processor for Inference). NNP-I is a PCIe connected compute
> device used for acceleration of AI deep learning inference applications in the
> data-center.
>
> The reason that this driver should be in the kernel is that it aims to serve
> multiple users and user-space applications which might share the same NNP-I
> card. Workloads from multiple applications can be processed simultanously by
> the NNP-I card if enough compute resources exist.
>
> Overview of the NNP-I device, driver structure and ABIs used in the driver is in
> patch#1, which adds the info as a document as it might be a useful info for
> anyone trying to understand the driver even past review.
>
> In order to ease the review process, there will be multiple series for the
> entire driver code. This is the first series, and it implements everything
> necessary to initialize the NNP-I device and allow a user-space inference
> application to use it. Other features, which are mostly related to maintenance,
> device status visibility and error-handling, will be submitted on the next stage.
>
> A basic user-space library and test application which illustrates the flow of
> an NNP-I inference application can be found here: https://github.com/IntelAI/nnpi-host
> (This series is enough for the test application to run)
>
> This patchset has gone through internal review inside Intel, the summary of the
> change log from the internal review follows.
>
> I would appreciate any feedback, questions or comments to this series.
>
> Changes in v22:
Why is "v22" not in the [PATCH...] part of the subjects here?
And has there really been 21 other series posted to lkml for this?
thanks,
greg k-h
On Wed, May 12, 2021 at 09:27:13AM +0200, Greg KH wrote:
> On Wed, May 12, 2021 at 10:10:31AM +0300, Guy Zadicario wrote:
> > Hi,
> >
> > The following series is a driver for a new PCIe device from Intel named NNP-I
> > (Nirvana Neural Processor for Inference). NNP-I is a PCIe connected compute
> > device used for acceleration of AI deep learning inference applications in the
> > data-center.
> >
> > The reason that this driver should be in the kernel is that it aims to serve
> > multiple users and user-space applications which might share the same NNP-I
> > card. Workloads from multiple applications can be processed simultanously by
> > the NNP-I card if enough compute resources exist.
> >
> > Overview of the NNP-I device, driver structure and ABIs used in the driver is in
> > patch#1, which adds the info as a document as it might be a useful info for
> > anyone trying to understand the driver even past review.
> >
> > In order to ease the review process, there will be multiple series for the
> > entire driver code. This is the first series, and it implements everything
> > necessary to initialize the NNP-I device and allow a user-space inference
> > application to use it. Other features, which are mostly related to maintenance,
> > device status visibility and error-handling, will be submitted on the next stage.
> >
> > A basic user-space library and test application which illustrates the flow of
> > an NNP-I inference application can be found here: https://github.com/IntelAI/nnpi-host
> > (This series is enough for the test application to run)
> >
> > This patchset has gone through internal review inside Intel, the summary of the
> > change log from the internal review follows.
> >
> > I would appreciate any feedback, questions or comments to this series.
> >
> > Changes in v22:
>
> Why is "v22" not in the [PATCH...] part of the subjects here?
>
> And has there really been 21 other series posted to lkml for this?
>
This is the first post to lkml. The first 21 versions was sent for
internal review in Intel only.
I could possibly remove the long change log from the cover letter.
Thanks,
Guy.
---------------------------------------------------------------------
Intel Israel (74) Limited
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
On Wed, May 12, 2021 at 10:51:41AM +0300, Guy Zadicario wrote:
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
>
Now deleted, this is not compatible with kernel development :(