This commit introduces a new misc device driver for the Intel(r) Dynamic
Load Balancer 2.0 (Intel(r) DLB 2.0). The Intel DLB 2.0 is a PCIe device that
provides load-balanced, prioritized scheduling of core-to-core communication.
The Intel DLB 2.0 consists of queues and arbiters that connect producer cores
and consumer cores. The device implements load-balanced queueing features
including:
- Lock-free multi-producer/multi-consumer operation.
- Multiple priority levels for varying traffic types.
- 'Direct' traffic (i.e. multi-producer/single-consumer)
- Simple unordered load-balanced distribution.
- Atomic lock free load balancing across multiple consumers.
- Queue element reordering feature allowing ordered load-balanced
distribution.
Intel DLB 2.0 can be used in an event-driven programming model, such as DPDK's
Event Device Library[2]. Such frameworks are commonly used in packet processing
pipelines that benefit from the framework's multi-core scalability, dynamic
load-balancing, and variety of packet distribution and synchronization schemes.
These distribution schemes include "parallel" (packets are load-balanced
across multiple cores and processed in parallel), "ordered" (similar to
"parallel" but packets are reordered into ingress order by the device), and
"atomic" (packet flows are scheduled to a single core at a time such that
locks are not required to access per-flow data, and dynamically migrated to
ensure load-balance).
The fundamental unit of communication through the device is a queue entry
(QE), which consists of 8B of data and 8B of metadata (destination queue,
priority, etc.). The data field can be any type that fits within 8B.
A core's interface to the device, a "port," consists of a memory-mappable
region through which the core enqueues a queue entry, and an in-memory
queue (the "consumer queue") to which the device schedules QEs. Each QE
is enqueued to a device-managed queue, and from there scheduled to a port.
Software specifies the "linking" of queues and ports; i.e. which ports the
device is allowed to schedule to for a given queue. The device uses a
credit scheme to prevent overflow of the on-device queue storage.
Applications can interface directly with the device by mapping the port's
memory and MMIO regions into the application's address space for enqueue
and dequeue operations, but call into the kernel driver for configuration
operations. An application can also be polling- or interrupt-driven;
Intel DLB 2.0 supports both modes of operation.
Device resources -- i.e. ports, queues, and credits -- are contained within
a scheduling domain. Scheduling domains are isolated from one another; a
port can only enqueue to and dequeue from queues within its scheduling
domain. A scheduling domain's resources are configured through a scheduling
domain file, which is acquired through an ioctl.
Intel DLB 2.0 supports SR-IOV and Scalable IOV, and allows for a flexible
division of its resources among the PF and its virtual devices. The virtual
devices are incapable of configuring the device directly; they use a hardware
mailbox to proxy configuration requests to the PF driver. This driver supports
both PF and virtual devices, as there is significant code re-use between the
two, with device-specific behavior handled through a callback interface.
Virtualization support will be added in a later patch set.
The dlb driver uses ioctls as its primary interface (it makes use of sysfs
as well, to a lesser extent). The dlb device file supports a different
ioctl interface than the scheduling domain file; the dlb device file
is used for device-wide operations (including scheduling domain creation),
and the scheduling domain file supports operations on the scheduling
domain's resources (primarily resource configuration).
[1] https://builders.intel.com/docs/networkbuilders/SKU-343247-001US-queue-management-and-load-balancing-on-intel-architecture.pdf
[2] https://doc.dpdk.org/guides/prog_guide/eventdev.html
v2:
- Change driver license to GPLv2 only
- Expand Kconfig help text and remove unnecessary (R)s
- Remove unnecessary prints
- Add a new entry in ioctl-number.rst
- Convert the ioctl handler into a switch statement
- Correct some instances of IOWR that should have been IOR
- Align macro blocks
- Don't break ioctl ABI when introducing new commands
- Remove indirect pointers from ioctl data structures
- Remove the get-sched-domain-fd ioctl command
Gage Eads (19):
dlb2: add skeleton for DLB 2.0 driver
dlb2: initialize PF device
dlb2: add resource and device initialization
dlb2: add device ioctl layer and first three ioctls
dlb2: add sched domain config and reset support
dlb2: add runtime power-management support
dlb2: add queue create and queue-depth-get ioctls
dlb2: add ioctl to configure ports, query poll mode
dlb2: add port mmap support
dlb2: add start domain ioctl
dlb2: add queue map and unmap ioctls
dlb2: add port enable/disable ioctls
dlb2: add CQ interrupt support
dlb2: add domain alert support
dlb2: add sequence-number management ioctls
dlb2: add cos bandwidth get/set ioctls
dlb2: add device FLR support
dlb2: add basic PF sysfs interfaces
dlb2: add ingress error handling
Documentation/ABI/testing/sysfs-driver-dlb2 | 202 +
Documentation/misc-devices/dlb2.rst | 310 +
Documentation/misc-devices/index.rst | 1 +
Documentation/userspace-api/ioctl/ioctl-number.rst | 1 +
MAINTAINERS | 7 +
drivers/misc/Kconfig | 1 +
drivers/misc/Makefile | 1 +
drivers/misc/dlb2/Kconfig | 17 +
drivers/misc/dlb2/Makefile | 13 +
drivers/misc/dlb2/dlb2_bitmap.h | 286 +
drivers/misc/dlb2/dlb2_file.c | 133 +
drivers/misc/dlb2/dlb2_file.h | 19 +
drivers/misc/dlb2/dlb2_hw_types.h | 353 +
drivers/misc/dlb2/dlb2_intr.c | 137 +
drivers/misc/dlb2/dlb2_intr.h | 30 +
drivers/misc/dlb2/dlb2_ioctl.c | 892 +++
drivers/misc/dlb2/dlb2_ioctl.h | 14 +
drivers/misc/dlb2/dlb2_main.c | 1091 +++
drivers/misc/dlb2/dlb2_main.h | 285 +
drivers/misc/dlb2/dlb2_pf_ops.c | 1286 ++++
drivers/misc/dlb2/dlb2_regs.h | 3702 ++++++++++
drivers/misc/dlb2/dlb2_resource.c | 7117 ++++++++++++++++++++
drivers/misc/dlb2/dlb2_resource.h | 924 +++
include/linux/pci_ids.h | 2 +
include/uapi/linux/dlb2_user.h | 1045 +++
25 files changed, 17869 insertions(+)
create mode 100644 Documentation/ABI/testing/sysfs-driver-dlb2
create mode 100644 Documentation/misc-devices/dlb2.rst
create mode 100644 drivers/misc/dlb2/Kconfig
create mode 100644 drivers/misc/dlb2/Makefile
create mode 100644 drivers/misc/dlb2/dlb2_bitmap.h
create mode 100644 drivers/misc/dlb2/dlb2_file.c
create mode 100644 drivers/misc/dlb2/dlb2_file.h
create mode 100644 drivers/misc/dlb2/dlb2_hw_types.h
create mode 100644 drivers/misc/dlb2/dlb2_intr.c
create mode 100644 drivers/misc/dlb2/dlb2_intr.h
create mode 100644 drivers/misc/dlb2/dlb2_ioctl.c
create mode 100644 drivers/misc/dlb2/dlb2_ioctl.h
create mode 100644 drivers/misc/dlb2/dlb2_main.c
create mode 100644 drivers/misc/dlb2/dlb2_main.h
create mode 100644 drivers/misc/dlb2/dlb2_pf_ops.c
create mode 100644 drivers/misc/dlb2/dlb2_regs.h
create mode 100644 drivers/misc/dlb2/dlb2_resource.c
create mode 100644 drivers/misc/dlb2/dlb2_resource.h
create mode 100644 include/uapi/linux/dlb2_user.h
base-commit: 00e4db51259a5f936fec1424b884f029479d3981
--
2.13.6
The port is a core's interface to the DLB, and it consists of an MMIO page
(the "producer port" (PP)) through which the core enqueues a queue entry
and an in-memory queue (the "consumer queue" (CQ)) to which the device
schedules QEs. The driver allocates DMA memory for each port's CQ, and
frees this memory during domain reset or driver removal. A subsequent
commit will add the mmap interface for an application to directly access
the PP and CQ regions.
The device supports two formats ("standard" and "sparse") for CQ entries,
dubbed the "poll mode". This (device-wide) mode is selected by the driver;
to determine the mode at run time, the driver provides an ioctl for
user-space software to query which mode the driver has configured. In this
way, the policy of which mode to use is decoupled from user-space software.
Signed-off-by: Gage Eads <[email protected]>
Reviewed-by: Magnus Karlsson <[email protected]>
---
drivers/misc/dlb2/dlb2_hw_types.h | 1 +
drivers/misc/dlb2/dlb2_ioctl.c | 150 +++++++
drivers/misc/dlb2/dlb2_main.c | 70 +++
drivers/misc/dlb2/dlb2_main.h | 23 +
drivers/misc/dlb2/dlb2_pf_ops.c | 46 ++
drivers/misc/dlb2/dlb2_resource.c | 922 ++++++++++++++++++++++++++++++++++++++
drivers/misc/dlb2/dlb2_resource.h | 89 ++++
include/uapi/linux/dlb2_user.h | 99 ++++
8 files changed, 1400 insertions(+)
diff --git a/drivers/misc/dlb2/dlb2_hw_types.h b/drivers/misc/dlb2/dlb2_hw_types.h
index e43860a215ff..7db59157da15 100644
--- a/drivers/misc/dlb2/dlb2_hw_types.h
+++ b/drivers/misc/dlb2/dlb2_hw_types.h
@@ -317,6 +317,7 @@ struct dlb2_hw {
/* Virtualization */
int virt_mode;
+ unsigned int pasid[DLB2_MAX_NUM_VDEVS];
};
#endif /* __DLB2_HW_TYPES_H */
diff --git a/drivers/misc/dlb2/dlb2_ioctl.c b/drivers/misc/dlb2/dlb2_ioctl.c
index f787fd19689f..7a3fea60c302 100644
--- a/drivers/misc/dlb2/dlb2_ioctl.c
+++ b/drivers/misc/dlb2/dlb2_ioctl.c
@@ -51,6 +51,126 @@ DLB2_DOMAIN_IOCTL_CALLBACK_TEMPLATE(create_dir_queue)
DLB2_DOMAIN_IOCTL_CALLBACK_TEMPLATE(get_ldb_queue_depth)
DLB2_DOMAIN_IOCTL_CALLBACK_TEMPLATE(get_dir_queue_depth)
+/*
+ * Port creation ioctls don't use the callback template macro because they have
+ * a number of OS-dependent memory operations.
+ */
+static int dlb2_domain_ioctl_create_ldb_port(struct dlb2_dev *dev,
+ struct dlb2_domain *domain,
+ unsigned long user_arg)
+{
+ struct dlb2_cmd_response response = {0};
+ struct dlb2_create_ldb_port_args arg;
+ dma_addr_t cq_dma_base = 0;
+ void *cq_base = NULL;
+ int ret;
+
+ if (copy_from_user(&arg, (void __user *)user_arg, sizeof(arg)))
+ return -EFAULT;
+
+ mutex_lock(&dev->resource_mutex);
+
+ cq_base = dma_alloc_coherent(&dev->pdev->dev,
+ DLB2_CQ_SIZE,
+ &cq_dma_base,
+ GFP_KERNEL);
+ if (!cq_base) {
+ response.status = DLB2_ST_NO_MEMORY;
+ ret = -ENOMEM;
+ goto unlock;
+ }
+
+ ret = dev->ops->create_ldb_port(&dev->hw,
+ domain->id,
+ &arg,
+ (uintptr_t)cq_dma_base,
+ &response);
+ if (ret)
+ goto unlock;
+
+ /* Fill out the per-port data structure */
+ dev->ldb_port[response.id].id = response.id;
+ dev->ldb_port[response.id].is_ldb = true;
+ dev->ldb_port[response.id].domain = domain;
+ dev->ldb_port[response.id].cq_base = cq_base;
+ dev->ldb_port[response.id].cq_dma_base = cq_dma_base;
+ dev->ldb_port[response.id].valid = true;
+
+unlock:
+ if (ret && cq_dma_base)
+ dma_free_coherent(&dev->pdev->dev,
+ DLB2_CQ_SIZE,
+ cq_base,
+ cq_dma_base);
+
+ mutex_unlock(&dev->resource_mutex);
+
+ BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
+
+ if (copy_to_user((void __user *)user_arg, &response, sizeof(response)))
+ return -EFAULT;
+
+ return ret;
+}
+
+static int dlb2_domain_ioctl_create_dir_port(struct dlb2_dev *dev,
+ struct dlb2_domain *domain,
+ unsigned long user_arg)
+{
+ struct dlb2_cmd_response response = {0};
+ struct dlb2_create_dir_port_args arg;
+ dma_addr_t cq_dma_base = 0;
+ void *cq_base = NULL;
+ int ret;
+
+ if (copy_from_user(&arg, (void __user *)user_arg, sizeof(arg)))
+ return -EFAULT;
+
+ mutex_lock(&dev->resource_mutex);
+
+ cq_base = dma_alloc_coherent(&dev->pdev->dev,
+ DLB2_CQ_SIZE,
+ &cq_dma_base,
+ GFP_KERNEL);
+ if (!cq_base) {
+ response.status = DLB2_ST_NO_MEMORY;
+ ret = -ENOMEM;
+ goto unlock;
+ }
+
+ ret = dev->ops->create_dir_port(&dev->hw,
+ domain->id,
+ &arg,
+ (uintptr_t)cq_dma_base,
+ &response);
+ if (ret)
+ goto unlock;
+
+ /* Fill out the per-port data structure */
+ dev->dir_port[response.id].id = response.id;
+ dev->dir_port[response.id].is_ldb = false;
+ dev->dir_port[response.id].domain = domain;
+ dev->dir_port[response.id].cq_base = cq_base;
+ dev->dir_port[response.id].cq_dma_base = cq_dma_base;
+ dev->dir_port[response.id].valid = true;
+
+unlock:
+ if (ret && cq_dma_base)
+ dma_free_coherent(&dev->pdev->dev,
+ DLB2_CQ_SIZE,
+ cq_base,
+ cq_dma_base);
+
+ mutex_unlock(&dev->resource_mutex);
+
+ BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
+
+ if (copy_to_user((void __user *)user_arg, &response, sizeof(response)))
+ return -EFAULT;
+
+ return ret;
+}
+
long dlb2_domain_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
{
struct dlb2_domain *dom = f->private_data;
@@ -65,6 +185,10 @@ long dlb2_domain_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
return dlb2_domain_ioctl_get_ldb_queue_depth(dev, dom, arg);
case DLB2_IOC_GET_DIR_QUEUE_DEPTH:
return dlb2_domain_ioctl_get_dir_queue_depth(dev, dom, arg);
+ case DLB2_IOC_CREATE_LDB_PORT:
+ return dlb2_domain_ioctl_create_ldb_port(dev, dom, arg);
+ case DLB2_IOC_CREATE_DIR_PORT:
+ return dlb2_domain_ioctl_create_dir_port(dev, dom, arg);
default:
return -ENOTTY;
}
@@ -168,6 +292,30 @@ static int dlb2_ioctl_get_num_resources(struct dlb2_dev *dev,
return ret;
}
+static int dlb2_ioctl_query_cq_poll_mode(struct dlb2_dev *dev,
+ unsigned long user_arg)
+{
+ struct dlb2_query_cq_poll_mode_args arg;
+ struct dlb2_cmd_response response = {0};
+ int ret;
+
+ if (copy_from_user(&arg, (void __user *)user_arg, sizeof(arg)))
+ return -EFAULT;
+
+ mutex_lock(&dev->resource_mutex);
+
+ ret = dev->ops->query_cq_poll_mode(dev, &response);
+
+ mutex_unlock(&dev->resource_mutex);
+
+ BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
+
+ if (copy_to_user((void __user *)user_arg, &response, sizeof(response)))
+ return -EFAULT;
+
+ return ret;
+}
+
long dlb2_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
{
struct dlb2_dev *dev;
@@ -181,6 +329,8 @@ long dlb2_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
return dlb2_ioctl_create_sched_domain(dev, arg);
case DLB2_IOC_GET_NUM_RESOURCES:
return dlb2_ioctl_get_num_resources(dev, arg);
+ case DLB2_IOC_QUERY_CQ_POLL_MODE:
+ return dlb2_ioctl_query_cq_poll_mode(dev, arg);
default:
return -ENOTTY;
}
diff --git a/drivers/misc/dlb2/dlb2_main.c b/drivers/misc/dlb2/dlb2_main.c
index b331e096d206..a7512832fc1c 100644
--- a/drivers/misc/dlb2/dlb2_main.c
+++ b/drivers/misc/dlb2/dlb2_main.c
@@ -107,11 +107,79 @@ int dlb2_init_domain(struct dlb2_dev *dlb2_dev, u32 domain_id)
return 0;
}
+static void dlb2_release_domain_memory(struct dlb2_dev *dev, u32 domain_id)
+{
+ struct dlb2_port *port;
+ int i;
+
+ for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+ port = &dev->ldb_port[i];
+
+ if (port->valid && port->domain->id == domain_id) {
+ dma_free_coherent(&dev->pdev->dev,
+ DLB2_CQ_SIZE,
+ port->cq_base,
+ port->cq_dma_base);
+
+ port->valid = false;
+ }
+ }
+
+ for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++) {
+ port = &dev->dir_port[i];
+
+ if (port->valid && port->domain->id == domain_id) {
+ dma_free_coherent(&dev->pdev->dev,
+ DLB2_CQ_SIZE,
+ port->cq_base,
+ port->cq_dma_base);
+
+ port->valid = false;
+ }
+ }
+}
+
+static void dlb2_release_device_memory(struct dlb2_dev *dev)
+{
+ struct dlb2_port *port;
+ int i;
+
+ for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+ port = &dev->ldb_port[i];
+
+ if (port->valid) {
+ dma_free_coherent(&dev->pdev->dev,
+ DLB2_CQ_SIZE,
+ port->cq_base,
+ port->cq_dma_base);
+
+ port->valid = false;
+ }
+ }
+
+ for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++) {
+ port = &dev->dir_port[i];
+
+ if (port->valid) {
+ dma_free_coherent(&dev->pdev->dev,
+ DLB2_CQ_SIZE,
+ port->cq_base,
+ port->cq_dma_base);
+
+ port->valid = false;
+ }
+ }
+}
+
static int __dlb2_free_domain(struct dlb2_dev *dev, struct dlb2_domain *domain)
{
int ret = 0;
ret = dev->ops->reset_domain(&dev->hw, domain->id);
+
+ /* Unpin and free all memory pages associated with the domain */
+ dlb2_release_domain_memory(dev, domain->id);
+
if (ret) {
dev->domain_reset_failed = true;
dev_err(dev->dlb2_device,
@@ -329,6 +397,8 @@ static void dlb2_remove(struct pci_dev *pdev)
dlb2_resource_free(&dlb2_dev->hw);
+ dlb2_release_device_memory(dlb2_dev);
+
dlb2_dev->ops->device_destroy(dlb2_dev, dlb2_class);
dlb2_dev->ops->cdev_del(dlb2_dev);
diff --git a/drivers/misc/dlb2/dlb2_main.h b/drivers/misc/dlb2/dlb2_main.h
index 41a3254a78ac..ceb5512a104c 100644
--- a/drivers/misc/dlb2/dlb2_main.h
+++ b/drivers/misc/dlb2/dlb2_main.h
@@ -68,6 +68,16 @@ struct dlb2_device_ops {
u32 domain_id,
struct dlb2_create_dir_queue_args *args,
struct dlb2_cmd_response *resp);
+ int (*create_ldb_port)(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_create_ldb_port_args *args,
+ uintptr_t cq_dma_base,
+ struct dlb2_cmd_response *resp);
+ int (*create_dir_port)(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_create_dir_port_args *args,
+ uintptr_t cq_dma_base,
+ struct dlb2_cmd_response *resp);
int (*get_num_resources)(struct dlb2_hw *hw,
struct dlb2_get_num_resources_args *args);
int (*reset_domain)(struct dlb2_hw *hw, u32 domain_id);
@@ -80,11 +90,22 @@ struct dlb2_device_ops {
struct dlb2_get_dir_queue_depth_args *args,
struct dlb2_cmd_response *resp);
void (*init_hardware)(struct dlb2_dev *dev);
+ int (*query_cq_poll_mode)(struct dlb2_dev *dev,
+ struct dlb2_cmd_response *user_resp);
};
extern struct dlb2_device_ops dlb2_pf_ops;
extern const struct file_operations dlb2_domain_fops;
+struct dlb2_port {
+ void *cq_base;
+ dma_addr_t cq_dma_base;
+ struct dlb2_domain *domain;
+ int id;
+ u8 is_ldb;
+ u8 valid;
+};
+
struct dlb2_domain {
struct dlb2_dev *dlb2_dev;
struct kref refcnt;
@@ -99,6 +120,8 @@ struct dlb2_dev {
struct list_head list;
struct device *dlb2_device;
struct dlb2_domain *sched_domains[DLB2_MAX_NUM_DOMAINS];
+ struct dlb2_port ldb_port[DLB2_MAX_NUM_LDB_PORTS];
+ struct dlb2_port dir_port[DLB2_MAX_NUM_DIR_PORTS];
/*
* The resource mutex serializes access to driver data structures and
* hardware registers.
diff --git a/drivers/misc/dlb2/dlb2_pf_ops.c b/drivers/misc/dlb2/dlb2_pf_ops.c
index 89c0e66b3c6b..8556dcdff8c4 100644
--- a/drivers/misc/dlb2/dlb2_pf_ops.c
+++ b/drivers/misc/dlb2/dlb2_pf_ops.c
@@ -198,9 +198,16 @@ dlb2_pf_wait_for_device_ready(struct dlb2_dev *dlb2_dev,
return 0;
}
+static bool dlb2_sparse_cq_enabled = true;
+
static void
dlb2_pf_init_hardware(struct dlb2_dev *dlb2_dev)
{
+ if (dlb2_sparse_cq_enabled) {
+ dlb2_hw_enable_sparse_ldb_cq_mode(&dlb2_dev->hw);
+
+ dlb2_hw_enable_sparse_dir_cq_mode(&dlb2_dev->hw);
+ }
}
/*****************************/
@@ -234,6 +241,28 @@ dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
}
static int
+dlb2_pf_create_ldb_port(struct dlb2_hw *hw,
+ u32 id,
+ struct dlb2_create_ldb_port_args *args,
+ uintptr_t cq_dma_base,
+ struct dlb2_cmd_response *resp)
+{
+ return dlb2_hw_create_ldb_port(hw, id, args, cq_dma_base,
+ resp, false, 0);
+}
+
+static int
+dlb2_pf_create_dir_port(struct dlb2_hw *hw,
+ u32 id,
+ struct dlb2_create_dir_port_args *args,
+ uintptr_t cq_dma_base,
+ struct dlb2_cmd_response *resp)
+{
+ return dlb2_hw_create_dir_port(hw, id, args, cq_dma_base,
+ resp, false, 0);
+}
+
+static int
dlb2_pf_get_num_resources(struct dlb2_hw *hw,
struct dlb2_get_num_resources_args *args)
{
@@ -264,6 +293,20 @@ dlb2_pf_get_dir_queue_depth(struct dlb2_hw *hw,
return dlb2_hw_get_dir_queue_depth(hw, id, args, resp, false, 0);
}
+static int
+dlb2_pf_query_cq_poll_mode(struct dlb2_dev *dlb2_dev,
+ struct dlb2_cmd_response *user_resp)
+{
+ user_resp->status = 0;
+
+ if (dlb2_sparse_cq_enabled)
+ user_resp->id = DLB2_CQ_POLL_MODE_SPARSE;
+ else
+ user_resp->id = DLB2_CQ_POLL_MODE_STD;
+
+ return 0;
+}
+
/********************************/
/****** DLB2 PF Device Ops ******/
/********************************/
@@ -284,9 +327,12 @@ struct dlb2_device_ops dlb2_pf_ops = {
.create_sched_domain = dlb2_pf_create_sched_domain,
.create_ldb_queue = dlb2_pf_create_ldb_queue,
.create_dir_queue = dlb2_pf_create_dir_queue,
+ .create_ldb_port = dlb2_pf_create_ldb_port,
+ .create_dir_port = dlb2_pf_create_dir_port,
.get_num_resources = dlb2_pf_get_num_resources,
.reset_domain = dlb2_pf_reset_domain,
.get_ldb_queue_depth = dlb2_pf_get_ldb_queue_depth,
.get_dir_queue_depth = dlb2_pf_get_dir_queue_depth,
.init_hardware = dlb2_pf_init_hardware,
+ .query_cq_poll_mode = dlb2_pf_query_cq_poll_mode,
};
diff --git a/drivers/misc/dlb2/dlb2_resource.c b/drivers/misc/dlb2/dlb2_resource.c
index 2531c6508994..03824fd48f60 100644
--- a/drivers/misc/dlb2/dlb2_resource.c
+++ b/drivers/misc/dlb2/dlb2_resource.c
@@ -1089,6 +1089,171 @@ static void dlb2_configure_dir_queue(struct dlb2_hw *hw,
queue->queue_configured = true;
}
+static int
+dlb2_verify_create_ldb_port_args(struct dlb2_hw *hw,
+ u32 domain_id,
+ uintptr_t cq_dma_base,
+ struct dlb2_create_ldb_port_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_hw_domain *domain;
+ int i;
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+ if (!domain) {
+ resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+ return -EINVAL;
+ }
+
+ if (!domain->configured) {
+ resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+ return -EINVAL;
+ }
+
+ if (domain->started) {
+ resp->status = DLB2_ST_DOMAIN_STARTED;
+ return -EINVAL;
+ }
+
+ if (args->cos_id >= DLB2_NUM_COS_DOMAINS) {
+ resp->status = DLB2_ST_INVALID_COS_ID;
+ return -EINVAL;
+ }
+
+ if (args->cos_strict) {
+ if (list_empty(&domain->avail_ldb_ports[args->cos_id])) {
+ resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+ return -EINVAL;
+ }
+ } else {
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ if (!list_empty(&domain->avail_ldb_ports[i]))
+ break;
+ }
+
+ if (i == DLB2_NUM_COS_DOMAINS) {
+ resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+ return -EINVAL;
+ }
+ }
+
+ /* Check cache-line alignment */
+ if ((cq_dma_base & 0x3F) != 0) {
+ resp->status = DLB2_ST_INVALID_CQ_VIRT_ADDR;
+ return -EINVAL;
+ }
+
+ if (args->cq_depth != 1 &&
+ args->cq_depth != 2 &&
+ args->cq_depth != 4 &&
+ args->cq_depth != 8 &&
+ args->cq_depth != 16 &&
+ args->cq_depth != 32 &&
+ args->cq_depth != 64 &&
+ args->cq_depth != 128 &&
+ args->cq_depth != 256 &&
+ args->cq_depth != 512 &&
+ args->cq_depth != 1024) {
+ resp->status = DLB2_ST_INVALID_CQ_DEPTH;
+ return -EINVAL;
+ }
+
+ /* The history list size must be >= 1 */
+ if (!args->cq_history_list_size) {
+ resp->status = DLB2_ST_INVALID_HIST_LIST_DEPTH;
+ return -EINVAL;
+ }
+
+ if (args->cq_history_list_size > domain->avail_hist_list_entries) {
+ resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int
+dlb2_verify_create_dir_port_args(struct dlb2_hw *hw,
+ u32 domain_id,
+ uintptr_t cq_dma_base,
+ struct dlb2_create_dir_port_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_hw_domain *domain;
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+ if (!domain) {
+ resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+ return -EINVAL;
+ }
+
+ if (!domain->configured) {
+ resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+ return -EINVAL;
+ }
+
+ if (domain->started) {
+ resp->status = DLB2_ST_DOMAIN_STARTED;
+ return -EINVAL;
+ }
+
+ /*
+ * If the user claims the queue is already configured, validate
+ * the queue ID, its domain, and whether the queue is configured.
+ */
+ if (args->queue_id != -1) {
+ struct dlb2_dir_pq_pair *queue;
+
+ queue = dlb2_get_domain_used_dir_pq(args->queue_id,
+ vdev_req,
+ domain);
+
+ if (!queue || queue->domain_id.phys_id != domain->id.phys_id ||
+ !queue->queue_configured) {
+ resp->status = DLB2_ST_INVALID_DIR_QUEUE_ID;
+ return -EINVAL;
+ }
+ }
+
+ /*
+ * If the port's queue is not configured, validate that a free
+ * port-queue pair is available.
+ */
+ if (args->queue_id == -1 && list_empty(&domain->avail_dir_pq_pairs)) {
+ resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ /* Check cache-line alignment */
+ if ((cq_dma_base & 0x3F) != 0) {
+ resp->status = DLB2_ST_INVALID_CQ_VIRT_ADDR;
+ return -EINVAL;
+ }
+
+ if (args->cq_depth != 1 &&
+ args->cq_depth != 2 &&
+ args->cq_depth != 4 &&
+ args->cq_depth != 8 &&
+ args->cq_depth != 16 &&
+ args->cq_depth != 32 &&
+ args->cq_depth != 64 &&
+ args->cq_depth != 128 &&
+ args->cq_depth != 256 &&
+ args->cq_depth != 512 &&
+ args->cq_depth != 1024) {
+ resp->status = DLB2_ST_INVALID_CQ_DEPTH;
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
static bool dlb2_port_find_slot(struct dlb2_ldb_port *port,
enum dlb2_qid_map_state state,
int *slot)
@@ -1455,6 +1620,495 @@ static void dlb2_dir_port_cq_disable(struct dlb2_hw *hw,
dlb2_flush_csr(hw);
}
+static void dlb2_ldb_port_configure_pp(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain,
+ struct dlb2_ldb_port *port,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ union dlb2_sys_ldb_pp2vas r0 = { {0} };
+ union dlb2_sys_ldb_pp_v r4 = { {0} };
+
+ r0.field.vas = domain->id.phys_id;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_LDB_PP2VAS(port->id.phys_id), r0.val);
+
+ if (vdev_req) {
+ union dlb2_sys_vf_ldb_vpp2pp r1 = { {0} };
+ union dlb2_sys_ldb_pp2vdev r2 = { {0} };
+ union dlb2_sys_vf_ldb_vpp_v r3 = { {0} };
+ unsigned int offs;
+ u32 virt_id;
+
+ /*
+ * DLB uses producer port address bits 17:12 to determine the
+ * producer port ID. In Scalable IOV mode, PP accesses come
+ * through the PF MMIO window for the physical producer port,
+ * so for translation purposes the virtual and physical port
+ * IDs are equal.
+ */
+ if (hw->virt_mode == DLB2_VIRT_SRIOV)
+ virt_id = port->id.virt_id;
+ else
+ virt_id = port->id.phys_id;
+
+ r1.field.pp = port->id.phys_id;
+
+ offs = vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP2PP(offs), r1.val);
+
+ r2.field.vdev = vdev_id;
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_LDB_PP2VDEV(port->id.phys_id),
+ r2.val);
+
+ r3.field.vpp_v = 1;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP_V(offs), r3.val);
+ }
+
+ r4.field.pp_v = 1;
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_LDB_PP_V(port->id.phys_id),
+ r4.val);
+}
+
+static int dlb2_ldb_port_configure_cq(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain,
+ struct dlb2_ldb_port *port,
+ uintptr_t cq_dma_base,
+ struct dlb2_create_ldb_port_args *args,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ union dlb2_sys_ldb_cq_addr_l r0 = { {0} };
+ union dlb2_sys_ldb_cq_addr_u r1 = { {0} };
+ union dlb2_sys_ldb_cq2vf_pf_ro r2 = { {0} };
+ union dlb2_chp_ldb_cq_tkn_depth_sel r3 = { {0} };
+ union dlb2_lsp_cq_ldb_tkn_depth_sel r4 = { {0} };
+ union dlb2_chp_hist_list_lim r5 = { {0} };
+ union dlb2_chp_hist_list_base r6 = { {0} };
+ union dlb2_lsp_cq_ldb_infl_lim r7 = { {0} };
+ union dlb2_chp_hist_list_push_ptr r8 = { {0} };
+ union dlb2_chp_hist_list_pop_ptr r9 = { {0} };
+ union dlb2_sys_ldb_cq_at r10 = { {0} };
+ union dlb2_sys_ldb_cq_pasid r11 = { {0} };
+ union dlb2_chp_ldb_cq2vas r12 = { {0} };
+ union dlb2_lsp_cq2priov r13 = { {0} };
+
+ /* The CQ address is 64B-aligned, and the DLB only wants bits [63:6] */
+ r0.field.addr_l = cq_dma_base >> 6;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_ADDR_L(port->id.phys_id), r0.val);
+
+ r1.field.addr_u = cq_dma_base >> 32;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_ADDR_U(port->id.phys_id), r1.val);
+
+ /*
+ * 'ro' == relaxed ordering. This setting allows DLB2 to write
+ * cache lines out-of-order (but QEs within a cache line are always
+ * updated in-order).
+ */
+ r2.field.vf = vdev_id;
+ r2.field.is_pf = !vdev_req && (hw->virt_mode != DLB2_VIRT_SIOV);
+ r2.field.ro = 1;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ2VF_PF_RO(port->id.phys_id), r2.val);
+
+ if (args->cq_depth <= 8) {
+ r3.field.token_depth_select = 1;
+ } else if (args->cq_depth == 16) {
+ r3.field.token_depth_select = 2;
+ } else if (args->cq_depth == 32) {
+ r3.field.token_depth_select = 3;
+ } else if (args->cq_depth == 64) {
+ r3.field.token_depth_select = 4;
+ } else if (args->cq_depth == 128) {
+ r3.field.token_depth_select = 5;
+ } else if (args->cq_depth == 256) {
+ r3.field.token_depth_select = 6;
+ } else if (args->cq_depth == 512) {
+ r3.field.token_depth_select = 7;
+ } else if (args->cq_depth == 1024) {
+ r3.field.token_depth_select = 8;
+ } else {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: invalid CQ depth\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+ r3.val);
+
+ /*
+ * To support CQs with depth less than 8, program the token count
+ * register with a non-zero initial value. Operations such as domain
+ * reset must take this initial value into account when quiescing the
+ * CQ.
+ */
+ port->init_tkn_cnt = 0;
+
+ if (args->cq_depth < 8) {
+ union dlb2_lsp_cq_ldb_tkn_cnt r14 = { {0} };
+
+ port->init_tkn_cnt = 8 - args->cq_depth;
+
+ r14.field.token_count = port->init_tkn_cnt;
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+ r14.val);
+ } else {
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+ DLB2_LSP_CQ_LDB_TKN_CNT_RST);
+ }
+
+ r4.field.token_depth_select = r3.field.token_depth_select;
+ r4.field.ignore_depth = 0;
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(port->id.phys_id),
+ r4.val);
+
+ /* Reset the CQ write pointer */
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_LDB_CQ_WPTR(port->id.phys_id),
+ DLB2_CHP_LDB_CQ_WPTR_RST);
+
+ r5.field.limit = port->hist_list_entry_limit - 1;
+
+ DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_LIM(port->id.phys_id), r5.val);
+
+ r6.field.base = port->hist_list_entry_base;
+
+ DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_BASE(port->id.phys_id), r6.val);
+
+ /*
+ * The inflight limit sets a cap on the number of QEs for which this CQ
+ * can owe completions at one time.
+ */
+ r7.field.limit = args->cq_history_list_size;
+
+ DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_INFL_LIM(port->id.phys_id), r7.val);
+
+ r8.field.push_ptr = r6.field.base;
+ r8.field.generation = 0;
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_HIST_LIST_PUSH_PTR(port->id.phys_id),
+ r8.val);
+
+ r9.field.pop_ptr = r6.field.base;
+ r9.field.generation = 0;
+
+ DLB2_CSR_WR(hw, DLB2_CHP_HIST_LIST_POP_PTR(port->id.phys_id), r9.val);
+
+ /*
+ * Address translation (AT) settings: 0: untranslated, 2: translated
+ * (see ATS spec regarding Address Type field for more details)
+ */
+ r10.field.cq_at = 0;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_AT(port->id.phys_id), r10.val);
+
+ if (vdev_req && hw->virt_mode == DLB2_VIRT_SIOV) {
+ r11.field.pasid = hw->pasid[vdev_id];
+ r11.field.fmt2 = 1;
+ }
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_LDB_CQ_PASID(port->id.phys_id),
+ r11.val);
+
+ r12.field.cq2vas = domain->id.phys_id;
+
+ DLB2_CSR_WR(hw, DLB2_CHP_LDB_CQ2VAS(port->id.phys_id), r12.val);
+
+ /* Disable the port's QID mappings */
+ r13.field.v = 0;
+
+ DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id), r13.val);
+
+ return 0;
+}
+
+static int dlb2_configure_ldb_port(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain,
+ struct dlb2_ldb_port *port,
+ uintptr_t cq_dma_base,
+ struct dlb2_create_ldb_port_args *args,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ int ret, i;
+
+ port->hist_list_entry_base = domain->hist_list_entry_base +
+ domain->hist_list_entry_offset;
+ port->hist_list_entry_limit = port->hist_list_entry_base +
+ args->cq_history_list_size;
+
+ domain->hist_list_entry_offset += args->cq_history_list_size;
+ domain->avail_hist_list_entries -= args->cq_history_list_size;
+
+ ret = dlb2_ldb_port_configure_cq(hw,
+ domain,
+ port,
+ cq_dma_base,
+ args,
+ vdev_req,
+ vdev_id);
+ if (ret < 0)
+ return ret;
+
+ dlb2_ldb_port_configure_pp(hw,
+ domain,
+ port,
+ vdev_req,
+ vdev_id);
+
+ dlb2_ldb_port_cq_enable(hw, port);
+
+ for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++)
+ port->qid_map[i].state = DLB2_QUEUE_UNMAPPED;
+ port->num_mappings = 0;
+
+ port->enabled = true;
+
+ port->configured = true;
+
+ return 0;
+}
+
+static void dlb2_dir_port_configure_pp(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain,
+ struct dlb2_dir_pq_pair *port,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ union dlb2_sys_dir_pp2vas r0 = { {0} };
+ union dlb2_sys_dir_pp_v r4 = { {0} };
+
+ r0.field.vas = domain->id.phys_id;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_DIR_PP2VAS(port->id.phys_id), r0.val);
+
+ if (vdev_req) {
+ union dlb2_sys_vf_dir_vpp2pp r1 = { {0} };
+ union dlb2_sys_dir_pp2vdev r2 = { {0} };
+ union dlb2_sys_vf_dir_vpp_v r3 = { {0} };
+ unsigned int offs;
+ u32 virt_id;
+
+ /*
+ * DLB uses producer port address bits 17:12 to determine the
+ * producer port ID. In Scalable IOV mode, PP accesses come
+ * through the PF MMIO window for the physical producer port,
+ * so for translation purposes the virtual and physical port
+ * IDs are equal.
+ */
+ if (hw->virt_mode == DLB2_VIRT_SRIOV)
+ virt_id = port->id.virt_id;
+ else
+ virt_id = port->id.phys_id;
+
+ r1.field.pp = port->id.phys_id;
+
+ offs = vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP2PP(offs), r1.val);
+
+ r2.field.vdev = vdev_id;
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_DIR_PP2VDEV(port->id.phys_id),
+ r2.val);
+
+ r3.field.vpp_v = 1;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP_V(offs), r3.val);
+ }
+
+ r4.field.pp_v = 1;
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_DIR_PP_V(port->id.phys_id),
+ r4.val);
+}
+
+static int dlb2_dir_port_configure_cq(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain,
+ struct dlb2_dir_pq_pair *port,
+ uintptr_t cq_dma_base,
+ struct dlb2_create_dir_port_args *args,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ union dlb2_sys_dir_cq_addr_l r0 = { {0} };
+ union dlb2_sys_dir_cq_addr_u r1 = { {0} };
+ union dlb2_sys_dir_cq2vf_pf_ro r2 = { {0} };
+ union dlb2_chp_dir_cq_tkn_depth_sel r3 = { {0} };
+ union dlb2_lsp_cq_dir_tkn_depth_sel_dsi r4 = { {0} };
+ union dlb2_sys_dir_cq_fmt r9 = { {0} };
+ union dlb2_sys_dir_cq_at r10 = { {0} };
+ union dlb2_sys_dir_cq_pasid r11 = { {0} };
+ union dlb2_chp_dir_cq2vas r12 = { {0} };
+
+ /* The CQ address is 64B-aligned, and the DLB only wants bits [63:6] */
+ r0.field.addr_l = cq_dma_base >> 6;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_ADDR_L(port->id.phys_id), r0.val);
+
+ r1.field.addr_u = cq_dma_base >> 32;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_ADDR_U(port->id.phys_id), r1.val);
+
+ /*
+ * 'ro' == relaxed ordering. This setting allows DLB2 to write
+ * cache lines out-of-order (but QEs within a cache line are always
+ * updated in-order).
+ */
+ r2.field.vf = vdev_id;
+ r2.field.is_pf = !vdev_req && (hw->virt_mode != DLB2_VIRT_SIOV);
+ r2.field.ro = 1;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ2VF_PF_RO(port->id.phys_id), r2.val);
+
+ if (args->cq_depth <= 8) {
+ r3.field.token_depth_select = 1;
+ } else if (args->cq_depth == 16) {
+ r3.field.token_depth_select = 2;
+ } else if (args->cq_depth == 32) {
+ r3.field.token_depth_select = 3;
+ } else if (args->cq_depth == 64) {
+ r3.field.token_depth_select = 4;
+ } else if (args->cq_depth == 128) {
+ r3.field.token_depth_select = 5;
+ } else if (args->cq_depth == 256) {
+ r3.field.token_depth_select = 6;
+ } else if (args->cq_depth == 512) {
+ r3.field.token_depth_select = 7;
+ } else if (args->cq_depth == 1024) {
+ r3.field.token_depth_select = 8;
+ } else {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: invalid CQ depth\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+ r3.val);
+
+ /*
+ * To support CQs with depth less than 8, program the token count
+ * register with a non-zero initial value. Operations such as domain
+ * reset must take this initial value into account when quiescing the
+ * CQ.
+ */
+ port->init_tkn_cnt = 0;
+
+ if (args->cq_depth < 8) {
+ union dlb2_lsp_cq_dir_tkn_cnt r13 = { {0} };
+
+ port->init_tkn_cnt = 8 - args->cq_depth;
+
+ r13.field.count = port->init_tkn_cnt;
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+ r13.val);
+ } else {
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+ DLB2_LSP_CQ_DIR_TKN_CNT_RST);
+ }
+
+ r4.field.token_depth_select = r3.field.token_depth_select;
+ r4.field.disable_wb_opt = 0;
+ r4.field.ignore_depth = 0;
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(port->id.phys_id),
+ r4.val);
+
+ /* Reset the CQ write pointer */
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_DIR_CQ_WPTR(port->id.phys_id),
+ DLB2_CHP_DIR_CQ_WPTR_RST);
+
+ /* Virtualize the PPID */
+ r9.field.keep_pf_ppid = 0;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_FMT(port->id.phys_id), r9.val);
+
+ /*
+ * Address translation (AT) settings: 0: untranslated, 2: translated
+ * (see ATS spec regarding Address Type field for more details)
+ */
+ r10.field.cq_at = 0;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_AT(port->id.phys_id), r10.val);
+
+ if (vdev_req && hw->virt_mode == DLB2_VIRT_SIOV) {
+ r11.field.pasid = hw->pasid[vdev_id];
+ r11.field.fmt2 = 1;
+ }
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_DIR_CQ_PASID(port->id.phys_id),
+ r11.val);
+
+ r12.field.cq2vas = domain->id.phys_id;
+
+ DLB2_CSR_WR(hw, DLB2_CHP_DIR_CQ2VAS(port->id.phys_id), r12.val);
+
+ return 0;
+}
+
+static int dlb2_configure_dir_port(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain,
+ struct dlb2_dir_pq_pair *port,
+ uintptr_t cq_dma_base,
+ struct dlb2_create_dir_port_args *args,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ int ret;
+
+ ret = dlb2_dir_port_configure_cq(hw,
+ domain,
+ port,
+ cq_dma_base,
+ args,
+ vdev_req,
+ vdev_id);
+
+ if (ret < 0)
+ return ret;
+
+ dlb2_dir_port_configure_pp(hw,
+ domain,
+ port,
+ vdev_req,
+ vdev_id);
+
+ dlb2_dir_port_cq_enable(hw, port);
+
+ port->enabled = true;
+
+ port->port_configured = true;
+
+ return 0;
+}
+
static int dlb2_ldb_port_map_qid_static(struct dlb2_hw *hw,
struct dlb2_ldb_port *p,
struct dlb2_ldb_queue *q,
@@ -2311,6 +2965,252 @@ int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
}
static void
+dlb2_log_create_ldb_port_args(struct dlb2_hw *hw,
+ u32 domain_id,
+ uintptr_t cq_dma_base,
+ struct dlb2_create_ldb_port_args *args,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ DLB2_HW_DBG(hw, "DLB2 create load-balanced port arguments:\n");
+ if (vdev_req)
+ DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+ DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+ domain_id);
+ DLB2_HW_DBG(hw, "\tCQ depth: %d\n",
+ args->cq_depth);
+ DLB2_HW_DBG(hw, "\tCQ hist list size: %d\n",
+ args->cq_history_list_size);
+ DLB2_HW_DBG(hw, "\tCQ base address: 0x%lx\n",
+ cq_dma_base);
+ DLB2_HW_DBG(hw, "\tCoS ID: %u\n", args->cos_id);
+ DLB2_HW_DBG(hw, "\tStrict CoS allocation: %u\n",
+ args->cos_strict);
+}
+
+/**
+ * dlb2_hw_create_ldb_port() - Allocate and initialize a load-balanced port and
+ * its resources.
+ * @hw: Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @cq_dma_base: Base DMA address for consumer queue memory
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_create_ldb_port_args *args,
+ uintptr_t cq_dma_base,
+ struct dlb2_cmd_response *resp,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_hw_domain *domain;
+ struct dlb2_ldb_port *port;
+ int ret, cos_id, i;
+
+ dlb2_log_create_ldb_port_args(hw,
+ domain_id,
+ cq_dma_base,
+ args,
+ vdev_req,
+ vdev_id);
+
+ /*
+ * Verify that hardware resources are available before attempting to
+ * satisfy the request. This simplifies the error unwinding code.
+ */
+ ret = dlb2_verify_create_ldb_port_args(hw,
+ domain_id,
+ cq_dma_base,
+ args,
+ resp,
+ vdev_req,
+ vdev_id);
+ if (ret)
+ return ret;
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+ if (!domain) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: domain not found\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ if (args->cos_strict) {
+ cos_id = args->cos_id;
+
+ port = DLB2_DOM_LIST_HEAD(domain->avail_ldb_ports[cos_id],
+ typeof(*port));
+ } else {
+ int idx;
+
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ idx = (args->cos_id + i) % DLB2_NUM_COS_DOMAINS;
+
+ port = DLB2_DOM_LIST_HEAD(domain->avail_ldb_ports[idx],
+ typeof(*port));
+ if (port)
+ break;
+ }
+
+ cos_id = idx;
+ }
+
+ if (!port) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: no available ldb ports\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ if (port->configured) {
+ DLB2_HW_ERR(hw,
+ "[%s()] Internal error: avail_ldb_ports contains configured ports.\n",
+ __func__);
+ return -EFAULT;
+ }
+
+ ret = dlb2_configure_ldb_port(hw,
+ domain,
+ port,
+ cq_dma_base,
+ args,
+ vdev_req,
+ vdev_id);
+ if (ret < 0)
+ return ret;
+
+ /*
+ * Configuration succeeded, so move the resource from the 'avail' to
+ * the 'used' list.
+ */
+ list_del(&port->domain_list);
+
+ list_add(&port->domain_list, &domain->used_ldb_ports[cos_id]);
+
+ resp->status = 0;
+ resp->id = (vdev_req) ? port->id.virt_id : port->id.phys_id;
+
+ return 0;
+}
+
+static void
+dlb2_log_create_dir_port_args(struct dlb2_hw *hw,
+ u32 domain_id,
+ uintptr_t cq_dma_base,
+ struct dlb2_create_dir_port_args *args,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ DLB2_HW_DBG(hw, "DLB2 create directed port arguments:\n");
+ if (vdev_req)
+ DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+ DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+ domain_id);
+ DLB2_HW_DBG(hw, "\tCQ depth: %d\n",
+ args->cq_depth);
+ DLB2_HW_DBG(hw, "\tCQ base address: 0x%lx\n",
+ cq_dma_base);
+}
+
+/**
+ * dlb2_hw_create_dir_port() - Allocate and initialize a DLB directed port
+ * and queue. The port/queue pair have the same ID and name.
+ * @hw: Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @cq_dma_base: Base DMA address for consumer queue memory
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_create_dir_port_args *args,
+ uintptr_t cq_dma_base,
+ struct dlb2_cmd_response *resp,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_dir_pq_pair *port;
+ struct dlb2_hw_domain *domain;
+ int ret;
+
+ dlb2_log_create_dir_port_args(hw,
+ domain_id,
+ cq_dma_base,
+ args,
+ vdev_req,
+ vdev_id);
+
+ /*
+ * Verify that hardware resources are available before attempting to
+ * satisfy the request. This simplifies the error unwinding code.
+ */
+ ret = dlb2_verify_create_dir_port_args(hw,
+ domain_id,
+ cq_dma_base,
+ args,
+ resp,
+ vdev_req,
+ vdev_id);
+ if (ret)
+ return ret;
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+ if (args->queue_id != -1)
+ port = dlb2_get_domain_used_dir_pq(args->queue_id,
+ vdev_req,
+ domain);
+ else
+ port = DLB2_DOM_LIST_HEAD(domain->avail_dir_pq_pairs,
+ typeof(*port));
+
+ if (!port) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: no available dir ports\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ ret = dlb2_configure_dir_port(hw,
+ domain,
+ port,
+ cq_dma_base,
+ args,
+ vdev_req,
+ vdev_id);
+ if (ret < 0)
+ return ret;
+
+ /*
+ * Configuration succeeded, so move the resource from the 'avail' to
+ * the 'used' list (if it's not already there).
+ */
+ if (args->queue_id == -1) {
+ list_del(&port->domain_list);
+
+ list_add(&port->domain_list, &domain->used_dir_pq_pairs);
+ }
+
+ resp->status = 0;
+ resp->id = (vdev_req) ? port->id.virt_id : port->id.phys_id;
+
+ return 0;
+}
+
+static void
dlb2_domain_finish_unmap_port_slot(struct dlb2_hw *hw,
struct dlb2_hw_domain *domain,
struct dlb2_ldb_port *port,
@@ -4105,3 +5005,25 @@ void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
}
+
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
+{
+ union dlb2_chp_cfg_chp_csr_ctrl r0;
+
+ r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
+
+ r0.field.cfg_64bytes_qe_ldb_cq_mode = 1;
+
+ DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
+}
+
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw)
+{
+ union dlb2_chp_cfg_chp_csr_ctrl r0;
+
+ r0.val = DLB2_CSR_RD(hw, DLB2_CHP_CFG_CHP_CSR_CTRL);
+
+ r0.field.cfg_64bytes_qe_dir_cq_mode = 1;
+
+ DLB2_CSR_WR(hw, DLB2_CHP_CFG_CHP_CSR_CTRL, r0.val);
+}
diff --git a/drivers/misc/dlb2/dlb2_resource.h b/drivers/misc/dlb2/dlb2_resource.h
index dfb42e0e13de..d355bc1a8f08 100644
--- a/drivers/misc/dlb2/dlb2_resource.h
+++ b/drivers/misc/dlb2/dlb2_resource.h
@@ -137,6 +137,78 @@ int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
unsigned int vdev_id);
/**
+ * dlb2_hw_create_dir_port() - create a directed port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port creation arguments.
+ * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a directed port.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the port ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
+ * pointer address is not properly aligned, the domain is not
+ * configured, or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_dir_port(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_create_dir_port_args *args,
+ uintptr_t cq_dma_base,
+ struct dlb2_cmd_response *resp,
+ bool vdev_request,
+ unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_ldb_port() - create a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port creation arguments.
+ * @cq_dma_base: base address of the CQ memory. This can be a PA or an IOVA.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a load-balanced port.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the port ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, a credit setting is invalid, a
+ * pointer address is not properly aligned, the domain is not
+ * configured, or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_ldb_port(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_create_ldb_port_args *args,
+ uintptr_t cq_dma_base,
+ struct dlb2_cmd_response *resp,
+ bool vdev_request,
+ unsigned int vdev_id);
+
+/**
* dlb2_reset_domain() - reset a scheduling domain
* @hw: dlb2_hw handle for a particular device.
* @domain_id: domain ID.
@@ -264,4 +336,21 @@ enum dlb2_virt_mode {
NUM_DLB2_VIRT_MODES,
};
+/**
+ * dlb2_hw_enable_sparse_ldb_cq_mode() - enable sparse mode for load-balanced
+ * ports.
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function must be called prior to configuring scheduling domains.
+ */
+void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw);
+
+/**
+ * dlb2_hw_enable_sparse_dir_cq_mode() - enable sparse mode for directed ports.
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function must be called prior to configuring scheduling domains.
+ */
+void dlb2_hw_enable_sparse_dir_cq_mode(struct dlb2_hw *hw);
+
#endif /* __DLB2_RESOURCE_H */
diff --git a/include/uapi/linux/dlb2_user.h b/include/uapi/linux/dlb2_user.h
index ee8f5a970065..df7bb72096cd 100644
--- a/include/uapi/linux/dlb2_user.h
+++ b/include/uapi/linux/dlb2_user.h
@@ -218,10 +218,32 @@ struct dlb2_get_num_resources_args {
__u32 num_dir_credits;
};
+enum dlb2_cq_poll_modes {
+ DLB2_CQ_POLL_MODE_STD,
+ DLB2_CQ_POLL_MODE_SPARSE,
+
+ /* NUM_DLB2_CQ_POLL_MODE must be last */
+ NUM_DLB2_CQ_POLL_MODE,
+};
+
+/*
+ * DLB2_CMD_QUERY_CQ_POLL_MODE: Query the CQ poll mode setting
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ * - response.id: CQ poll mode (see enum dlb2_cq_poll_modes).
+ */
+struct dlb2_query_cq_poll_mode_args {
+ /* Output parameters */
+ struct dlb2_cmd_response response;
+};
+
enum dlb2_user_interface_commands {
DLB2_CMD_GET_DEVICE_VERSION,
DLB2_CMD_CREATE_SCHED_DOMAIN,
DLB2_CMD_GET_NUM_RESOURCES,
+ DLB2_CMD_QUERY_CQ_POLL_MODE,
/* NUM_DLB2_CMD must be last */
NUM_DLB2_CMD,
@@ -339,16 +361,81 @@ struct dlb2_get_dir_queue_depth_args {
__u32 padding0;
};
+/*
+ * DLB2_DOMAIN_CMD_CREATE_LDB_PORT: Configure a load-balanced port.
+ * Input parameters:
+ * - cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and
+ * 1024, inclusive.
+ * - cq_depth_threshold: CQ depth interrupt threshold. A value of N means that
+ * the CQ interrupt won't fire until there are N or more outstanding CQ
+ * tokens.
+ * - num_hist_list_entries: Number of history list entries. This must be
+ * greater than or equal cq_depth.
+ * - cos_id: class-of-service to allocate this port from. Must be between 0 and
+ * 3, inclusive.
+ * - cos_strict: If set, return an error if there are no available ports in the
+ * requested class-of-service. Else, allocate the port from a different
+ * class-of-service if the requested class has no available ports.
+ *
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ * - response.id: port ID.
+ */
+
+struct dlb2_create_ldb_port_args {
+ /* Output parameters */
+ struct dlb2_cmd_response response;
+ /* Input parameters */
+ __u16 cq_depth;
+ __u16 cq_depth_threshold;
+ __u16 cq_history_list_size;
+ __u8 cos_id;
+ __u8 cos_strict;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_DIR_PORT: Configure a directed port.
+ * Input parameters:
+ * - cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and
+ * 1024, inclusive.
+ * - cq_depth_threshold: CQ depth interrupt threshold. A value of N means that
+ * the CQ interrupt won't fire until there are N or more outstanding CQ
+ * tokens.
+ * - qid: Queue ID. If the corresponding directed queue is already created,
+ * specify its ID here. Else this argument must be 0xFFFFFFFF to indicate
+ * that the port is being created before the queue.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Port ID.
+ */
+struct dlb2_create_dir_port_args {
+ /* Output parameters */
+ struct dlb2_cmd_response response;
+ /* Input parameters */
+ __u16 cq_depth;
+ __u16 cq_depth_threshold;
+ __s32 queue_id;
+};
+
enum dlb2_domain_user_interface_commands {
DLB2_DOMAIN_CMD_CREATE_LDB_QUEUE,
DLB2_DOMAIN_CMD_CREATE_DIR_QUEUE,
DLB2_DOMAIN_CMD_GET_LDB_QUEUE_DEPTH,
DLB2_DOMAIN_CMD_GET_DIR_QUEUE_DEPTH,
+ DLB2_DOMAIN_CMD_CREATE_LDB_PORT,
+ DLB2_DOMAIN_CMD_CREATE_DIR_PORT,
/* NUM_DLB2_DOMAIN_CMD must be last */
NUM_DLB2_DOMAIN_CMD,
};
+#define DLB2_CQ_SIZE 65536
+
/********************/
/* dlb2 ioctl codes */
/********************/
@@ -367,6 +454,10 @@ enum dlb2_domain_user_interface_commands {
_IOR(DLB2_IOC_MAGIC, \
DLB2_CMD_GET_NUM_RESOURCES, \
struct dlb2_get_num_resources_args)
+#define DLB2_IOC_QUERY_CQ_POLL_MODE \
+ _IOR(DLB2_IOC_MAGIC, \
+ DLB2_CMD_QUERY_CQ_POLL_MODE, \
+ struct dlb2_query_cq_poll_mode_args)
#define DLB2_IOC_CREATE_LDB_QUEUE \
_IOWR(DLB2_IOC_MAGIC, \
DLB2_DOMAIN_CMD_CREATE_LDB_QUEUE, \
@@ -383,5 +474,13 @@ enum dlb2_domain_user_interface_commands {
_IOWR(DLB2_IOC_MAGIC, \
DLB2_DOMAIN_CMD_GET_DIR_QUEUE_DEPTH, \
struct dlb2_get_dir_queue_depth_args)
+#define DLB2_IOC_CREATE_LDB_PORT \
+ _IOWR(DLB2_IOC_MAGIC, \
+ DLB2_DOMAIN_CMD_CREATE_LDB_PORT, \
+ struct dlb2_create_ldb_port_args)
+#define DLB2_IOC_CREATE_DIR_PORT \
+ _IOWR(DLB2_IOC_MAGIC, \
+ DLB2_DOMAIN_CMD_CREATE_DIR_PORT, \
+ struct dlb2_create_dir_port_args)
#endif /* __DLB2_USER_H */
--
2.13.6
The driver detects the device type (PF/VF) at probe time, and assigns the
corresponding 'ops' callbacks from that. These callbacks include mapping
and unmapping the PCI BAR space, creating/destroying the device, and
adding/deleting a char device.
Signed-off-by: Gage Eads <[email protected]>
Reviewed-by: Magnus Karlsson <[email protected]>
---
drivers/misc/dlb2/Makefile | 5 +-
drivers/misc/dlb2/dlb2_hw_types.h | 28 +++++++++
drivers/misc/dlb2/dlb2_main.c | 49 ++++++++++++++-
drivers/misc/dlb2/dlb2_main.h | 30 +++++++++
drivers/misc/dlb2/dlb2_pf_ops.c | 126 ++++++++++++++++++++++++++++++++++++++
5 files changed, 235 insertions(+), 3 deletions(-)
create mode 100644 drivers/misc/dlb2/dlb2_pf_ops.c
diff --git a/drivers/misc/dlb2/Makefile b/drivers/misc/dlb2/Makefile
index 90ae953d2a8f..95e67e5bd8ff 100644
--- a/drivers/misc/dlb2/Makefile
+++ b/drivers/misc/dlb2/Makefile
@@ -4,5 +4,6 @@
obj-$(CONFIG_INTEL_DLB2) := dlb2.o
-dlb2-objs := \
- dlb2_main.o \
+dlb2-objs := \
+ dlb2_main.o \
+ dlb2_pf_ops.o \
diff --git a/drivers/misc/dlb2/dlb2_hw_types.h b/drivers/misc/dlb2/dlb2_hw_types.h
index a0f908c603eb..10b26356535f 100644
--- a/drivers/misc/dlb2/dlb2_hw_types.h
+++ b/drivers/misc/dlb2/dlb2_hw_types.h
@@ -5,6 +5,25 @@
#ifndef __DLB2_HW_TYPES_H
#define __DLB2_HW_TYPES_H
+#include <linux/io.h>
+
+#define DLB2_PCI_REG_READ(addr) ioread32(addr)
+#define DLB2_PCI_REG_WRITE(reg, value) iowrite32(value, reg)
+
+/* Read/write register 'reg' in the CSR BAR space */
+#define DLB2_CSR_REG_ADDR(a, reg) ((a)->csr_kva + (reg))
+#define DLB2_CSR_RD(hw, reg) \
+ DLB2_PCI_REG_READ(DLB2_CSR_REG_ADDR((hw), (reg)))
+#define DLB2_CSR_WR(hw, reg, value) \
+ DLB2_PCI_REG_WRITE(DLB2_CSR_REG_ADDR((hw), (reg)), (value))
+
+/* Read/write register 'reg' in the func BAR space */
+#define DLB2_FUNC_REG_ADDR(a, reg) ((a)->func_kva + (reg))
+#define DLB2_FUNC_RD(hw, reg) \
+ DLB2_PCI_REG_READ(DLB2_FUNC_REG_ADDR((hw), (reg)))
+#define DLB2_FUNC_WR(hw, reg, value) \
+ DLB2_PCI_REG_WRITE(DLB2_FUNC_REG_ADDR((hw), (reg)), (value))
+
#define DLB2_MAX_NUM_VDEVS 16
#define DLB2_MAX_NUM_DOMAINS 32
#define DLB2_MAX_NUM_LDB_QUEUES 32 /* LDB == load-balanced */
@@ -26,4 +45,13 @@
#define DLB2_MAX_QID_EMPTY_CHECK_LOOPS (32 * 64 * 1024 * (800 / 30))
#define DLB2_HZ 800000000
+struct dlb2_hw {
+ /* BAR 0 address */
+ void __iomem *csr_kva;
+ unsigned long csr_phys_addr;
+ /* BAR 2 address */
+ void __iomem *func_kva;
+ unsigned long func_phys_addr;
+};
+
#endif /* __DLB2_HW_TYPES_H */
diff --git a/drivers/misc/dlb2/dlb2_main.c b/drivers/misc/dlb2/dlb2_main.c
index ffd6df788e2e..5d530ea862a3 100644
--- a/drivers/misc/dlb2/dlb2_main.c
+++ b/drivers/misc/dlb2/dlb2_main.c
@@ -51,6 +51,18 @@ static const struct file_operations dlb2_fops = {
/****** PCI driver callbacks ******/
/**********************************/
+static void dlb2_assign_ops(struct dlb2_dev *dlb2_dev,
+ const struct pci_device_id *pdev_id)
+{
+ dlb2_dev->type = pdev_id->driver_data;
+
+ switch (pdev_id->driver_data) {
+ case DLB2_PF:
+ dlb2_dev->ops = &dlb2_pf_ops;
+ break;
+ }
+}
+
static DEFINE_IDA(dlb2_ids);
static int dlb2_alloc_id(void)
@@ -73,6 +85,8 @@ static int dlb2_probe(struct pci_dev *pdev,
if (!dlb2_dev)
return -ENOMEM;
+ dlb2_assign_ops(dlb2_dev, pdev_id);
+
pci_set_drvdata(pdev, dlb2_dev);
dlb2_dev->pdev = pdev;
@@ -105,12 +119,39 @@ static int dlb2_probe(struct pci_dev *pdev,
if (pci_enable_pcie_error_reporting(pdev))
dev_info(&pdev->dev, "[%s()] Failed to enable AER\n", __func__);
+ ret = dlb2_dev->ops->map_pci_bar_space(dlb2_dev, pdev);
+ if (ret)
+ goto map_pci_bar_fail;
+
+ ret = dlb2_dev->ops->cdev_add(dlb2_dev,
+ dlb2_dev_number_base,
+ &dlb2_fops);
+ if (ret)
+ goto cdev_add_fail;
+
+ ret = dlb2_dev->ops->device_create(dlb2_dev, pdev, dlb2_class);
+ if (ret)
+ goto device_add_fail;
+
+ ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64));
+ if (ret)
+ goto dma_set_mask_fail;
+
mutex_lock(&dlb2_driver_lock);
list_add(&dlb2_dev->list, &dlb2_dev_list);
mutex_unlock(&dlb2_driver_lock);
return 0;
+dma_set_mask_fail:
+ dlb2_dev->ops->device_destroy(dlb2_dev, dlb2_class);
+device_add_fail:
+ dlb2_dev->ops->cdev_del(dlb2_dev);
+cdev_add_fail:
+ dlb2_dev->ops->unmap_pci_bar_space(dlb2_dev, pdev);
+map_pci_bar_fail:
+ pci_disable_pcie_error_reporting(pdev);
+ pci_release_regions(pdev);
pci_request_regions_fail:
pci_disable_device(pdev);
pci_enable_device_fail:
@@ -131,6 +172,12 @@ static void dlb2_remove(struct pci_dev *pdev)
list_del(&dlb2_dev->list);
mutex_unlock(&dlb2_driver_lock);
+ dlb2_dev->ops->device_destroy(dlb2_dev, dlb2_class);
+
+ dlb2_dev->ops->cdev_del(dlb2_dev);
+
+ dlb2_dev->ops->unmap_pci_bar_space(dlb2_dev, pdev);
+
pci_disable_pcie_error_reporting(pdev);
pci_release_regions(pdev);
@@ -143,7 +190,7 @@ static void dlb2_remove(struct pci_dev *pdev)
}
static struct pci_device_id dlb2_id_table[] = {
- { PCI_DEVICE_DATA(INTEL, DLB2_PF, NULL) },
+ { PCI_DEVICE_DATA(INTEL, DLB2_PF, DLB2_PF) },
{ 0 }
};
MODULE_DEVICE_TABLE(pci, dlb2_id_table);
diff --git a/drivers/misc/dlb2/dlb2_main.h b/drivers/misc/dlb2/dlb2_main.h
index cc05546fba13..7d04e0d8869d 100644
--- a/drivers/misc/dlb2/dlb2_main.h
+++ b/drivers/misc/dlb2/dlb2_main.h
@@ -28,10 +28,40 @@ static const char dlb2_driver_name[] = KBUILD_MODNAME;
#define DLB2_NUM_FUNCS_PER_DEVICE (1 + DLB2_MAX_NUM_VDEVS)
#define DLB2_MAX_NUM_DEVICES (DLB2_MAX_NUM_PFS * DLB2_NUM_FUNCS_PER_DEVICE)
+enum dlb2_device_type {
+ DLB2_PF,
+ DLB2_VF,
+};
+
+struct dlb2_dev;
+
+struct dlb2_device_ops {
+ int (*map_pci_bar_space)(struct dlb2_dev *dev, struct pci_dev *pdev);
+ void (*unmap_pci_bar_space)(struct dlb2_dev *dev,
+ struct pci_dev *pdev);
+ int (*device_create)(struct dlb2_dev *dlb2_dev,
+ struct pci_dev *pdev,
+ struct class *dlb2_class);
+ void (*device_destroy)(struct dlb2_dev *dlb2_dev,
+ struct class *dlb2_class);
+ int (*cdev_add)(struct dlb2_dev *dlb2_dev,
+ dev_t base,
+ const struct file_operations *fops);
+ void (*cdev_del)(struct dlb2_dev *dlb2_dev);
+};
+
+extern struct dlb2_device_ops dlb2_pf_ops;
+
struct dlb2_dev {
struct pci_dev *pdev;
+ struct dlb2_hw hw;
+ struct cdev cdev;
+ struct dlb2_device_ops *ops;
struct list_head list;
+ struct device *dlb2_device;
+ enum dlb2_device_type type;
int id;
+ dev_t dev_number;
};
#endif /* __DLB2_MAIN_H */
diff --git a/drivers/misc/dlb2/dlb2_pf_ops.c b/drivers/misc/dlb2/dlb2_pf_ops.c
new file mode 100644
index 000000000000..7920e491110f
--- /dev/null
+++ b/drivers/misc/dlb2/dlb2_pf_ops.c
@@ -0,0 +1,126 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(c) 2017-2020 Intel Corporation */
+
+#include "dlb2_main.h"
+
+/********************************/
+/****** PCI BAR management ******/
+/********************************/
+
+static void
+dlb2_pf_unmap_pci_bar_space(struct dlb2_dev *dlb2_dev,
+ struct pci_dev *pdev)
+{
+ pci_iounmap(pdev, dlb2_dev->hw.csr_kva);
+ pci_iounmap(pdev, dlb2_dev->hw.func_kva);
+}
+
+static int
+dlb2_pf_map_pci_bar_space(struct dlb2_dev *dlb2_dev,
+ struct pci_dev *pdev)
+{
+ /* BAR 0: PF FUNC BAR space */
+ dlb2_dev->hw.func_kva = pci_iomap(pdev, 0, 0);
+ dlb2_dev->hw.func_phys_addr = pci_resource_start(pdev, 0);
+
+ if (!dlb2_dev->hw.func_kva) {
+ dev_err(&pdev->dev, "Cannot iomap BAR 0 (size %llu)\n",
+ pci_resource_len(pdev, 0));
+
+ return -EIO;
+ }
+
+ /* BAR 2: PF CSR BAR space */
+ dlb2_dev->hw.csr_kva = pci_iomap(pdev, 2, 0);
+ dlb2_dev->hw.csr_phys_addr = pci_resource_start(pdev, 2);
+
+ if (!dlb2_dev->hw.csr_kva) {
+ dev_err(&pdev->dev, "Cannot iomap BAR 2 (size %llu)\n",
+ pci_resource_len(pdev, 2));
+
+ pci_iounmap(pdev, dlb2_dev->hw.func_kva);
+ return -EIO;
+ }
+
+ return 0;
+}
+
+/*******************************/
+/****** Driver management ******/
+/*******************************/
+
+static int
+dlb2_pf_cdev_add(struct dlb2_dev *dlb2_dev,
+ dev_t base,
+ const struct file_operations *fops)
+{
+ int ret;
+
+ dlb2_dev->dev_number = MKDEV(MAJOR(base), MINOR(base) + dlb2_dev->id);
+
+ cdev_init(&dlb2_dev->cdev, fops);
+
+ dlb2_dev->cdev.dev = dlb2_dev->dev_number;
+ dlb2_dev->cdev.owner = THIS_MODULE;
+
+ ret = cdev_add(&dlb2_dev->cdev, dlb2_dev->cdev.dev, 1);
+ if (ret < 0)
+ dev_err(dlb2_dev->dlb2_device,
+ "%s: cdev_add() returned %d\n",
+ dlb2_driver_name, ret);
+
+ return ret;
+}
+
+static void
+dlb2_pf_cdev_del(struct dlb2_dev *dlb2_dev)
+{
+ cdev_del(&dlb2_dev->cdev);
+}
+
+static int
+dlb2_pf_device_create(struct dlb2_dev *dlb2_dev,
+ struct pci_dev *pdev,
+ struct class *dlb2_class)
+{
+ /*
+ * Create a new device in order to create a /dev/dlb node. This device
+ * is a child of the DLB PCI device.
+ */
+ dlb2_dev->dlb2_device = device_create(dlb2_class,
+ &pdev->dev,
+ dlb2_dev->dev_number,
+ dlb2_dev,
+ "dlb%d",
+ dlb2_dev->id);
+
+ if (IS_ERR(dlb2_dev->dlb2_device)) {
+ dev_err(&pdev->dev,
+ "%s: device_create() returned %ld\n",
+ dlb2_driver_name, PTR_ERR(dlb2_dev->dlb2_device));
+
+ return PTR_ERR(dlb2_dev->dlb2_device);
+ }
+
+ return 0;
+}
+
+static void
+dlb2_pf_device_destroy(struct dlb2_dev *dlb2_dev,
+ struct class *dlb2_class)
+{
+ device_destroy(dlb2_class, dlb2_dev->dev_number);
+}
+
+/********************************/
+/****** DLB2 PF Device Ops ******/
+/********************************/
+
+struct dlb2_device_ops dlb2_pf_ops = {
+ .map_pci_bar_space = dlb2_pf_map_pci_bar_space,
+ .unmap_pci_bar_space = dlb2_pf_unmap_pci_bar_space,
+ .device_create = dlb2_pf_device_create,
+ .device_destroy = dlb2_pf_device_destroy,
+ .cdev_add = dlb2_pf_cdev_add,
+ .cdev_del = dlb2_pf_cdev_del,
+};
--
2.13.6
The driver's power-management policy is to put the device in D0 when in use
(when there are any open device files or memory mappings, or there are any
virtual devices), and leave it in D3Hot otherwise. This includes
resume/suspend callbacks; when the device resumes, the driver resets the
hardware to a known good state.
Signed-off-by: Gage Eads <[email protected]>
Reviewed-by: Magnus Karlsson <[email protected]>
---
drivers/misc/dlb2/dlb2_main.c | 72 +++++++++++++++++++++++++++++++++++++++++
drivers/misc/dlb2/dlb2_main.h | 2 ++
drivers/misc/dlb2/dlb2_pf_ops.c | 30 +++++++++++++++++
3 files changed, 104 insertions(+)
diff --git a/drivers/misc/dlb2/dlb2_main.c b/drivers/misc/dlb2/dlb2_main.c
index 2e9b8c2d479c..17a26e6856d1 100644
--- a/drivers/misc/dlb2/dlb2_main.c
+++ b/drivers/misc/dlb2/dlb2_main.c
@@ -61,11 +61,19 @@ static int dlb2_open(struct inode *i, struct file *f)
f->private_data = dev;
+ dev->ops->inc_pm_refcnt(dev->pdev, true);
+
return 0;
}
static int dlb2_close(struct inode *i, struct file *f)
{
+ struct dlb2_dev *dev;
+
+ dev = container_of(f->f_inode->i_cdev, struct dlb2_dev, cdev);
+
+ dev->ops->dec_pm_refcnt(dev->pdev);
+
return 0;
}
@@ -94,6 +102,8 @@ int dlb2_init_domain(struct dlb2_dev *dlb2_dev, u32 domain_id)
dlb2_dev->sched_domains[domain_id] = domain;
+ dlb2_dev->ops->inc_pm_refcnt(dlb2_dev->pdev, true);
+
return 0;
}
@@ -134,6 +144,8 @@ static int dlb2_domain_close(struct inode *i, struct file *f)
kref_put(&domain->refcnt, dlb2_free_domain);
+ dev->ops->dec_pm_refcnt(dev->pdev);
+
mutex_unlock(&dev->resource_mutex);
return ret;
@@ -262,6 +274,15 @@ static int dlb2_probe(struct pci_dev *pdev,
list_add(&dlb2_dev->list, &dlb2_dev_list);
mutex_unlock(&dlb2_driver_lock);
+ /*
+ * The driver puts the device to sleep (D3hot) while there are no
+ * scheduling domains to service. The usage counter of a PCI device at
+ * probe time is 2, so decrement it twice here. (The PCI layer has
+ * already called pm_runtime_enable().)
+ */
+ dlb2_dev->ops->dec_pm_refcnt(pdev);
+ dlb2_dev->ops->dec_pm_refcnt(pdev);
+
return 0;
init_driver_state_fail:
@@ -298,6 +319,10 @@ static void dlb2_remove(struct pci_dev *pdev)
list_del(&dlb2_dev->list);
mutex_unlock(&dlb2_driver_lock);
+ /* Undo the PM operations in dlb2_probe(). */
+ dlb2_dev->ops->inc_pm_refcnt(pdev, false);
+ dlb2_dev->ops->inc_pm_refcnt(pdev, false);
+
dlb2_dev->ops->free_driver_state(dlb2_dev);
dlb2_resource_free(&dlb2_dev->hw);
@@ -319,17 +344,64 @@ static void dlb2_remove(struct pci_dev *pdev)
devm_kfree(&pdev->dev, dlb2_dev);
}
+#ifdef CONFIG_PM
+static void dlb2_reset_hardware_state(struct dlb2_dev *dev)
+{
+ dlb2_reset_device(dev->pdev);
+
+ /* Reinitialize any other hardware state */
+ dev->ops->init_hardware(dev);
+}
+
+static int dlb2_runtime_suspend(struct device *dev)
+{
+ /* Return and let the PCI subsystem put the device in D3hot. */
+
+ return 0;
+}
+
+static int dlb2_runtime_resume(struct device *dev)
+{
+ struct pci_dev *pdev = container_of(dev, struct pci_dev, dev);
+ struct dlb2_dev *dlb2_dev = pci_get_drvdata(pdev);
+ int ret;
+
+ /*
+ * The PCI subsystem put the device in D0, but the device may not have
+ * completed powering up. Wait until the device is ready before
+ * proceeding.
+ */
+ ret = dlb2_dev->ops->wait_for_device_ready(dlb2_dev, pdev);
+ if (ret)
+ return ret;
+
+ /* Now reinitialize the device state. */
+ dlb2_reset_hardware_state(dlb2_dev);
+
+ return 0;
+}
+#endif
+
static struct pci_device_id dlb2_id_table[] = {
{ PCI_DEVICE_DATA(INTEL, DLB2_PF, DLB2_PF) },
{ 0 }
};
MODULE_DEVICE_TABLE(pci, dlb2_id_table);
+#ifdef CONFIG_PM
+static const struct dev_pm_ops dlb2_pm_ops = {
+ SET_RUNTIME_PM_OPS(dlb2_runtime_suspend, dlb2_runtime_resume, NULL)
+};
+#endif
+
static struct pci_driver dlb2_pci_driver = {
.name = (char *)dlb2_driver_name,
.id_table = dlb2_id_table,
.probe = dlb2_probe,
.remove = dlb2_remove,
+#ifdef CONFIG_PM
+ .driver.pm = &dlb2_pm_ops,
+#endif
};
static int __init dlb2_init_module(void)
diff --git a/drivers/misc/dlb2/dlb2_main.h b/drivers/misc/dlb2/dlb2_main.h
index 79378a47127f..86344f223649 100644
--- a/drivers/misc/dlb2/dlb2_main.h
+++ b/drivers/misc/dlb2/dlb2_main.h
@@ -41,6 +41,8 @@ struct dlb2_device_ops {
int (*map_pci_bar_space)(struct dlb2_dev *dev, struct pci_dev *pdev);
void (*unmap_pci_bar_space)(struct dlb2_dev *dev,
struct pci_dev *pdev);
+ void (*inc_pm_refcnt)(struct pci_dev *pdev, bool resume);
+ void (*dec_pm_refcnt)(struct pci_dev *pdev);
int (*init_driver_state)(struct dlb2_dev *dev);
void (*free_driver_state)(struct dlb2_dev *dev);
int (*device_create)(struct dlb2_dev *dlb2_dev,
diff --git a/drivers/misc/dlb2/dlb2_pf_ops.c b/drivers/misc/dlb2/dlb2_pf_ops.c
index d7bff677abda..6ca06406b0f2 100644
--- a/drivers/misc/dlb2/dlb2_pf_ops.c
+++ b/drivers/misc/dlb2/dlb2_pf_ops.c
@@ -2,11 +2,39 @@
/* Copyright(c) 2017-2020 Intel Corporation */
#include <linux/delay.h>
+#include <linux/pm_runtime.h>
#include "dlb2_main.h"
#include "dlb2_regs.h"
#include "dlb2_resource.h"
+/***********************************/
+/****** Runtime PM management ******/
+/***********************************/
+
+static void
+dlb2_pf_pm_inc_refcnt(struct pci_dev *pdev, bool resume)
+{
+ if (resume)
+ /*
+ * Increment the device's usage count and immediately wake it
+ * if it was suspended.
+ */
+ pm_runtime_get_sync(&pdev->dev);
+ else
+ pm_runtime_get_noresume(&pdev->dev);
+}
+
+static void
+dlb2_pf_pm_dec_refcnt(struct pci_dev *pdev)
+{
+ /*
+ * Decrement the device's usage count and suspend it if the
+ * count reaches zero.
+ */
+ pm_runtime_put_sync_suspend(&pdev->dev);
+}
+
/********************************/
/****** PCI BAR management ******/
/********************************/
@@ -207,6 +235,8 @@ dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 id)
struct dlb2_device_ops dlb2_pf_ops = {
.map_pci_bar_space = dlb2_pf_map_pci_bar_space,
.unmap_pci_bar_space = dlb2_pf_unmap_pci_bar_space,
+ .inc_pm_refcnt = dlb2_pf_pm_inc_refcnt,
+ .dec_pm_refcnt = dlb2_pf_pm_dec_refcnt,
.init_driver_state = dlb2_pf_init_driver_state,
.free_driver_state = dlb2_pf_free_driver_state,
.device_create = dlb2_pf_device_create,
--
2.13.6
This commit introduces the dlb2 device ioctl layer, and the first three
ioctls: query device version, query available resources, and create a
scheduling domain. This commit also introduces the user-space interface
file dlb2_user.h.
The device version query is designed to allow each DLB device version/type
to have its own unique ioctl API through the /dev/dlb%d node. Each such API
would share in common the device version command as its first command, and
all subsequent commands can be unique to the particular device.
The PF hardware operation for scheduling domain creation will be added in a
subsequent commit.
Signed-off-by: Gage Eads <[email protected]>
Reviewed-by: Magnus Karlsson <[email protected]>
---
Documentation/userspace-api/ioctl/ioctl-number.rst | 1 +
drivers/misc/dlb2/Makefile | 1 +
drivers/misc/dlb2/dlb2_bitmap.h | 63 ++++++++
drivers/misc/dlb2/dlb2_ioctl.c | 86 +++++++++++
drivers/misc/dlb2/dlb2_ioctl.h | 12 ++
drivers/misc/dlb2/dlb2_main.c | 6 +
drivers/misc/dlb2/dlb2_main.h | 7 +
drivers/misc/dlb2/dlb2_pf_ops.c | 21 +++
drivers/misc/dlb2/dlb2_resource.c | 46 ++++++
drivers/misc/dlb2/dlb2_resource.h | 22 +++
include/uapi/linux/dlb2_user.h | 163 +++++++++++++++++++++
11 files changed, 428 insertions(+)
create mode 100644 drivers/misc/dlb2/dlb2_ioctl.c
create mode 100644 drivers/misc/dlb2/dlb2_ioctl.h
create mode 100644 include/uapi/linux/dlb2_user.h
diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst
index 59472cd6a11d..5f84ea0dd3cb 100644
--- a/Documentation/userspace-api/ioctl/ioctl-number.rst
+++ b/Documentation/userspace-api/ioctl/ioctl-number.rst
@@ -241,6 +241,7 @@ Code Seq# Include File Comments
'h' 00-7F conflict! Charon filesystem
<mailto:[email protected]>
'h' 00-1F linux/hpet.h conflict!
+'h' 00-1F uapi/linux/dlb2_user.h conflict!
'h' 80-8F fs/hfsplus/ioctl.c
'i' 00-3F linux/i2o-dev.h conflict!
'i' 0B-1F linux/ipmi.h conflict!
diff --git a/drivers/misc/dlb2/Makefile b/drivers/misc/dlb2/Makefile
index 4fdf7ffc555b..18b5498b20e6 100644
--- a/drivers/misc/dlb2/Makefile
+++ b/drivers/misc/dlb2/Makefile
@@ -6,5 +6,6 @@ obj-$(CONFIG_INTEL_DLB2) := dlb2.o
dlb2-objs := \
dlb2_main.o \
+ dlb2_ioctl.o \
dlb2_pf_ops.o \
dlb2_resource.o \
diff --git a/drivers/misc/dlb2/dlb2_bitmap.h b/drivers/misc/dlb2/dlb2_bitmap.h
index c5bb4ba84d5c..2d2d2927b0ec 100644
--- a/drivers/misc/dlb2/dlb2_bitmap.h
+++ b/drivers/misc/dlb2/dlb2_bitmap.h
@@ -118,4 +118,67 @@ static inline int dlb2_bitmap_zero(struct dlb2_bitmap *bitmap)
return 0;
}
+/**
+ * dlb2_bitmap_count() - returns the number of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function looks for a single set bit.
+ *
+ * Return:
+ * Returns the number of set bits upon success, <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_count(struct dlb2_bitmap *bitmap)
+{
+ if (!bitmap || !bitmap->map)
+ return -EINVAL;
+
+ return bitmap_weight(bitmap->map, bitmap->len);
+}
+
+/**
+ * dlb2_bitmap_longest_set_range() - returns longest contiguous range of set
+ * bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * Return:
+ * Returns the bitmap's longest contiguous range of of set bits upon success,
+ * <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_longest_set_range(struct dlb2_bitmap *bitmap)
+{
+ unsigned int bits_per_long;
+ unsigned int i, j;
+ int max_len, len;
+
+ if (!bitmap || !bitmap->map)
+ return -EINVAL;
+
+ if (dlb2_bitmap_count(bitmap) == 0)
+ return 0;
+
+ max_len = 0;
+ len = 0;
+ bits_per_long = sizeof(unsigned long) * BITS_PER_BYTE;
+
+ for (i = 0; i < BITS_TO_LONGS(bitmap->len); i++) {
+ for (j = 0; j < bits_per_long; j++) {
+ if ((i * bits_per_long + j) >= bitmap->len)
+ break;
+
+ len = (test_bit(j, &bitmap->map[i])) ? len + 1 : 0;
+
+ if (len > max_len)
+ max_len = len;
+ }
+ }
+
+ return max_len;
+}
+
#endif /* __DLB2_OSDEP_BITMAP_H */
diff --git a/drivers/misc/dlb2/dlb2_ioctl.c b/drivers/misc/dlb2/dlb2_ioctl.c
new file mode 100644
index 000000000000..39f772d80215
--- /dev/null
+++ b/drivers/misc/dlb2/dlb2_ioctl.c
@@ -0,0 +1,86 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(c) 2017-2020 Intel Corporation */
+
+#include <linux/uaccess.h>
+
+#include <uapi/linux/dlb2_user.h>
+
+#include "dlb2_ioctl.h"
+#include "dlb2_main.h"
+
+/* [7:0]: device revision, [15:8]: device version */
+#define DLB2_SET_DEVICE_VERSION(ver, rev) (((ver) << 8) | (rev))
+
+static int dlb2_ioctl_get_device_version(struct dlb2_dev *dev,
+ unsigned long user_arg)
+{
+ struct dlb2_get_device_version_args arg;
+
+ arg.response.status = 0;
+ arg.response.id = DLB2_SET_DEVICE_VERSION(2, DLB2_REV_A0);
+
+ if (copy_to_user((void __user *)user_arg, &arg, sizeof(arg)))
+ return -EFAULT;
+
+ return 0;
+}
+
+static int dlb2_ioctl_create_sched_domain(struct dlb2_dev *dev,
+ unsigned long user_arg)
+{
+ struct dlb2_create_sched_domain_args arg;
+ struct dlb2_cmd_response response = {0};
+ int ret;
+
+ if (copy_from_user(&arg, (void __user *)user_arg, sizeof(arg)))
+ return -EFAULT;
+
+ mutex_lock(&dev->resource_mutex);
+
+ ret = dev->ops->create_sched_domain(&dev->hw, &arg, &response);
+
+ mutex_unlock(&dev->resource_mutex);
+
+ BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
+
+ if (copy_to_user((void __user *)user_arg, &response, sizeof(response)))
+ return -EFAULT;
+
+ return ret;
+}
+
+static int dlb2_ioctl_get_num_resources(struct dlb2_dev *dev,
+ unsigned long user_arg)
+{
+ struct dlb2_get_num_resources_args arg;
+ int ret;
+
+ mutex_lock(&dev->resource_mutex);
+
+ ret = dev->ops->get_num_resources(&dev->hw, &arg);
+
+ mutex_unlock(&dev->resource_mutex);
+
+ if (copy_to_user((void __user *)user_arg, &arg, sizeof(arg)))
+ return -EFAULT;
+
+ return ret;
+}
+
+long dlb2_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
+{
+ struct dlb2_dev *dev;
+
+ dev = container_of(f->f_inode->i_cdev, struct dlb2_dev, cdev);
+
+ switch (cmd) {
+ case DLB2_IOC_GET_DEVICE_VERSION:
+ return dlb2_ioctl_get_device_version(dev, arg);
+ case DLB2_IOC_CREATE_SCHED_DOMAIN:
+ return dlb2_ioctl_create_sched_domain(dev, arg);
+ case DLB2_IOC_GET_NUM_RESOURCES:
+ return dlb2_ioctl_get_num_resources(dev, arg);
+ default:
+ return -ENOTTY;
+ }
+}
diff --git a/drivers/misc/dlb2/dlb2_ioctl.h b/drivers/misc/dlb2/dlb2_ioctl.h
new file mode 100644
index 000000000000..367c93b98414
--- /dev/null
+++ b/drivers/misc/dlb2/dlb2_ioctl.h
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ * Copyright(c) 2017-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_IOCTL_H
+#define __DLB2_IOCTL_H
+
+#include "dlb2_main.h"
+
+long dlb2_ioctl(struct file *f, unsigned int cmd, unsigned long arg);
+
+#endif /* __DLB2_IOCTL_H */
diff --git a/drivers/misc/dlb2/dlb2_main.c b/drivers/misc/dlb2/dlb2_main.c
index f6d78d1532b0..a6e0eead7de4 100644
--- a/drivers/misc/dlb2/dlb2_main.c
+++ b/drivers/misc/dlb2/dlb2_main.c
@@ -11,12 +11,16 @@
#include <linux/pci.h>
#include <linux/uaccess.h>
+#include "dlb2_ioctl.h"
#include "dlb2_main.h"
#include "dlb2_resource.h"
static const char
dlb2_driver_copyright[] = "Copyright(c) 2018-2020 Intel Corporation";
+#define TO_STR2(s) #s
+#define TO_STR(s) TO_STR2(s)
+
MODULE_LICENSE("GPL v2");
MODULE_AUTHOR("Copyright(c) 2018-2020 Intel Corporation");
MODULE_DESCRIPTION("Intel(R) Dynamic Load Balancer 2.0 Driver");
@@ -63,6 +67,8 @@ static const struct file_operations dlb2_fops = {
.owner = THIS_MODULE,
.open = dlb2_open,
.release = dlb2_close,
+ .unlocked_ioctl = dlb2_ioctl,
+ .compat_ioctl = compat_ptr_ioctl,
};
/**********************************/
diff --git a/drivers/misc/dlb2/dlb2_main.h b/drivers/misc/dlb2/dlb2_main.h
index 115cfadc0f12..5445736d86b2 100644
--- a/drivers/misc/dlb2/dlb2_main.h
+++ b/drivers/misc/dlb2/dlb2_main.h
@@ -13,6 +13,8 @@
#include <linux/pci.h>
#include <linux/types.h>
+#include <uapi/linux/dlb2_user.h>
+
#include "dlb2_hw_types.h"
static const char dlb2_driver_name[] = KBUILD_MODNAME;
@@ -53,6 +55,11 @@ struct dlb2_device_ops {
void (*enable_pm)(struct dlb2_dev *dev);
int (*wait_for_device_ready)(struct dlb2_dev *dev,
struct pci_dev *pdev);
+ int (*create_sched_domain)(struct dlb2_hw *hw,
+ struct dlb2_create_sched_domain_args *args,
+ struct dlb2_cmd_response *resp);
+ int (*get_num_resources)(struct dlb2_hw *hw,
+ struct dlb2_get_num_resources_args *args);
void (*init_hardware)(struct dlb2_dev *dev);
};
diff --git a/drivers/misc/dlb2/dlb2_pf_ops.c b/drivers/misc/dlb2/dlb2_pf_ops.c
index cc35c3537507..1f5998862e31 100644
--- a/drivers/misc/dlb2/dlb2_pf_ops.c
+++ b/drivers/misc/dlb2/dlb2_pf_ops.c
@@ -175,6 +175,25 @@ dlb2_pf_init_hardware(struct dlb2_dev *dlb2_dev)
{
}
+/*****************************/
+/****** IOCTL callbacks ******/
+/*****************************/
+
+static int
+dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
+ struct dlb2_create_sched_domain_args *args,
+ struct dlb2_cmd_response *resp)
+{
+ return 0;
+}
+
+static int
+dlb2_pf_get_num_resources(struct dlb2_hw *hw,
+ struct dlb2_get_num_resources_args *args)
+{
+ return dlb2_hw_get_num_resources(hw, args, false, 0);
+}
+
/********************************/
/****** DLB2 PF Device Ops ******/
/********************************/
@@ -190,5 +209,7 @@ struct dlb2_device_ops dlb2_pf_ops = {
.cdev_del = dlb2_pf_cdev_del,
.enable_pm = dlb2_pf_enable_pm,
.wait_for_device_ready = dlb2_pf_wait_for_device_ready,
+ .create_sched_domain = dlb2_pf_create_sched_domain,
+ .get_num_resources = dlb2_pf_get_num_resources,
.init_hardware = dlb2_pf_init_hardware,
};
diff --git a/drivers/misc/dlb2/dlb2_resource.c b/drivers/misc/dlb2/dlb2_resource.c
index 70036a2f5d60..3b36bfef1e9b 100644
--- a/drivers/misc/dlb2/dlb2_resource.c
+++ b/drivers/misc/dlb2/dlb2_resource.c
@@ -185,6 +185,52 @@ int dlb2_resource_init(struct dlb2_hw *hw)
return ret;
}
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+ struct dlb2_get_num_resources_args *arg,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_function_resources *rsrcs;
+ struct dlb2_bitmap *map;
+ int i;
+
+ if (vdev_req && vdev_id >= DLB2_MAX_NUM_VDEVS)
+ return -EINVAL;
+
+ if (vdev_req)
+ rsrcs = &hw->vdev[vdev_id];
+ else
+ rsrcs = &hw->pf;
+
+ arg->num_sched_domains = rsrcs->num_avail_domains;
+
+ arg->num_ldb_queues = rsrcs->num_avail_ldb_queues;
+
+ arg->num_ldb_ports = 0;
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+ arg->num_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+ arg->num_cos_ldb_ports[i] = rsrcs->num_avail_ldb_ports[i];
+
+ arg->num_dir_ports = rsrcs->num_avail_dir_pq_pairs;
+
+ arg->num_atomic_inflights = rsrcs->num_avail_aqed_entries;
+
+ map = rsrcs->avail_hist_list_entries;
+
+ arg->num_hist_list_entries = dlb2_bitmap_count(map);
+
+ arg->max_contiguous_hist_list_entries =
+ dlb2_bitmap_longest_set_range(map);
+
+ arg->num_ldb_credits = rsrcs->num_avail_qed_entries;
+
+ arg->num_dir_credits = rsrcs->num_avail_dqed_entries;
+
+ return 0;
+}
+
void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
{
union dlb2_cfg_mstr_cfg_pm_pmcsr_disable r0;
diff --git a/drivers/misc/dlb2/dlb2_resource.h b/drivers/misc/dlb2/dlb2_resource.h
index af20fcf0a0cb..b23676df22a6 100644
--- a/drivers/misc/dlb2/dlb2_resource.h
+++ b/drivers/misc/dlb2/dlb2_resource.h
@@ -35,6 +35,28 @@ int dlb2_resource_init(struct dlb2_hw *hw);
void dlb2_resource_free(struct dlb2_hw *hw);
/**
+ * dlb2_hw_get_num_resources() - query the PCI function's available resources
+ * @hw: dlb2_hw handle for a particular device.
+ * @arg: pointer to resource counts.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the number of available resources for the PF or for a
+ * VF.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -EINVAL if vdev_request is true and vdev_id is
+ * invalid.
+ */
+int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
+ struct dlb2_get_num_resources_args *arg,
+ bool vdev_request,
+ unsigned int vdev_id);
+
+/**
* dlb2_clr_pmcsr_disable() - power on bulk of DLB 2.0 logic
* @hw: dlb2_hw handle for a particular device.
*
diff --git a/include/uapi/linux/dlb2_user.h b/include/uapi/linux/dlb2_user.h
new file mode 100644
index 000000000000..49a8a39f79a2
--- /dev/null
+++ b/include/uapi/linux/dlb2_user.h
@@ -0,0 +1,163 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_USER_H
+#define __DLB2_USER_H
+
+#include <linux/types.h>
+
+struct dlb2_cmd_response {
+ __u32 status; /* Interpret using enum dlb2_error */
+ __u32 id;
+};
+
+/********************************/
+/* 'dlb2' device file commands */
+/********************************/
+
+#define DLB2_DEVICE_VERSION(x) (((x) >> 8) & 0xFF)
+#define DLB2_DEVICE_REVISION(x) ((x) & 0xFF)
+
+enum dlb2_revisions {
+ DLB2_REV_A0 = 0,
+};
+
+/*
+ * DLB2_CMD_GET_DEVICE_VERSION: Query the DLB device version.
+ *
+ * Each DLB device version has its own unique ioctl API, but all share
+ * this as the first command in their interface, which tells user-space
+ * which API to use. The device revision is provided in case of any
+ * hardware errata.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ * - response.id[7:0]: Device revision.
+ * - response.id[15:8]: Device version.
+ */
+
+struct dlb2_get_device_version_args {
+ /* Output parameters */
+ struct dlb2_cmd_response response;
+};
+
+/*
+ * DLB2_CMD_CREATE_SCHED_DOMAIN: Create a DLB 2.0 scheduling domain and reserve
+ * its hardware resources. This command returns the newly created domain
+ * ID and a file descriptor for accessing the domain.
+ *
+ * Input parameters:
+ * - num_ldb_queues: Number of load-balanced queues.
+ * - num_ldb_ports: Number of load-balanced ports that can be allocated from
+ * from any class-of-service with available ports.
+ * - num_cos_ldb_ports[4]: Number of load-balanced ports from
+ * classes-of-service 0-3.
+ * - num_dir_ports: Number of directed ports. A directed port has one directed
+ * queue, so no num_dir_queues argument is necessary.
+ * - num_atomic_inflights: This specifies the amount of temporary atomic QE
+ * storage for the domain. This storage is divided among the domain's
+ * load-balanced queues that are configured for atomic scheduling.
+ * - num_hist_list_entries: Amount of history list storage. This is divided
+ * among the domain's CQs.
+ * - num_ldb_credits: Amount of load-balanced QE storage (QED). QEs occupy this
+ * space until they are scheduled to a load-balanced CQ. One credit
+ * represents the storage for one QE.
+ * - num_dir_credits: Amount of directed QE storage (DQED). QEs occupy this
+ * space until they are scheduled to a directed CQ. One credit represents
+ * the storage for one QE.
+ * - cos_strict: If set, return an error if there are insufficient ports in
+ * class-of-service N to satisfy the num_ldb_ports_cosN argument. If
+ * unset, attempt to fulfill num_ldb_ports_cosN arguments from other
+ * classes-of-service if class N does not contain enough free ports.
+ * - padding1: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ * - response.id: domain ID.
+ * - domain_fd: file descriptor for performing the domain's ioctl operations
+ * - padding0: Reserved for future use.
+ */
+struct dlb2_create_sched_domain_args {
+ /* Output parameters */
+ struct dlb2_cmd_response response;
+ __u32 domain_fd;
+ __u32 padding0;
+ /* Input parameters */
+ __u32 num_ldb_queues;
+ __u32 num_ldb_ports;
+ __u32 num_cos_ldb_ports[4];
+ __u32 num_dir_ports;
+ __u32 num_atomic_inflights;
+ __u32 num_hist_list_entries;
+ __u32 num_ldb_credits;
+ __u32 num_dir_credits;
+ __u8 cos_strict;
+ __u8 padding1[3];
+};
+
+/*
+ * DLB2_CMD_GET_NUM_RESOURCES: Return the number of available resources
+ * (queues, ports, etc.) that this device owns.
+ *
+ * Output parameters:
+ * - num_domains: Number of available scheduling domains.
+ * - num_ldb_queues: Number of available load-balanced queues.
+ * - num_ldb_ports: Total number of available load-balanced ports.
+ * - num_cos_ldb_ports[4]: Number of available load-balanced ports from
+ * classes-of-service 0-3.
+ * - num_dir_ports: Number of available directed ports. There is one directed
+ * queue for every directed port.
+ * - num_atomic_inflights: Amount of available temporary atomic QE storage.
+ * - num_hist_list_entries: Amount of history list storage.
+ * - max_contiguous_hist_list_entries: History list storage is allocated in
+ * a contiguous chunk, and this return value is the longest available
+ * contiguous range of history list entries.
+ * - num_ldb_credits: Amount of available load-balanced QE storage.
+ * - num_dir_credits: Amount of available directed QE storage.
+ */
+struct dlb2_get_num_resources_args {
+ /* Output parameters */
+ __u32 num_sched_domains;
+ __u32 num_ldb_queues;
+ __u32 num_ldb_ports;
+ __u32 num_cos_ldb_ports[4];
+ __u32 num_dir_ports;
+ __u32 num_atomic_inflights;
+ __u32 num_hist_list_entries;
+ __u32 max_contiguous_hist_list_entries;
+ __u32 num_ldb_credits;
+ __u32 num_dir_credits;
+};
+
+enum dlb2_user_interface_commands {
+ DLB2_CMD_GET_DEVICE_VERSION,
+ DLB2_CMD_CREATE_SCHED_DOMAIN,
+ DLB2_CMD_GET_NUM_RESOURCES,
+
+ /* NUM_DLB2_CMD must be last */
+ NUM_DLB2_CMD,
+};
+
+/********************/
+/* dlb2 ioctl codes */
+/********************/
+
+#define DLB2_IOC_MAGIC 'h'
+
+#define DLB2_IOC_GET_DEVICE_VERSION \
+ _IOR(DLB2_IOC_MAGIC, \
+ DLB2_CMD_GET_DEVICE_VERSION, \
+ struct dlb2_get_device_version_args)
+#define DLB2_IOC_CREATE_SCHED_DOMAIN \
+ _IOWR(DLB2_IOC_MAGIC, \
+ DLB2_CMD_CREATE_SCHED_DOMAIN, \
+ struct dlb2_create_sched_domain_args)
+#define DLB2_IOC_GET_NUM_RESOURCES \
+ _IOR(DLB2_IOC_MAGIC, \
+ DLB2_CMD_GET_NUM_RESOURCES, \
+ struct dlb2_get_num_resources_args)
+
+#endif /* __DLB2_USER_H */
--
2.13.6
When a user requests to create a scheduling domain, the requested resources
are validated then reserved for the scheduling domain. This takes place in
dlb2_resource.c. Finally, the ioctl handler allocates an anonymous file
descriptor for the domain.
Once created, user-space can use its file descriptor for the scheduling
domain to configure the domain's resources (to be added in a subsequent
commit). User-space can also get additional file descriptors (for
multiprocess usage).
The driver maintains a reference count for each scheduling domain,
incrementing it each time user-space requests a file descriptor and
decrementing it in the file's release callback.
When the reference count transitions from 1->0 the driver automatically
resets the scheduling domain's resources and makes them available for use
by future applications. This ensures that applications that crash without
explicitly cleaning up do not orphan device resources.
Broadly speaking, the scheduling domain reset logic progressively disables
resources and flushes/drains them. The driver uses iosubmit_cmds512() to
flush load-balanced and directed ports. At the end of the domain reset
function, it resets hardware (register) state and software state.
Signed-off-by: Gage Eads <[email protected]>
Reviewed-by: Magnus Karlsson <[email protected]>
---
drivers/misc/dlb2/dlb2_bitmap.h | 102 ++
drivers/misc/dlb2/dlb2_hw_types.h | 92 +
drivers/misc/dlb2/dlb2_ioctl.c | 42 +-
drivers/misc/dlb2/dlb2_main.c | 73 +
drivers/misc/dlb2/dlb2_main.h | 25 +
drivers/misc/dlb2/dlb2_pf_ops.c | 9 +-
drivers/misc/dlb2/dlb2_regs.h | 3619 +++++++++++++++++++++++++++++++++++++
drivers/misc/dlb2/dlb2_resource.c | 3132 ++++++++++++++++++++++++++++++++
drivers/misc/dlb2/dlb2_resource.h | 74 +
include/uapi/linux/dlb2_user.h | 86 +
10 files changed, 7252 insertions(+), 2 deletions(-)
diff --git a/drivers/misc/dlb2/dlb2_bitmap.h b/drivers/misc/dlb2/dlb2_bitmap.h
index 2d2d2927b0ec..ecc31c8b8288 100644
--- a/drivers/misc/dlb2/dlb2_bitmap.h
+++ b/drivers/misc/dlb2/dlb2_bitmap.h
@@ -119,6 +119,108 @@ static inline int dlb2_bitmap_zero(struct dlb2_bitmap *bitmap)
}
/**
+ * dlb2_bitmap_set_range() - set a range of bitmap entries
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ * length.
+ */
+static inline int dlb2_bitmap_set_range(struct dlb2_bitmap *bitmap,
+ unsigned int bit,
+ unsigned int len)
+{
+ if (!bitmap || !bitmap->map)
+ return -EINVAL;
+
+ if (bitmap->len <= bit)
+ return -EINVAL;
+
+ bitmap_set(bitmap->map, bit, len);
+
+ return 0;
+}
+
+/**
+ * dlb2_bitmap_clear_range() - clear a range of bitmap entries
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ * length.
+ */
+static inline int dlb2_bitmap_clear_range(struct dlb2_bitmap *bitmap,
+ unsigned int bit,
+ unsigned int len)
+{
+ if (!bitmap || !bitmap->map)
+ return -EINVAL;
+
+ if (bitmap->len <= bit)
+ return -EINVAL;
+
+ bitmap_clear(bitmap->map, bit, len);
+
+ return 0;
+}
+
+/**
+ * dlb2_bitmap_find_set_bit_range() - find an range of set bits
+ * @bitmap: pointer to dlb2_bitmap structure.
+ * @len: length of the range.
+ *
+ * This function looks for a range of set bits of length @len.
+ *
+ * Return:
+ * Returns the base bit index upon success, < 0 otherwise.
+ *
+ * Errors:
+ * ENOENT - unable to find a length *len* range of set bits.
+ * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
+ */
+static inline int dlb2_bitmap_find_set_bit_range(struct dlb2_bitmap *bitmap,
+ unsigned int len)
+{
+ struct dlb2_bitmap *complement_mask = NULL;
+ int ret;
+
+ if (!bitmap || !bitmap->map || len == 0)
+ return -EINVAL;
+
+ if (bitmap->len < len)
+ return -ENOENT;
+
+ ret = dlb2_bitmap_alloc(&complement_mask, bitmap->len);
+ if (ret)
+ return ret;
+
+ dlb2_bitmap_zero(complement_mask);
+
+ bitmap_complement(complement_mask->map, bitmap->map, bitmap->len);
+
+ ret = bitmap_find_next_zero_area(complement_mask->map,
+ complement_mask->len,
+ 0,
+ len,
+ 0);
+
+ dlb2_bitmap_free(complement_mask);
+
+ /* No set bit range of length len? */
+ return (ret >= (int)bitmap->len) ? -ENOENT : ret;
+}
+
+/**
* dlb2_bitmap_count() - returns the number of set bits
* @bitmap: pointer to dlb2_bitmap structure.
*
diff --git a/drivers/misc/dlb2/dlb2_hw_types.h b/drivers/misc/dlb2/dlb2_hw_types.h
index 0db398aeccd5..e43860a215ff 100644
--- a/drivers/misc/dlb2/dlb2_hw_types.h
+++ b/drivers/misc/dlb2/dlb2_hw_types.h
@@ -48,6 +48,29 @@
#define DLB2_MAX_QID_EMPTY_CHECK_LOOPS (32 * 64 * 1024 * (800 / 30))
#define DLB2_HZ 800000000
+/*
+ * Hardware-defined base addresses. Those prefixed 'DLB2_DRV' are only used by
+ * the PF driver.
+ */
+#define DLB2_DRV_LDB_PP_BASE 0x2300000
+#define DLB2_DRV_LDB_PP_STRIDE 0x1000
+#define DLB2_DRV_LDB_PP_BOUND (DLB2_DRV_LDB_PP_BASE + \
+ DLB2_DRV_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_DRV_DIR_PP_BASE 0x2200000
+#define DLB2_DRV_DIR_PP_STRIDE 0x1000
+#define DLB2_DRV_DIR_PP_BOUND (DLB2_DRV_DIR_PP_BASE + \
+ DLB2_DRV_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
+#define DLB2_LDB_PP_BASE 0x2100000
+#define DLB2_LDB_PP_STRIDE 0x1000
+#define DLB2_LDB_PP_BOUND (DLB2_LDB_PP_BASE + \
+ DLB2_LDB_PP_STRIDE * DLB2_MAX_NUM_LDB_PORTS)
+#define DLB2_LDB_PP_OFFS(id) (DLB2_LDB_PP_BASE + (id) * DLB2_PP_SIZE)
+#define DLB2_DIR_PP_BASE 0x2000000
+#define DLB2_DIR_PP_STRIDE 0x1000
+#define DLB2_DIR_PP_BOUND (DLB2_DIR_PP_BASE + \
+ DLB2_DIR_PP_STRIDE * DLB2_MAX_NUM_DIR_PORTS)
+#define DLB2_DIR_PP_OFFS(id) (DLB2_DIR_PP_BASE + (id) * DLB2_PP_SIZE)
+
struct dlb2_resource_id {
u32 phys_id;
u32 virt_id;
@@ -66,6 +89,29 @@ static inline u32 dlb2_freelist_count(struct dlb2_freelist *list)
return list->bound - list->base - list->offset;
}
+struct dlb2_hcw {
+ u64 data;
+ /* Word 3 */
+ u16 opaque;
+ u8 qid;
+ u8 sched_type:2;
+ u8 priority:3;
+ u8 msg_type:3;
+ /* Word 4 */
+ u16 lock_id;
+ u8 ts_flag:1;
+ u8 rsvd1:2;
+ u8 no_dec:1;
+ u8 cmp_id:4;
+ u8 cq_token:1;
+ u8 qe_comp:1;
+ u8 qe_frag:1;
+ u8 qe_valid:1;
+ u8 int_arm:1;
+ u8 error:1;
+ u8 rsvd:2;
+};
+
struct dlb2_ldb_queue {
struct list_head domain_list;
struct list_head func_list;
@@ -148,6 +194,49 @@ struct dlb2_sn_group {
u32 id;
};
+static inline bool dlb2_sn_group_full(struct dlb2_sn_group *group)
+{
+ u32 mask[] = {
+ 0x0000ffff, /* 64 SNs per queue */
+ 0x000000ff, /* 128 SNs per queue */
+ 0x0000000f, /* 256 SNs per queue */
+ 0x00000003, /* 512 SNs per queue */
+ 0x00000001}; /* 1024 SNs per queue */
+
+ return group->slot_use_bitmap == mask[group->mode];
+}
+
+static inline int dlb2_sn_group_alloc_slot(struct dlb2_sn_group *group)
+{
+ u32 bound[6] = {16, 8, 4, 2, 1};
+ u32 i;
+
+ for (i = 0; i < bound[group->mode]; i++) {
+ if (!(group->slot_use_bitmap & (1 << i))) {
+ group->slot_use_bitmap |= 1 << i;
+ return i;
+ }
+ }
+
+ return -1;
+}
+
+static inline void
+dlb2_sn_group_free_slot(struct dlb2_sn_group *group, int slot)
+{
+ group->slot_use_bitmap &= ~(1 << slot);
+}
+
+static inline int dlb2_sn_group_used_slots(struct dlb2_sn_group *group)
+{
+ int i, cnt = 0;
+
+ for (i = 0; i < 32; i++)
+ cnt += !!(group->slot_use_bitmap & (1 << i));
+
+ return cnt;
+}
+
struct dlb2_hw_domain {
struct dlb2_function_resources *parent_func;
struct list_head func_list;
@@ -225,6 +314,9 @@ struct dlb2_hw {
struct dlb2_function_resources vdev[DLB2_MAX_NUM_VDEVS];
struct dlb2_hw_domain domains[DLB2_MAX_NUM_DOMAINS];
u8 cos_reservation[DLB2_NUM_COS_DOMAINS];
+
+ /* Virtualization */
+ int virt_mode;
};
#endif /* __DLB2_HW_TYPES_H */
diff --git a/drivers/misc/dlb2/dlb2_ioctl.c b/drivers/misc/dlb2/dlb2_ioctl.c
index 39f772d80215..56ebd9387e31 100644
--- a/drivers/misc/dlb2/dlb2_ioctl.c
+++ b/drivers/misc/dlb2/dlb2_ioctl.c
@@ -1,6 +1,7 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright(c) 2017-2020 Intel Corporation */
+#include <linux/anon_inodes.h>
#include <linux/uaccess.h>
#include <uapi/linux/dlb2_user.h>
@@ -30,15 +31,54 @@ static int dlb2_ioctl_create_sched_domain(struct dlb2_dev *dev,
{
struct dlb2_create_sched_domain_args arg;
struct dlb2_cmd_response response = {0};
- int ret;
+ struct dlb2_domain *domain;
+ size_t offset;
+ int ret, fd;
if (copy_from_user(&arg, (void __user *)user_arg, sizeof(arg)))
return -EFAULT;
mutex_lock(&dev->resource_mutex);
+ if (dev->domain_reset_failed) {
+ response.status = DLB2_ST_DOMAIN_RESET_FAILED;
+ ret = -EINVAL;
+ goto unlock;
+ }
+
ret = dev->ops->create_sched_domain(&dev->hw, &arg, &response);
+ if (ret)
+ goto unlock;
+
+ ret = dlb2_init_domain(dev, response.id);
+ if (ret) {
+ dev->ops->reset_domain(&dev->hw, response.id);
+ goto unlock;
+ }
+
+ domain = dev->sched_domains[response.id];
+
+ fd = anon_inode_getfd("[dlb2domain]", &dlb2_domain_fops,
+ domain, O_RDWR);
+
+ if (fd < 0) {
+ dev_err(dev->dlb2_device,
+ "[%s()] Failed to get anon fd.\n", __func__);
+ kref_put(&domain->refcnt, dlb2_free_domain);
+ ret = fd;
+ goto unlock;
+ }
+
+ offset = offsetof(struct dlb2_create_sched_domain_args, domain_fd);
+
+ /* There's no reason this should fail, since the copy was validated by
+ * dlb2_copy_from_user() earlier in the function. Regardless, check for
+ * an error (but skip the unwind code).
+ */
+ if (copy_to_user((void __user *)user_arg + offset, &fd, sizeof(fd)))
+ return -EFAULT;
+unlock:
mutex_unlock(&dev->resource_mutex);
BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
diff --git a/drivers/misc/dlb2/dlb2_main.c b/drivers/misc/dlb2/dlb2_main.c
index a6e0eead7de4..2e9b8c2d479c 100644
--- a/drivers/misc/dlb2/dlb2_main.c
+++ b/drivers/misc/dlb2/dlb2_main.c
@@ -55,6 +55,12 @@ static int dlb2_reset_device(struct pci_dev *pdev)
static int dlb2_open(struct inode *i, struct file *f)
{
+ struct dlb2_dev *dev;
+
+ dev = container_of(f->f_inode->i_cdev, struct dlb2_dev, cdev);
+
+ f->private_data = dev;
+
return 0;
}
@@ -71,6 +77,73 @@ static const struct file_operations dlb2_fops = {
.compat_ioctl = compat_ptr_ioctl,
};
+int dlb2_init_domain(struct dlb2_dev *dlb2_dev, u32 domain_id)
+{
+ struct dlb2_domain *domain;
+
+ domain = devm_kzalloc(dlb2_dev->dlb2_device,
+ sizeof(*domain),
+ GFP_KERNEL);
+ if (!domain)
+ return -ENOMEM;
+
+ domain->id = domain_id;
+
+ kref_init(&domain->refcnt);
+ domain->dlb2_dev = dlb2_dev;
+
+ dlb2_dev->sched_domains[domain_id] = domain;
+
+ return 0;
+}
+
+static int __dlb2_free_domain(struct dlb2_dev *dev, struct dlb2_domain *domain)
+{
+ int ret = 0;
+
+ ret = dev->ops->reset_domain(&dev->hw, domain->id);
+ if (ret) {
+ dev->domain_reset_failed = true;
+ dev_err(dev->dlb2_device,
+ "Internal error: Domain reset failed. To recover, reset the device.\n");
+ }
+
+ dev->sched_domains[domain->id] = NULL;
+
+ devm_kfree(dev->dlb2_device, domain);
+
+ return ret;
+}
+
+void dlb2_free_domain(struct kref *kref)
+{
+ struct dlb2_domain *domain;
+
+ domain = container_of(kref, struct dlb2_domain, refcnt);
+
+ __dlb2_free_domain(domain->dlb2_dev, domain);
+}
+
+static int dlb2_domain_close(struct inode *i, struct file *f)
+{
+ struct dlb2_domain *domain = f->private_data;
+ struct dlb2_dev *dev = domain->dlb2_dev;
+ int ret = 0;
+
+ mutex_lock(&dev->resource_mutex);
+
+ kref_put(&domain->refcnt, dlb2_free_domain);
+
+ mutex_unlock(&dev->resource_mutex);
+
+ return ret;
+}
+
+const struct file_operations dlb2_domain_fops = {
+ .owner = THIS_MODULE,
+ .release = dlb2_domain_close,
+};
+
/**********************************/
/****** PCI driver callbacks ******/
/**********************************/
diff --git a/drivers/misc/dlb2/dlb2_main.h b/drivers/misc/dlb2/dlb2_main.h
index 5445736d86b2..79378a47127f 100644
--- a/drivers/misc/dlb2/dlb2_main.h
+++ b/drivers/misc/dlb2/dlb2_main.h
@@ -60,10 +60,18 @@ struct dlb2_device_ops {
struct dlb2_cmd_response *resp);
int (*get_num_resources)(struct dlb2_hw *hw,
struct dlb2_get_num_resources_args *args);
+ int (*reset_domain)(struct dlb2_hw *hw, u32 domain_id);
void (*init_hardware)(struct dlb2_dev *dev);
};
extern struct dlb2_device_ops dlb2_pf_ops;
+extern const struct file_operations dlb2_domain_fops;
+
+struct dlb2_domain {
+ struct dlb2_dev *dlb2_dev;
+ struct kref refcnt;
+ u8 id;
+};
struct dlb2_dev {
struct pci_dev *pdev;
@@ -72,6 +80,7 @@ struct dlb2_dev {
struct dlb2_device_ops *ops;
struct list_head list;
struct device *dlb2_device;
+ struct dlb2_domain *sched_domains[DLB2_MAX_NUM_DOMAINS];
/*
* The resource mutex serializes access to driver data structures and
* hardware registers.
@@ -80,6 +89,22 @@ struct dlb2_dev {
enum dlb2_device_type type;
int id;
dev_t dev_number;
+ u8 domain_reset_failed;
};
+int dlb2_init_domain(struct dlb2_dev *dlb2_dev, u32 domain_id);
+void dlb2_free_domain(struct kref *kref);
+
+#define DLB2_HW_ERR(dlb2, ...) do { \
+ struct dlb2_dev *dev; \
+ dev = container_of(dlb2, struct dlb2_dev, hw); \
+ dev_err(dev->dlb2_device, __VA_ARGS__); \
+} while (0)
+
+#define DLB2_HW_DBG(dlb2, ...) do { \
+ struct dlb2_dev *dev; \
+ dev = container_of(dlb2, struct dlb2_dev, hw); \
+ dev_dbg(dev->dlb2_device, __VA_ARGS__); \
+} while (0)
+
#endif /* __DLB2_MAIN_H */
diff --git a/drivers/misc/dlb2/dlb2_pf_ops.c b/drivers/misc/dlb2/dlb2_pf_ops.c
index 1f5998862e31..d7bff677abda 100644
--- a/drivers/misc/dlb2/dlb2_pf_ops.c
+++ b/drivers/misc/dlb2/dlb2_pf_ops.c
@@ -184,7 +184,7 @@ dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
struct dlb2_create_sched_domain_args *args,
struct dlb2_cmd_response *resp)
{
- return 0;
+ return dlb2_hw_create_sched_domain(hw, args, resp, false, 0);
}
static int
@@ -194,6 +194,12 @@ dlb2_pf_get_num_resources(struct dlb2_hw *hw,
return dlb2_hw_get_num_resources(hw, args, false, 0);
}
+static int
+dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 id)
+{
+ return dlb2_reset_domain(hw, id, false, 0);
+}
+
/********************************/
/****** DLB2 PF Device Ops ******/
/********************************/
@@ -211,5 +217,6 @@ struct dlb2_device_ops dlb2_pf_ops = {
.wait_for_device_ready = dlb2_pf_wait_for_device_ready,
.create_sched_domain = dlb2_pf_create_sched_domain,
.get_num_resources = dlb2_pf_get_num_resources,
+ .reset_domain = dlb2_pf_reset_domain,
.init_hardware = dlb2_pf_init_hardware,
};
diff --git a/drivers/misc/dlb2/dlb2_regs.h b/drivers/misc/dlb2/dlb2_regs.h
index b1a3ef2f4537..6b5ac6c2932d 100644
--- a/drivers/misc/dlb2/dlb2_regs.h
+++ b/drivers/misc/dlb2/dlb2_regs.h
@@ -7,6 +7,3554 @@
#include "linux/types.h"
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_BYTES 256
+#define DLB2_FUNC_PF_VF2PF_MAILBOX(vf_id, x) \
+ (0x1000 + 0x4 * (x) + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_RST 0x0
+union dlb2_func_pf_vf2pf_mailbox {
+ struct {
+ u32 msg : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR(vf_id) \
+ (0x1f00 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_MAILBOX_ISR_RST 0x0
+union dlb2_func_pf_vf2pf_mailbox_isr {
+ struct {
+ u32 vf0_isr : 1;
+ u32 vf1_isr : 1;
+ u32 vf2_isr : 1;
+ u32 vf3_isr : 1;
+ u32 vf4_isr : 1;
+ u32 vf5_isr : 1;
+ u32 vf6_isr : 1;
+ u32 vf7_isr : 1;
+ u32 vf8_isr : 1;
+ u32 vf9_isr : 1;
+ u32 vf10_isr : 1;
+ u32 vf11_isr : 1;
+ u32 vf12_isr : 1;
+ u32 vf13_isr : 1;
+ u32 vf14_isr : 1;
+ u32 vf15_isr : 1;
+ u32 rsvd0 : 16;
+ } field;
+ u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_FLR_ISR(vf_id) \
+ (0x1f04 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_FLR_ISR_RST 0x0
+union dlb2_func_pf_vf2pf_flr_isr {
+ struct {
+ u32 vf0_isr : 1;
+ u32 vf1_isr : 1;
+ u32 vf2_isr : 1;
+ u32 vf3_isr : 1;
+ u32 vf4_isr : 1;
+ u32 vf5_isr : 1;
+ u32 vf6_isr : 1;
+ u32 vf7_isr : 1;
+ u32 vf8_isr : 1;
+ u32 vf9_isr : 1;
+ u32 vf10_isr : 1;
+ u32 vf11_isr : 1;
+ u32 vf12_isr : 1;
+ u32 vf13_isr : 1;
+ u32 vf14_isr : 1;
+ u32 vf15_isr : 1;
+ u32 rsvd0 : 16;
+ } field;
+ u32 val;
+};
+
+#define DLB2_FUNC_PF_VF2PF_ISR_PEND(vf_id) \
+ (0x1f10 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF2PF_ISR_PEND_RST 0x0
+union dlb2_func_pf_vf2pf_isr_pend {
+ struct {
+ u32 isr_pend : 1;
+ u32 rsvd0 : 31;
+ } field;
+ u32 val;
+};
+
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_BYTES 64
+#define DLB2_FUNC_PF_PF2VF_MAILBOX(vf_id, x) \
+ (0x2000 + 0x4 * (x) + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_RST 0x0
+union dlb2_func_pf_pf2vf_mailbox {
+ struct {
+ u32 msg : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR(vf_id) \
+ (0x2f00 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_PF2VF_MAILBOX_ISR_RST 0x0
+union dlb2_func_pf_pf2vf_mailbox_isr {
+ struct {
+ u32 vf0_isr : 1;
+ u32 vf1_isr : 1;
+ u32 vf2_isr : 1;
+ u32 vf3_isr : 1;
+ u32 vf4_isr : 1;
+ u32 vf5_isr : 1;
+ u32 vf6_isr : 1;
+ u32 vf7_isr : 1;
+ u32 vf8_isr : 1;
+ u32 vf9_isr : 1;
+ u32 vf10_isr : 1;
+ u32 vf11_isr : 1;
+ u32 vf12_isr : 1;
+ u32 vf13_isr : 1;
+ u32 vf14_isr : 1;
+ u32 vf15_isr : 1;
+ u32 rsvd0 : 16;
+ } field;
+ u32 val;
+};
+
+#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS(vf_id) \
+ (0x3000 + (vf_id) * 0x10000)
+#define DLB2_FUNC_PF_VF_RESET_IN_PROGRESS_RST 0xffff
+union dlb2_func_pf_vf_reset_in_progress {
+ struct {
+ u32 vf0_reset_in_progress : 1;
+ u32 vf1_reset_in_progress : 1;
+ u32 vf2_reset_in_progress : 1;
+ u32 vf3_reset_in_progress : 1;
+ u32 vf4_reset_in_progress : 1;
+ u32 vf5_reset_in_progress : 1;
+ u32 vf6_reset_in_progress : 1;
+ u32 vf7_reset_in_progress : 1;
+ u32 vf8_reset_in_progress : 1;
+ u32 vf9_reset_in_progress : 1;
+ u32 vf10_reset_in_progress : 1;
+ u32 vf11_reset_in_progress : 1;
+ u32 vf12_reset_in_progress : 1;
+ u32 vf13_reset_in_progress : 1;
+ u32 vf14_reset_in_progress : 1;
+ u32 vf15_reset_in_progress : 1;
+ u32 rsvd0 : 16;
+ } field;
+ u32 val;
+};
+
+#define DLB2_MSIX_MEM_VECTOR_CTRL(x) \
+ (0x100000c + (x) * 0x10)
+#define DLB2_MSIX_MEM_VECTOR_CTRL_RST 0x1
+union dlb2_msix_mem_vector_ctrl {
+ struct {
+ u32 vec_mask : 1;
+ u32 rsvd0 : 31;
+ } field;
+ u32 val;
+};
+
+#define DLB2_IOSF_SMON_COMP_MASK1(x) \
+ (0x8002024 + (x) * 0x40)
+#define DLB2_IOSF_SMON_COMP_MASK1_RST 0xffffffff
+union dlb2_iosf_smon_comp_mask1 {
+ struct {
+ u32 comp_mask1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_IOSF_SMON_COMP_MASK0(x) \
+ (0x8002020 + (x) * 0x40)
+#define DLB2_IOSF_SMON_COMP_MASK0_RST 0xffffffff
+union dlb2_iosf_smon_comp_mask0 {
+ struct {
+ u32 comp_mask0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_IOSF_SMON_MAX_TMR(x) \
+ (0x800201c + (x) * 0x40)
+#define DLB2_IOSF_SMON_MAX_TMR_RST 0x0
+union dlb2_iosf_smon_max_tmr {
+ struct {
+ u32 maxvalue : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_IOSF_SMON_TMR(x) \
+ (0x8002018 + (x) * 0x40)
+#define DLB2_IOSF_SMON_TMR_RST 0x0
+union dlb2_iosf_smon_tmr {
+ struct {
+ u32 timer_val : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_IOSF_SMON_ACTIVITYCNTR1(x) \
+ (0x8002014 + (x) * 0x40)
+#define DLB2_IOSF_SMON_ACTIVITYCNTR1_RST 0x0
+union dlb2_iosf_smon_activitycntr1 {
+ struct {
+ u32 counter1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_IOSF_SMON_ACTIVITYCNTR0(x) \
+ (0x8002010 + (x) * 0x40)
+#define DLB2_IOSF_SMON_ACTIVITYCNTR0_RST 0x0
+union dlb2_iosf_smon_activitycntr0 {
+ struct {
+ u32 counter0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_IOSF_SMON_COMPARE1(x) \
+ (0x800200c + (x) * 0x40)
+#define DLB2_IOSF_SMON_COMPARE1_RST 0x0
+union dlb2_iosf_smon_compare1 {
+ struct {
+ u32 compare1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_IOSF_SMON_COMPARE0(x) \
+ (0x8002008 + (x) * 0x40)
+#define DLB2_IOSF_SMON_COMPARE0_RST 0x0
+union dlb2_iosf_smon_compare0 {
+ struct {
+ u32 compare0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_IOSF_SMON_CFG1(x) \
+ (0x8002004 + (x) * 0x40)
+#define DLB2_IOSF_SMON_CFG1_RST 0x0
+union dlb2_iosf_smon_cfg1 {
+ struct {
+ u32 mode0 : 8;
+ u32 mode1 : 8;
+ u32 rsvd : 16;
+ } field;
+ u32 val;
+};
+
+#define DLB2_IOSF_SMON_CFG0(x) \
+ (0x8002000 + (x) * 0x40)
+#define DLB2_IOSF_SMON_CFG0_RST 0x40000000
+union dlb2_iosf_smon_cfg0 {
+ struct {
+ u32 smon_enable : 1;
+ u32 rsvd2 : 3;
+ u32 smon0_function : 3;
+ u32 smon0_function_compare : 1;
+ u32 smon1_function : 3;
+ u32 smon1_function_compare : 1;
+ u32 smon_mode : 4;
+ u32 stopcounterovfl : 1;
+ u32 intcounterovfl : 1;
+ u32 statcounter0ovfl : 1;
+ u32 statcounter1ovfl : 1;
+ u32 stoptimerovfl : 1;
+ u32 inttimerovfl : 1;
+ u32 stattimerovfl : 1;
+ u32 rsvd1 : 1;
+ u32 timer_prescale : 5;
+ u32 rsvd0 : 1;
+ u32 version : 2;
+ } field;
+ u32 val;
+};
+
+#define DLB2_IOSF_FUNC_VF_BAR_DSBL(x) \
+ (0x20 + (x) * 0x4)
+#define DLB2_IOSF_FUNC_VF_BAR_DSBL_RST 0x0
+union dlb2_iosf_func_vf_bar_dsbl {
+ struct {
+ u32 func_vf_bar_dis : 1;
+ u32 rsvd0 : 31;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_TOTAL_VAS 0x1000011c
+#define DLB2_SYS_TOTAL_VAS_RST 0x20
+union dlb2_sys_total_vas {
+ struct {
+ u32 total_vas : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_PORTS 0x10000118
+#define DLB2_SYS_TOTAL_DIR_PORTS_RST 0x40
+union dlb2_sys_total_dir_ports {
+ struct {
+ u32 total_dir_ports : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_PORTS 0x10000114
+#define DLB2_SYS_TOTAL_LDB_PORTS_RST 0x40
+union dlb2_sys_total_ldb_ports {
+ struct {
+ u32 total_ldb_ports : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_QID 0x10000110
+#define DLB2_SYS_TOTAL_DIR_QID_RST 0x40
+union dlb2_sys_total_dir_qid {
+ struct {
+ u32 total_dir_qid : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_QID 0x1000010c
+#define DLB2_SYS_TOTAL_LDB_QID_RST 0x20
+union dlb2_sys_total_ldb_qid {
+ struct {
+ u32 total_ldb_qid : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_TOTAL_DIR_CRDS 0x10000108
+#define DLB2_SYS_TOTAL_DIR_CRDS_RST 0x1000
+union dlb2_sys_total_dir_crds {
+ struct {
+ u32 total_dir_credits : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_TOTAL_LDB_CRDS 0x10000104
+#define DLB2_SYS_TOTAL_LDB_CRDS_RST 0x2000
+union dlb2_sys_total_ldb_crds {
+ struct {
+ u32 total_ldb_credits : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND2 0x10000508
+#define DLB2_SYS_ALARM_PF_SYND2_RST 0x0
+union dlb2_sys_alarm_pf_synd2 {
+ struct {
+ u32 lock_id : 16;
+ u32 meas : 1;
+ u32 debug : 7;
+ u32 cq_pop : 1;
+ u32 qe_uhl : 1;
+ u32 qe_orsp : 1;
+ u32 qe_valid : 1;
+ u32 cq_int_rearm : 1;
+ u32 dsi_error : 1;
+ u32 rsvd0 : 2;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND1 0x10000504
+#define DLB2_SYS_ALARM_PF_SYND1_RST 0x0
+union dlb2_sys_alarm_pf_synd1 {
+ struct {
+ u32 dsi : 16;
+ u32 qid : 8;
+ u32 qtype : 2;
+ u32 qpri : 3;
+ u32 msg_type : 3;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_ALARM_PF_SYND0 0x10000500
+#define DLB2_SYS_ALARM_PF_SYND0_RST 0x0
+union dlb2_sys_alarm_pf_synd0 {
+ struct {
+ u32 syndrome : 8;
+ u32 rtype : 2;
+ u32 rsvd0 : 3;
+ u32 is_ldb : 1;
+ u32 cls : 2;
+ u32 aid : 6;
+ u32 unit : 4;
+ u32 source : 4;
+ u32 more : 1;
+ u32 valid : 1;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VPP_V(x) \
+ (0x10000f00 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VPP_V_RST 0x0
+union dlb2_sys_vf_ldb_vpp_v {
+ struct {
+ u32 vpp_v : 1;
+ u32 rsvd0 : 31;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VPP2PP(x) \
+ (0x10000f04 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VPP2PP_RST 0x0
+union dlb2_sys_vf_ldb_vpp2pp {
+ struct {
+ u32 pp : 6;
+ u32 rsvd0 : 26;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VPP_V(x) \
+ (0x10000f08 + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VPP_V_RST 0x0
+union dlb2_sys_vf_dir_vpp_v {
+ struct {
+ u32 vpp_v : 1;
+ u32 rsvd0 : 31;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VPP2PP(x) \
+ (0x10000f0c + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VPP2PP_RST 0x0
+union dlb2_sys_vf_dir_vpp2pp {
+ struct {
+ u32 pp : 6;
+ u32 rsvd0 : 26;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VQID_V(x) \
+ (0x10000f10 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VQID_V_RST 0x0
+union dlb2_sys_vf_ldb_vqid_v {
+ struct {
+ u32 vqid_v : 1;
+ u32 rsvd0 : 31;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_VF_LDB_VQID2QID(x) \
+ (0x10000f14 + (x) * 0x1000)
+#define DLB2_SYS_VF_LDB_VQID2QID_RST 0x0
+union dlb2_sys_vf_ldb_vqid2qid {
+ struct {
+ u32 qid : 5;
+ u32 rsvd0 : 27;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_LDB_QID2VQID(x) \
+ (0x10000f18 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID2VQID_RST 0x0
+union dlb2_sys_ldb_qid2vqid {
+ struct {
+ u32 vqid : 5;
+ u32 rsvd0 : 27;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VQID_V(x) \
+ (0x10000f1c + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VQID_V_RST 0x0
+union dlb2_sys_vf_dir_vqid_v {
+ struct {
+ u32 vqid_v : 1;
+ u32 rsvd0 : 31;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_VF_DIR_VQID2QID(x) \
+ (0x10000f20 + (x) * 0x1000)
+#define DLB2_SYS_VF_DIR_VQID2QID_RST 0x0
+union dlb2_sys_vf_dir_vqid2qid {
+ struct {
+ u32 qid : 6;
+ u32 rsvd0 : 26;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_LDB_VASQID_V(x) \
+ (0x10000f24 + (x) * 0x1000)
+#define DLB2_SYS_LDB_VASQID_V_RST 0x0
+union dlb2_sys_ldb_vasqid_v {
+ struct {
+ u32 vasqid_v : 1;
+ u32 rsvd0 : 31;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_DIR_VASQID_V(x) \
+ (0x10000f28 + (x) * 0x1000)
+#define DLB2_SYS_DIR_VASQID_V_RST 0x0
+union dlb2_sys_dir_vasqid_v {
+ struct {
+ u32 vasqid_v : 1;
+ u32 rsvd0 : 31;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND2(x) \
+ (0x10000f48 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND2_RST 0x0
+union dlb2_sys_alarm_vf_synd2 {
+ struct {
+ u32 lock_id : 16;
+ u32 debug : 8;
+ u32 cq_pop : 1;
+ u32 qe_uhl : 1;
+ u32 qe_orsp : 1;
+ u32 qe_valid : 1;
+ u32 isz : 1;
+ u32 dsi_error : 1;
+ u32 dlbrsvd : 2;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND1(x) \
+ (0x10000f44 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND1_RST 0x0
+union dlb2_sys_alarm_vf_synd1 {
+ struct {
+ u32 dsi : 16;
+ u32 qid : 8;
+ u32 qtype : 2;
+ u32 qpri : 3;
+ u32 msg_type : 3;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_ALARM_VF_SYND0(x) \
+ (0x10000f40 + (x) * 0x1000)
+#define DLB2_SYS_ALARM_VF_SYND0_RST 0x0
+union dlb2_sys_alarm_vf_synd0 {
+ struct {
+ u32 syndrome : 8;
+ u32 rtype : 2;
+ u32 vf_synd0_parity : 1;
+ u32 vf_synd1_parity : 1;
+ u32 vf_synd2_parity : 1;
+ u32 is_ldb : 1;
+ u32 cls : 2;
+ u32 aid : 6;
+ u32 unit : 4;
+ u32 source : 4;
+ u32 more : 1;
+ u32 valid : 1;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_CFG_V(x) \
+ (0x10000f58 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_CFG_V_RST 0x0
+union dlb2_sys_ldb_qid_cfg_v {
+ struct {
+ u32 sn_cfg_v : 1;
+ u32 fid_cfg_v : 1;
+ u32 rsvd0 : 30;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_ITS(x) \
+ (0x10000f54 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_ITS_RST 0x0
+union dlb2_sys_ldb_qid_its {
+ struct {
+ u32 qid_its : 1;
+ u32 rsvd0 : 31;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_LDB_QID_V(x) \
+ (0x10000f50 + (x) * 0x1000)
+#define DLB2_SYS_LDB_QID_V_RST 0x0
+union dlb2_sys_ldb_qid_v {
+ struct {
+ u32 qid_v : 1;
+ u32 rsvd0 : 31;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_DIR_QID_ITS(x) \
+ (0x10000f64 + (x) * 0x1000)
+#define DLB2_SYS_DIR_QID_ITS_RST 0x0
+union dlb2_sys_dir_qid_its {
+ struct {
+ u32 qid_its : 1;
+ u32 rsvd0 : 31;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_DIR_QID_V(x) \
+ (0x10000f60 + (x) * 0x1000)
+#define DLB2_SYS_DIR_QID_V_RST 0x0
+union dlb2_sys_dir_qid_v {
+ struct {
+ u32 qid_v : 1;
+ u32 rsvd0 : 31;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AI_DATA(x) \
+ (0x10000fa8 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AI_DATA_RST 0x0
+union dlb2_sys_ldb_cq_ai_data {
+ struct {
+ u32 cq_ai_data : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AI_ADDR(x) \
+ (0x10000fa4 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AI_ADDR_RST 0x0
+union dlb2_sys_ldb_cq_ai_addr {
+ struct {
+ u32 rsvd1 : 2;
+ u32 cq_ai_addr : 18;
+ u32 rsvd0 : 12;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_PASID(x) \
+ (0x10000fa0 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_PASID_RST 0x0
+union dlb2_sys_ldb_cq_pasid {
+ struct {
+ u32 pasid : 20;
+ u32 exe_req : 1;
+ u32 priv_req : 1;
+ u32 fmt2 : 1;
+ u32 rsvd0 : 9;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_AT(x) \
+ (0x10000f9c + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_AT_RST 0x0
+union dlb2_sys_ldb_cq_at {
+ struct {
+ u32 cq_at : 2;
+ u32 rsvd0 : 30;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ISR(x) \
+ (0x10000f98 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ISR_RST 0x0
+/* CQ Interrupt Modes */
+#define DLB2_CQ_ISR_MODE_DIS 0
+#define DLB2_CQ_ISR_MODE_MSI 1
+#define DLB2_CQ_ISR_MODE_MSIX 2
+#define DLB2_CQ_ISR_MODE_ADI 3
+union dlb2_sys_ldb_cq_isr {
+ struct {
+ u32 vector : 6;
+ u32 vf : 4;
+ u32 en_code : 2;
+ u32 rsvd0 : 20;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ2VF_PF_RO(x) \
+ (0x10000f94 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ2VF_PF_RO_RST 0x0
+union dlb2_sys_ldb_cq2vf_pf_ro {
+ struct {
+ u32 vf : 4;
+ u32 is_pf : 1;
+ u32 ro : 1;
+ u32 rsvd0 : 26;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_LDB_PP_V(x) \
+ (0x10000f90 + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP_V_RST 0x0
+union dlb2_sys_ldb_pp_v {
+ struct {
+ u32 pp_v : 1;
+ u32 rsvd0 : 31;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_LDB_PP2VDEV(x) \
+ (0x10000f8c + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP2VDEV_RST 0x0
+union dlb2_sys_ldb_pp2vdev {
+ struct {
+ u32 vdev : 4;
+ u32 rsvd0 : 28;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_LDB_PP2VAS(x) \
+ (0x10000f88 + (x) * 0x1000)
+#define DLB2_SYS_LDB_PP2VAS_RST 0x0
+union dlb2_sys_ldb_pp2vas {
+ struct {
+ u32 vas : 5;
+ u32 rsvd0 : 27;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ADDR_U(x) \
+ (0x10000f84 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ADDR_U_RST 0x0
+union dlb2_sys_ldb_cq_addr_u {
+ struct {
+ u32 addr_u : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_ADDR_L(x) \
+ (0x10000f80 + (x) * 0x1000)
+#define DLB2_SYS_LDB_CQ_ADDR_L_RST 0x0
+union dlb2_sys_ldb_cq_addr_l {
+ struct {
+ u32 rsvd0 : 6;
+ u32 addr_l : 26;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_FMT(x) \
+ (0x10000fec + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_FMT_RST 0x0
+union dlb2_sys_dir_cq_fmt {
+ struct {
+ u32 keep_pf_ppid : 1;
+ u32 rsvd0 : 31;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AI_DATA(x) \
+ (0x10000fe8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AI_DATA_RST 0x0
+union dlb2_sys_dir_cq_ai_data {
+ struct {
+ u32 cq_ai_data : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AI_ADDR(x) \
+ (0x10000fe4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AI_ADDR_RST 0x0
+union dlb2_sys_dir_cq_ai_addr {
+ struct {
+ u32 rsvd1 : 2;
+ u32 cq_ai_addr : 18;
+ u32 rsvd0 : 12;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_PASID(x) \
+ (0x10000fe0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_PASID_RST 0x0
+union dlb2_sys_dir_cq_pasid {
+ struct {
+ u32 pasid : 20;
+ u32 exe_req : 1;
+ u32 priv_req : 1;
+ u32 fmt2 : 1;
+ u32 rsvd0 : 9;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_AT(x) \
+ (0x10000fdc + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_AT_RST 0x0
+union dlb2_sys_dir_cq_at {
+ struct {
+ u32 cq_at : 2;
+ u32 rsvd0 : 30;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ISR(x) \
+ (0x10000fd8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ISR_RST 0x0
+union dlb2_sys_dir_cq_isr {
+ struct {
+ u32 vector : 6;
+ u32 vf : 4;
+ u32 en_code : 2;
+ u32 rsvd0 : 20;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ2VF_PF_RO(x) \
+ (0x10000fd4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ2VF_PF_RO_RST 0x0
+union dlb2_sys_dir_cq2vf_pf_ro {
+ struct {
+ u32 vf : 4;
+ u32 is_pf : 1;
+ u32 ro : 1;
+ u32 rsvd0 : 26;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_DIR_PP_V(x) \
+ (0x10000fd0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP_V_RST 0x0
+union dlb2_sys_dir_pp_v {
+ struct {
+ u32 pp_v : 1;
+ u32 rsvd0 : 31;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_DIR_PP2VDEV(x) \
+ (0x10000fcc + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP2VDEV_RST 0x0
+union dlb2_sys_dir_pp2vdev {
+ struct {
+ u32 vdev : 4;
+ u32 rsvd0 : 28;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_DIR_PP2VAS(x) \
+ (0x10000fc8 + (x) * 0x1000)
+#define DLB2_SYS_DIR_PP2VAS_RST 0x0
+union dlb2_sys_dir_pp2vas {
+ struct {
+ u32 vas : 5;
+ u32 rsvd0 : 27;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ADDR_U(x) \
+ (0x10000fc4 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ADDR_U_RST 0x0
+union dlb2_sys_dir_cq_addr_u {
+ struct {
+ u32 addr_u : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_ADDR_L(x) \
+ (0x10000fc0 + (x) * 0x1000)
+#define DLB2_SYS_DIR_CQ_ADDR_L_RST 0x0
+union dlb2_sys_dir_cq_addr_l {
+ struct {
+ u32 rsvd0 : 6;
+ u32 addr_l : 26;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_PM_SMON_COMP_MASK1 0x10003024
+#define DLB2_SYS_PM_SMON_COMP_MASK1_RST 0xffffffff
+union dlb2_sys_pm_smon_comp_mask1 {
+ struct {
+ u32 comp_mask1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_PM_SMON_COMP_MASK0 0x10003020
+#define DLB2_SYS_PM_SMON_COMP_MASK0_RST 0xffffffff
+union dlb2_sys_pm_smon_comp_mask0 {
+ struct {
+ u32 comp_mask0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_PM_SMON_MAX_TMR 0x1000301c
+#define DLB2_SYS_PM_SMON_MAX_TMR_RST 0x0
+union dlb2_sys_pm_smon_max_tmr {
+ struct {
+ u32 maxvalue : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_PM_SMON_TMR 0x10003018
+#define DLB2_SYS_PM_SMON_TMR_RST 0x0
+union dlb2_sys_pm_smon_tmr {
+ struct {
+ u32 timer_val : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_PM_SMON_ACTIVITYCNTR1 0x10003014
+#define DLB2_SYS_PM_SMON_ACTIVITYCNTR1_RST 0x0
+union dlb2_sys_pm_smon_activitycntr1 {
+ struct {
+ u32 counter1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_PM_SMON_ACTIVITYCNTR0 0x10003010
+#define DLB2_SYS_PM_SMON_ACTIVITYCNTR0_RST 0x0
+union dlb2_sys_pm_smon_activitycntr0 {
+ struct {
+ u32 counter0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_PM_SMON_COMPARE1 0x1000300c
+#define DLB2_SYS_PM_SMON_COMPARE1_RST 0x0
+union dlb2_sys_pm_smon_compare1 {
+ struct {
+ u32 compare1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_PM_SMON_COMPARE0 0x10003008
+#define DLB2_SYS_PM_SMON_COMPARE0_RST 0x0
+union dlb2_sys_pm_smon_compare0 {
+ struct {
+ u32 compare0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_PM_SMON_CFG1 0x10003004
+#define DLB2_SYS_PM_SMON_CFG1_RST 0x0
+union dlb2_sys_pm_smon_cfg1 {
+ struct {
+ u32 mode0 : 8;
+ u32 mode1 : 8;
+ u32 rsvd : 16;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_PM_SMON_CFG0 0x10003000
+#define DLB2_SYS_PM_SMON_CFG0_RST 0x40000000
+union dlb2_sys_pm_smon_cfg0 {
+ struct {
+ u32 smon_enable : 1;
+ u32 rsvd2 : 3;
+ u32 smon0_function : 3;
+ u32 smon0_function_compare : 1;
+ u32 smon1_function : 3;
+ u32 smon1_function_compare : 1;
+ u32 smon_mode : 4;
+ u32 stopcounterovfl : 1;
+ u32 intcounterovfl : 1;
+ u32 statcounter0ovfl : 1;
+ u32 statcounter1ovfl : 1;
+ u32 stoptimerovfl : 1;
+ u32 inttimerovfl : 1;
+ u32 stattimerovfl : 1;
+ u32 rsvd1 : 1;
+ u32 timer_prescale : 5;
+ u32 rsvd0 : 1;
+ u32 version : 2;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_SMON_COMP_MASK1(x) \
+ (0x18002024 + (x) * 0x40)
+#define DLB2_SYS_SMON_COMP_MASK1_RST 0xffffffff
+union dlb2_sys_smon_comp_mask1 {
+ struct {
+ u32 comp_mask1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_SMON_COMP_MASK0(x) \
+ (0x18002020 + (x) * 0x40)
+#define DLB2_SYS_SMON_COMP_MASK0_RST 0xffffffff
+union dlb2_sys_smon_comp_mask0 {
+ struct {
+ u32 comp_mask0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_SMON_MAX_TMR(x) \
+ (0x1800201c + (x) * 0x40)
+#define DLB2_SYS_SMON_MAX_TMR_RST 0x0
+union dlb2_sys_smon_max_tmr {
+ struct {
+ u32 maxvalue : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_SMON_TMR(x) \
+ (0x18002018 + (x) * 0x40)
+#define DLB2_SYS_SMON_TMR_RST 0x0
+union dlb2_sys_smon_tmr {
+ struct {
+ u32 timer_val : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_SMON_ACTIVITYCNTR1(x) \
+ (0x18002014 + (x) * 0x40)
+#define DLB2_SYS_SMON_ACTIVITYCNTR1_RST 0x0
+union dlb2_sys_smon_activitycntr1 {
+ struct {
+ u32 counter1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_SMON_ACTIVITYCNTR0(x) \
+ (0x18002010 + (x) * 0x40)
+#define DLB2_SYS_SMON_ACTIVITYCNTR0_RST 0x0
+union dlb2_sys_smon_activitycntr0 {
+ struct {
+ u32 counter0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_SMON_COMPARE1(x) \
+ (0x1800200c + (x) * 0x40)
+#define DLB2_SYS_SMON_COMPARE1_RST 0x0
+union dlb2_sys_smon_compare1 {
+ struct {
+ u32 compare1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_SMON_COMPARE0(x) \
+ (0x18002008 + (x) * 0x40)
+#define DLB2_SYS_SMON_COMPARE0_RST 0x0
+union dlb2_sys_smon_compare0 {
+ struct {
+ u32 compare0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_SMON_CFG1(x) \
+ (0x18002004 + (x) * 0x40)
+#define DLB2_SYS_SMON_CFG1_RST 0x0
+union dlb2_sys_smon_cfg1 {
+ struct {
+ u32 mode0 : 8;
+ u32 mode1 : 8;
+ u32 rsvd : 16;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_SMON_CFG0(x) \
+ (0x18002000 + (x) * 0x40)
+#define DLB2_SYS_SMON_CFG0_RST 0x40000000
+union dlb2_sys_smon_cfg0 {
+ struct {
+ u32 smon_enable : 1;
+ u32 rsvd2 : 3;
+ u32 smon0_function : 3;
+ u32 smon0_function_compare : 1;
+ u32 smon1_function : 3;
+ u32 smon1_function_compare : 1;
+ u32 smon_mode : 4;
+ u32 stopcounterovfl : 1;
+ u32 intcounterovfl : 1;
+ u32 statcounter0ovfl : 1;
+ u32 statcounter1ovfl : 1;
+ u32 stoptimerovfl : 1;
+ u32 inttimerovfl : 1;
+ u32 stattimerovfl : 1;
+ u32 rsvd1 : 1;
+ u32 timer_prescale : 5;
+ u32 rsvd0 : 1;
+ u32 version : 2;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_INGRESS_ALARM_ENBL 0x10000300
+#define DLB2_SYS_INGRESS_ALARM_ENBL_RST 0x0
+union dlb2_sys_ingress_alarm_enbl {
+ struct {
+ u32 illegal_hcw : 1;
+ u32 illegal_pp : 1;
+ u32 illegal_pasid : 1;
+ u32 illegal_qid : 1;
+ u32 disabled_qid : 1;
+ u32 illegal_ldb_qid_cfg : 1;
+ u32 rsvd0 : 26;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_MSIX_ACK 0x10000400
+#define DLB2_SYS_MSIX_ACK_RST 0x0
+union dlb2_sys_msix_ack {
+ struct {
+ u32 msix_0_ack : 1;
+ u32 msix_1_ack : 1;
+ u32 rsvd0 : 30;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_MSIX_PASSTHRU 0x10000404
+#define DLB2_SYS_MSIX_PASSTHRU_RST 0x0
+union dlb2_sys_msix_passthru {
+ struct {
+ u32 msix_0_passthru : 1;
+ u32 msix_1_passthru : 1;
+ u32 rsvd0 : 30;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_MSIX_MODE 0x10000408
+#define DLB2_SYS_MSIX_MODE_RST 0x0
+/* MSI-X Modes */
+#define DLB2_MSIX_MODE_PACKED 0
+#define DLB2_MSIX_MODE_COMPRESSED 1
+union dlb2_sys_msix_mode {
+ struct {
+ u32 mode : 1;
+ u32 poll_mode : 1;
+ u32 poll_mask : 1;
+ u32 poll_lock : 1;
+ u32 rsvd0 : 28;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS 0x10000440
+#define DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS_RST 0x0
+union dlb2_sys_dir_cq_31_0_occ_int_sts {
+ struct {
+ u32 cq_0_occ_int : 1;
+ u32 cq_1_occ_int : 1;
+ u32 cq_2_occ_int : 1;
+ u32 cq_3_occ_int : 1;
+ u32 cq_4_occ_int : 1;
+ u32 cq_5_occ_int : 1;
+ u32 cq_6_occ_int : 1;
+ u32 cq_7_occ_int : 1;
+ u32 cq_8_occ_int : 1;
+ u32 cq_9_occ_int : 1;
+ u32 cq_10_occ_int : 1;
+ u32 cq_11_occ_int : 1;
+ u32 cq_12_occ_int : 1;
+ u32 cq_13_occ_int : 1;
+ u32 cq_14_occ_int : 1;
+ u32 cq_15_occ_int : 1;
+ u32 cq_16_occ_int : 1;
+ u32 cq_17_occ_int : 1;
+ u32 cq_18_occ_int : 1;
+ u32 cq_19_occ_int : 1;
+ u32 cq_20_occ_int : 1;
+ u32 cq_21_occ_int : 1;
+ u32 cq_22_occ_int : 1;
+ u32 cq_23_occ_int : 1;
+ u32 cq_24_occ_int : 1;
+ u32 cq_25_occ_int : 1;
+ u32 cq_26_occ_int : 1;
+ u32 cq_27_occ_int : 1;
+ u32 cq_28_occ_int : 1;
+ u32 cq_29_occ_int : 1;
+ u32 cq_30_occ_int : 1;
+ u32 cq_31_occ_int : 1;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS 0x10000444
+#define DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS_RST 0x0
+union dlb2_sys_dir_cq_63_32_occ_int_sts {
+ struct {
+ u32 cq_32_occ_int : 1;
+ u32 cq_33_occ_int : 1;
+ u32 cq_34_occ_int : 1;
+ u32 cq_35_occ_int : 1;
+ u32 cq_36_occ_int : 1;
+ u32 cq_37_occ_int : 1;
+ u32 cq_38_occ_int : 1;
+ u32 cq_39_occ_int : 1;
+ u32 cq_40_occ_int : 1;
+ u32 cq_41_occ_int : 1;
+ u32 cq_42_occ_int : 1;
+ u32 cq_43_occ_int : 1;
+ u32 cq_44_occ_int : 1;
+ u32 cq_45_occ_int : 1;
+ u32 cq_46_occ_int : 1;
+ u32 cq_47_occ_int : 1;
+ u32 cq_48_occ_int : 1;
+ u32 cq_49_occ_int : 1;
+ u32 cq_50_occ_int : 1;
+ u32 cq_51_occ_int : 1;
+ u32 cq_52_occ_int : 1;
+ u32 cq_53_occ_int : 1;
+ u32 cq_54_occ_int : 1;
+ u32 cq_55_occ_int : 1;
+ u32 cq_56_occ_int : 1;
+ u32 cq_57_occ_int : 1;
+ u32 cq_58_occ_int : 1;
+ u32 cq_59_occ_int : 1;
+ u32 cq_60_occ_int : 1;
+ u32 cq_61_occ_int : 1;
+ u32 cq_62_occ_int : 1;
+ u32 cq_63_occ_int : 1;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS 0x10000460
+#define DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS_RST 0x0
+union dlb2_sys_ldb_cq_31_0_occ_int_sts {
+ struct {
+ u32 cq_0_occ_int : 1;
+ u32 cq_1_occ_int : 1;
+ u32 cq_2_occ_int : 1;
+ u32 cq_3_occ_int : 1;
+ u32 cq_4_occ_int : 1;
+ u32 cq_5_occ_int : 1;
+ u32 cq_6_occ_int : 1;
+ u32 cq_7_occ_int : 1;
+ u32 cq_8_occ_int : 1;
+ u32 cq_9_occ_int : 1;
+ u32 cq_10_occ_int : 1;
+ u32 cq_11_occ_int : 1;
+ u32 cq_12_occ_int : 1;
+ u32 cq_13_occ_int : 1;
+ u32 cq_14_occ_int : 1;
+ u32 cq_15_occ_int : 1;
+ u32 cq_16_occ_int : 1;
+ u32 cq_17_occ_int : 1;
+ u32 cq_18_occ_int : 1;
+ u32 cq_19_occ_int : 1;
+ u32 cq_20_occ_int : 1;
+ u32 cq_21_occ_int : 1;
+ u32 cq_22_occ_int : 1;
+ u32 cq_23_occ_int : 1;
+ u32 cq_24_occ_int : 1;
+ u32 cq_25_occ_int : 1;
+ u32 cq_26_occ_int : 1;
+ u32 cq_27_occ_int : 1;
+ u32 cq_28_occ_int : 1;
+ u32 cq_29_occ_int : 1;
+ u32 cq_30_occ_int : 1;
+ u32 cq_31_occ_int : 1;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS 0x10000464
+#define DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS_RST 0x0
+union dlb2_sys_ldb_cq_63_32_occ_int_sts {
+ struct {
+ u32 cq_32_occ_int : 1;
+ u32 cq_33_occ_int : 1;
+ u32 cq_34_occ_int : 1;
+ u32 cq_35_occ_int : 1;
+ u32 cq_36_occ_int : 1;
+ u32 cq_37_occ_int : 1;
+ u32 cq_38_occ_int : 1;
+ u32 cq_39_occ_int : 1;
+ u32 cq_40_occ_int : 1;
+ u32 cq_41_occ_int : 1;
+ u32 cq_42_occ_int : 1;
+ u32 cq_43_occ_int : 1;
+ u32 cq_44_occ_int : 1;
+ u32 cq_45_occ_int : 1;
+ u32 cq_46_occ_int : 1;
+ u32 cq_47_occ_int : 1;
+ u32 cq_48_occ_int : 1;
+ u32 cq_49_occ_int : 1;
+ u32 cq_50_occ_int : 1;
+ u32 cq_51_occ_int : 1;
+ u32 cq_52_occ_int : 1;
+ u32 cq_53_occ_int : 1;
+ u32 cq_54_occ_int : 1;
+ u32 cq_55_occ_int : 1;
+ u32 cq_56_occ_int : 1;
+ u32 cq_57_occ_int : 1;
+ u32 cq_58_occ_int : 1;
+ u32 cq_59_occ_int : 1;
+ u32 cq_60_occ_int : 1;
+ u32 cq_61_occ_int : 1;
+ u32 cq_62_occ_int : 1;
+ u32 cq_63_occ_int : 1;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_DIR_CQ_OPT_CLR 0x100004c0
+#define DLB2_SYS_DIR_CQ_OPT_CLR_RST 0x0
+union dlb2_sys_dir_cq_opt_clr {
+ struct {
+ u32 cq : 6;
+ u32 rsvd0 : 26;
+ } field;
+ u32 val;
+};
+
+#define DLB2_SYS_ALARM_HW_SYND 0x1000050c
+#define DLB2_SYS_ALARM_HW_SYND_RST 0x0
+union dlb2_sys_alarm_hw_synd {
+ struct {
+ u32 syndrome : 8;
+ u32 rtype : 2;
+ u32 alarm : 1;
+ u32 cwd : 1;
+ u32 vf_pf_mb : 1;
+ u32 rsvd0 : 1;
+ u32 cls : 2;
+ u32 aid : 6;
+ u32 unit : 4;
+ u32 source : 4;
+ u32 more : 1;
+ u32 valid : 1;
+ } field;
+ u32 val;
+};
+
+#define DLB2_AQED_PIPE_QID_FID_LIM(x) \
+ (0x20000000 + (x) * 0x1000)
+#define DLB2_AQED_PIPE_QID_FID_LIM_RST 0x7ff
+union dlb2_aqed_pipe_qid_fid_lim {
+ struct {
+ u32 qid_fid_limit : 13;
+ u32 rsvd0 : 19;
+ } field;
+ u32 val;
+};
+
+#define DLB2_AQED_PIPE_QID_HID_WIDTH(x) \
+ (0x20080000 + (x) * 0x1000)
+#define DLB2_AQED_PIPE_QID_HID_WIDTH_RST 0x0
+union dlb2_aqed_pipe_qid_hid_width {
+ struct {
+ u32 compress_code : 3;
+ u32 rsvd0 : 29;
+ } field;
+ u32 val;
+};
+
+#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0 0x24000004
+#define DLB2_AQED_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATM_0_RST 0xfefcfaf8
+union dlb2_aqed_pipe_cfg_arb_weights_tqpri_atm_0 {
+ struct {
+ u32 pri0 : 8;
+ u32 pri1 : 8;
+ u32 pri2 : 8;
+ u32 pri3 : 8;
+ } field;
+ u32 val;
+};
+
+#define DLB2_AQED_PIPE_SMON_ACTIVITYCNTR0 0x2c00004c
+#define DLB2_AQED_PIPE_SMON_ACTIVITYCNTR0_RST 0x0
+union dlb2_aqed_pipe_smon_activitycntr0 {
+ struct {
+ u32 counter0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_AQED_PIPE_SMON_ACTIVITYCNTR1 0x2c000050
+#define DLB2_AQED_PIPE_SMON_ACTIVITYCNTR1_RST 0x0
+union dlb2_aqed_pipe_smon_activitycntr1 {
+ struct {
+ u32 counter1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_AQED_PIPE_SMON_COMPARE0 0x2c000054
+#define DLB2_AQED_PIPE_SMON_COMPARE0_RST 0x0
+union dlb2_aqed_pipe_smon_compare0 {
+ struct {
+ u32 compare0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_AQED_PIPE_SMON_COMPARE1 0x2c000058
+#define DLB2_AQED_PIPE_SMON_COMPARE1_RST 0x0
+union dlb2_aqed_pipe_smon_compare1 {
+ struct {
+ u32 compare1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_AQED_PIPE_SMON_CFG0 0x2c00005c
+#define DLB2_AQED_PIPE_SMON_CFG0_RST 0x40000000
+union dlb2_aqed_pipe_smon_cfg0 {
+ struct {
+ u32 smon_enable : 1;
+ u32 smon_0trigger_enable : 1;
+ u32 rsvz0 : 2;
+ u32 smon0_function : 3;
+ u32 smon0_function_compare : 1;
+ u32 smon1_function : 3;
+ u32 smon1_function_compare : 1;
+ u32 smon_mode : 4;
+ u32 stopcounterovfl : 1;
+ u32 intcounterovfl : 1;
+ u32 statcounter0ovfl : 1;
+ u32 statcounter1ovfl : 1;
+ u32 stoptimerovfl : 1;
+ u32 inttimerovfl : 1;
+ u32 stattimerovfl : 1;
+ u32 rsvz1 : 1;
+ u32 timer_prescale : 5;
+ u32 rsvz2 : 1;
+ u32 version : 2;
+ } field;
+ u32 val;
+};
+
+#define DLB2_AQED_PIPE_SMON_CFG1 0x2c000060
+#define DLB2_AQED_PIPE_SMON_CFG1_RST 0x0
+union dlb2_aqed_pipe_smon_cfg1 {
+ struct {
+ u32 mode0 : 8;
+ u32 mode1 : 8;
+ u32 rsvz0 : 16;
+ } field;
+ u32 val;
+};
+
+#define DLB2_AQED_PIPE_SMON_MAX_TMR 0x2c000064
+#define DLB2_AQED_PIPE_SMON_MAX_TMR_RST 0x0
+union dlb2_aqed_pipe_smon_max_tmr {
+ struct {
+ u32 maxvalue : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_AQED_PIPE_SMON_TMR 0x2c000068
+#define DLB2_AQED_PIPE_SMON_TMR_RST 0x0
+union dlb2_aqed_pipe_smon_tmr {
+ struct {
+ u32 timer : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_ATM_QID2CQIDIX_00(x) \
+ (0x30080000 + (x) * 0x1000)
+#define DLB2_ATM_QID2CQIDIX_00_RST 0x0
+#define DLB2_ATM_QID2CQIDIX(x, y) \
+ (DLB2_ATM_QID2CQIDIX_00(x) + 0x80000 * (y))
+#define DLB2_ATM_QID2CQIDIX_NUM 16
+union dlb2_atm_qid2cqidix_00 {
+ struct {
+ u32 cq_p0 : 8;
+ u32 cq_p1 : 8;
+ u32 cq_p2 : 8;
+ u32 cq_p3 : 8;
+ } field;
+ u32 val;
+};
+
+#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN 0x34000004
+#define DLB2_ATM_CFG_ARB_WEIGHTS_RDY_BIN_RST 0xfffefdfc
+union dlb2_atm_cfg_arb_weights_rdy_bin {
+ struct {
+ u32 bin0 : 8;
+ u32 bin1 : 8;
+ u32 bin2 : 8;
+ u32 bin3 : 8;
+ } field;
+ u32 val;
+};
+
+#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN 0x34000008
+#define DLB2_ATM_CFG_ARB_WEIGHTS_SCHED_BIN_RST 0xfffefdfc
+union dlb2_atm_cfg_arb_weights_sched_bin {
+ struct {
+ u32 bin0 : 8;
+ u32 bin1 : 8;
+ u32 bin2 : 8;
+ u32 bin3 : 8;
+ } field;
+ u32 val;
+};
+
+#define DLB2_ATM_SMON_ACTIVITYCNTR0 0x3c000050
+#define DLB2_ATM_SMON_ACTIVITYCNTR0_RST 0x0
+union dlb2_atm_smon_activitycntr0 {
+ struct {
+ u32 counter0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_ATM_SMON_ACTIVITYCNTR1 0x3c000054
+#define DLB2_ATM_SMON_ACTIVITYCNTR1_RST 0x0
+union dlb2_atm_smon_activitycntr1 {
+ struct {
+ u32 counter1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_ATM_SMON_COMPARE0 0x3c000058
+#define DLB2_ATM_SMON_COMPARE0_RST 0x0
+union dlb2_atm_smon_compare0 {
+ struct {
+ u32 compare0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_ATM_SMON_COMPARE1 0x3c00005c
+#define DLB2_ATM_SMON_COMPARE1_RST 0x0
+union dlb2_atm_smon_compare1 {
+ struct {
+ u32 compare1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_ATM_SMON_CFG0 0x3c000060
+#define DLB2_ATM_SMON_CFG0_RST 0x40000000
+union dlb2_atm_smon_cfg0 {
+ struct {
+ u32 smon_enable : 1;
+ u32 smon_0trigger_enable : 1;
+ u32 rsvz0 : 2;
+ u32 smon0_function : 3;
+ u32 smon0_function_compare : 1;
+ u32 smon1_function : 3;
+ u32 smon1_function_compare : 1;
+ u32 smon_mode : 4;
+ u32 stopcounterovfl : 1;
+ u32 intcounterovfl : 1;
+ u32 statcounter0ovfl : 1;
+ u32 statcounter1ovfl : 1;
+ u32 stoptimerovfl : 1;
+ u32 inttimerovfl : 1;
+ u32 stattimerovfl : 1;
+ u32 rsvz1 : 1;
+ u32 timer_prescale : 5;
+ u32 rsvz2 : 1;
+ u32 version : 2;
+ } field;
+ u32 val;
+};
+
+#define DLB2_ATM_SMON_CFG1 0x3c000064
+#define DLB2_ATM_SMON_CFG1_RST 0x0
+union dlb2_atm_smon_cfg1 {
+ struct {
+ u32 mode0 : 8;
+ u32 mode1 : 8;
+ u32 rsvz0 : 16;
+ } field;
+ u32 val;
+};
+
+#define DLB2_ATM_SMON_MAX_TMR 0x3c000068
+#define DLB2_ATM_SMON_MAX_TMR_RST 0x0
+union dlb2_atm_smon_max_tmr {
+ struct {
+ u32 maxvalue : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_ATM_SMON_TMR 0x3c00006c
+#define DLB2_ATM_SMON_TMR_RST 0x0
+union dlb2_atm_smon_tmr {
+ struct {
+ u32 timer : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_VAS_CRD(x) \
+ (0x40000000 + (x) * 0x1000)
+#define DLB2_CHP_CFG_DIR_VAS_CRD_RST 0x0
+union dlb2_chp_cfg_dir_vas_crd {
+ struct {
+ u32 count : 14;
+ u32 rsvd0 : 18;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_VAS_CRD(x) \
+ (0x40080000 + (x) * 0x1000)
+#define DLB2_CHP_CFG_LDB_VAS_CRD_RST 0x0
+union dlb2_chp_cfg_ldb_vas_crd {
+ struct {
+ u32 count : 15;
+ u32 rsvd0 : 17;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_ORD_QID_SN(x) \
+ (0x40100000 + (x) * 0x1000)
+#define DLB2_CHP_ORD_QID_SN_RST 0x0
+union dlb2_chp_ord_qid_sn {
+ struct {
+ u32 sn : 10;
+ u32 rsvd0 : 22;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_ORD_QID_SN_MAP(x) \
+ (0x40180000 + (x) * 0x1000)
+#define DLB2_CHP_ORD_QID_SN_MAP_RST 0x0
+union dlb2_chp_ord_qid_sn_map {
+ struct {
+ u32 mode : 3;
+ u32 slot : 4;
+ u32 rsvz0 : 1;
+ u32 grp : 1;
+ u32 rsvz1 : 1;
+ u32 rsvd0 : 22;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_SN_CHK_ENBL(x) \
+ (0x40200000 + (x) * 0x1000)
+#define DLB2_CHP_SN_CHK_ENBL_RST 0x0
+union dlb2_chp_sn_chk_enbl {
+ struct {
+ u32 en : 1;
+ u32 rsvd0 : 31;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_DEPTH(x) \
+ (0x40280000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_DEPTH_RST 0x0
+union dlb2_chp_dir_cq_depth {
+ struct {
+ u32 depth : 13;
+ u32 rsvd0 : 19;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(x) \
+ (0x40300000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST 0x0
+union dlb2_chp_dir_cq_int_depth_thrsh {
+ struct {
+ u32 depth_threshold : 13;
+ u32 rsvd0 : 19;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INT_ENB(x) \
+ (0x40380000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_INT_ENB_RST 0x0
+union dlb2_chp_dir_cq_int_enb {
+ struct {
+ u32 en_tim : 1;
+ u32 en_depth : 1;
+ u32 rsvd0 : 30;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_TMR_THRSH(x) \
+ (0x40480000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_TMR_THRSH_RST 0x1
+union dlb2_chp_dir_cq_tmr_thrsh {
+ struct {
+ u32 thrsh_0 : 1;
+ u32 thrsh_13_1 : 13;
+ u32 rsvd0 : 18;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(x) \
+ (0x40500000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST 0x0
+union dlb2_chp_dir_cq_tkn_depth_sel {
+ struct {
+ u32 token_depth_select : 4;
+ u32 rsvd0 : 28;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_WD_ENB(x) \
+ (0x40580000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_WD_ENB_RST 0x0
+union dlb2_chp_dir_cq_wd_enb {
+ struct {
+ u32 wd_enable : 1;
+ u32 rsvd0 : 31;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_WPTR(x) \
+ (0x40600000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ_WPTR_RST 0x0
+union dlb2_chp_dir_cq_wptr {
+ struct {
+ u32 write_pointer : 13;
+ u32 rsvd0 : 19;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ2VAS(x) \
+ (0x40680000 + (x) * 0x1000)
+#define DLB2_CHP_DIR_CQ2VAS_RST 0x0
+union dlb2_chp_dir_cq2vas {
+ struct {
+ u32 cq2vas : 5;
+ u32 rsvd0 : 27;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_BASE(x) \
+ (0x40700000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_BASE_RST 0x0
+union dlb2_chp_hist_list_base {
+ struct {
+ u32 base : 13;
+ u32 rsvd0 : 19;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_LIM(x) \
+ (0x40780000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_LIM_RST 0x0
+union dlb2_chp_hist_list_lim {
+ struct {
+ u32 limit : 13;
+ u32 rsvd0 : 19;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_POP_PTR(x) \
+ (0x40800000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_POP_PTR_RST 0x0
+union dlb2_chp_hist_list_pop_ptr {
+ struct {
+ u32 pop_ptr : 13;
+ u32 generation : 1;
+ u32 rsvd0 : 18;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_HIST_LIST_PUSH_PTR(x) \
+ (0x40880000 + (x) * 0x1000)
+#define DLB2_CHP_HIST_LIST_PUSH_PTR_RST 0x0
+union dlb2_chp_hist_list_push_ptr {
+ struct {
+ u32 push_ptr : 13;
+ u32 generation : 1;
+ u32 rsvd0 : 18;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_DEPTH(x) \
+ (0x40900000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_DEPTH_RST 0x0
+union dlb2_chp_ldb_cq_depth {
+ struct {
+ u32 depth : 11;
+ u32 rsvd0 : 21;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(x) \
+ (0x40980000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST 0x0
+union dlb2_chp_ldb_cq_int_depth_thrsh {
+ struct {
+ u32 depth_threshold : 11;
+ u32 rsvd0 : 21;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INT_ENB(x) \
+ (0x40a00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_INT_ENB_RST 0x0
+union dlb2_chp_ldb_cq_int_enb {
+ struct {
+ u32 en_tim : 1;
+ u32 en_depth : 1;
+ u32 rsvd0 : 30;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_TMR_THRSH(x) \
+ (0x40b00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_TMR_THRSH_RST 0x1
+union dlb2_chp_ldb_cq_tmr_thrsh {
+ struct {
+ u32 thrsh_0 : 1;
+ u32 thrsh_13_1 : 13;
+ u32 rsvd0 : 18;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(x) \
+ (0x40b80000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST 0x0
+union dlb2_chp_ldb_cq_tkn_depth_sel {
+ struct {
+ u32 token_depth_select : 4;
+ u32 rsvd0 : 28;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_WD_ENB(x) \
+ (0x40c00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_WD_ENB_RST 0x0
+union dlb2_chp_ldb_cq_wd_enb {
+ struct {
+ u32 wd_enable : 1;
+ u32 rsvd0 : 31;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_WPTR(x) \
+ (0x40c80000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ_WPTR_RST 0x0
+union dlb2_chp_ldb_cq_wptr {
+ struct {
+ u32 write_pointer : 11;
+ u32 rsvd0 : 21;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ2VAS(x) \
+ (0x40d00000 + (x) * 0x1000)
+#define DLB2_CHP_LDB_CQ2VAS_RST 0x0
+union dlb2_chp_ldb_cq2vas {
+ struct {
+ u32 cq2vas : 5;
+ u32 rsvd0 : 27;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_CFG_CHP_CSR_CTRL 0x44000008
+#define DLB2_CHP_CFG_CHP_CSR_CTRL_RST 0x180002
+union dlb2_chp_cfg_chp_csr_ctrl {
+ struct {
+ u32 int_cor_alarm_dis : 1;
+ u32 int_cor_synd_dis : 1;
+ u32 int_uncr_alarm_dis : 1;
+ u32 int_unc_synd_dis : 1;
+ u32 int_inf0_alarm_dis : 1;
+ u32 int_inf0_synd_dis : 1;
+ u32 int_inf1_alarm_dis : 1;
+ u32 int_inf1_synd_dis : 1;
+ u32 int_inf2_alarm_dis : 1;
+ u32 int_inf2_synd_dis : 1;
+ u32 int_inf3_alarm_dis : 1;
+ u32 int_inf3_synd_dis : 1;
+ u32 int_inf4_alarm_dis : 1;
+ u32 int_inf4_synd_dis : 1;
+ u32 int_inf5_alarm_dis : 1;
+ u32 int_inf5_synd_dis : 1;
+ u32 dlb_cor_alarm_enable : 1;
+ u32 cfg_64bytes_qe_ldb_cq_mode : 1;
+ u32 cfg_64bytes_qe_dir_cq_mode : 1;
+ u32 pad_write_ldb : 1;
+ u32 pad_write_dir : 1;
+ u32 pad_first_write_ldb : 1;
+ u32 pad_first_write_dir : 1;
+ u32 rsvz0 : 9;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INTR_ARMED0 0x4400005c
+#define DLB2_CHP_DIR_CQ_INTR_ARMED0_RST 0x0
+union dlb2_chp_dir_cq_intr_armed0 {
+ struct {
+ u32 armed : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_DIR_CQ_INTR_ARMED1 0x44000060
+#define DLB2_CHP_DIR_CQ_INTR_ARMED1_RST 0x0
+union dlb2_chp_dir_cq_intr_armed1 {
+ struct {
+ u32 armed : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL 0x44000084
+#define DLB2_CHP_CFG_DIR_CQ_TIMER_CTL_RST 0x0
+union dlb2_chp_cfg_dir_cq_timer_ctl {
+ struct {
+ u32 sample_interval : 8;
+ u32 enb : 1;
+ u32 rsvz0 : 23;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WDTO_0 0x44000088
+#define DLB2_CHP_CFG_DIR_WDTO_0_RST 0x0
+union dlb2_chp_cfg_dir_wdto_0 {
+ struct {
+ u32 wdto : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WDTO_1 0x4400008c
+#define DLB2_CHP_CFG_DIR_WDTO_1_RST 0x0
+union dlb2_chp_cfg_dir_wdto_1 {
+ struct {
+ u32 wdto : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_DISABLE0 0x44000098
+#define DLB2_CHP_CFG_DIR_WD_DISABLE0_RST 0xffffffff
+union dlb2_chp_cfg_dir_wd_disable0 {
+ struct {
+ u32 wd_disable : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_DISABLE1 0x4400009c
+#define DLB2_CHP_CFG_DIR_WD_DISABLE1_RST 0xffffffff
+union dlb2_chp_cfg_dir_wd_disable1 {
+ struct {
+ u32 wd_disable : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL 0x440000a0
+#define DLB2_CHP_CFG_DIR_WD_ENB_INTERVAL_RST 0x0
+union dlb2_chp_cfg_dir_wd_enb_interval {
+ struct {
+ u32 sample_interval : 28;
+ u32 enb : 1;
+ u32 rsvz0 : 3;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_CFG_DIR_WD_THRESHOLD 0x440000ac
+#define DLB2_CHP_CFG_DIR_WD_THRESHOLD_RST 0x0
+union dlb2_chp_cfg_dir_wd_threshold {
+ struct {
+ u32 wd_threshold : 8;
+ u32 rsvz0 : 24;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INTR_ARMED0 0x440000b0
+#define DLB2_CHP_LDB_CQ_INTR_ARMED0_RST 0x0
+union dlb2_chp_ldb_cq_intr_armed0 {
+ struct {
+ u32 armed : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_LDB_CQ_INTR_ARMED1 0x440000b4
+#define DLB2_CHP_LDB_CQ_INTR_ARMED1_RST 0x0
+union dlb2_chp_ldb_cq_intr_armed1 {
+ struct {
+ u32 armed : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL 0x440000d8
+#define DLB2_CHP_CFG_LDB_CQ_TIMER_CTL_RST 0x0
+union dlb2_chp_cfg_ldb_cq_timer_ctl {
+ struct {
+ u32 sample_interval : 8;
+ u32 enb : 1;
+ u32 rsvz0 : 23;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WDTO_0 0x440000dc
+#define DLB2_CHP_CFG_LDB_WDTO_0_RST 0x0
+union dlb2_chp_cfg_ldb_wdto_0 {
+ struct {
+ u32 wdto : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WDTO_1 0x440000e0
+#define DLB2_CHP_CFG_LDB_WDTO_1_RST 0x0
+union dlb2_chp_cfg_ldb_wdto_1 {
+ struct {
+ u32 wdto : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_DISABLE0 0x440000ec
+#define DLB2_CHP_CFG_LDB_WD_DISABLE0_RST 0xffffffff
+union dlb2_chp_cfg_ldb_wd_disable0 {
+ struct {
+ u32 wd_disable : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_DISABLE1 0x440000f0
+#define DLB2_CHP_CFG_LDB_WD_DISABLE1_RST 0xffffffff
+union dlb2_chp_cfg_ldb_wd_disable1 {
+ struct {
+ u32 wd_disable : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL 0x440000f4
+#define DLB2_CHP_CFG_LDB_WD_ENB_INTERVAL_RST 0x0
+union dlb2_chp_cfg_ldb_wd_enb_interval {
+ struct {
+ u32 sample_interval : 28;
+ u32 enb : 1;
+ u32 rsvz0 : 3;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_CFG_LDB_WD_THRESHOLD 0x44000100
+#define DLB2_CHP_CFG_LDB_WD_THRESHOLD_RST 0x0
+union dlb2_chp_cfg_ldb_wd_threshold {
+ struct {
+ u32 wd_threshold : 8;
+ u32 rsvz0 : 24;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_SMON_COMPARE0 0x4c000000
+#define DLB2_CHP_SMON_COMPARE0_RST 0x0
+union dlb2_chp_smon_compare0 {
+ struct {
+ u32 compare0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_SMON_COMPARE1 0x4c000004
+#define DLB2_CHP_SMON_COMPARE1_RST 0x0
+union dlb2_chp_smon_compare1 {
+ struct {
+ u32 compare1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_SMON_CFG0 0x4c000008
+#define DLB2_CHP_SMON_CFG0_RST 0x40000000
+union dlb2_chp_smon_cfg0 {
+ struct {
+ u32 smon_enable : 1;
+ u32 smon_0trigger_enable : 1;
+ u32 rsvz0 : 2;
+ u32 smon0_function : 3;
+ u32 smon0_function_compare : 1;
+ u32 smon1_function : 3;
+ u32 smon1_function_compare : 1;
+ u32 smon_mode : 4;
+ u32 stopcounterovfl : 1;
+ u32 intcounterovfl : 1;
+ u32 statcounter0ovfl : 1;
+ u32 statcounter1ovfl : 1;
+ u32 stoptimerovfl : 1;
+ u32 inttimerovfl : 1;
+ u32 stattimerovfl : 1;
+ u32 rsvz1 : 1;
+ u32 timer_prescale : 5;
+ u32 rsvz2 : 1;
+ u32 version : 2;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_SMON_CFG1 0x4c00000c
+#define DLB2_CHP_SMON_CFG1_RST 0x0
+union dlb2_chp_smon_cfg1 {
+ struct {
+ u32 mode0 : 8;
+ u32 mode1 : 8;
+ u32 rsvz0 : 16;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_SMON_ACTIVITYCNTR0 0x4c000010
+#define DLB2_CHP_SMON_ACTIVITYCNTR0_RST 0x0
+union dlb2_chp_smon_activitycntr0 {
+ struct {
+ u32 counter0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_SMON_ACTIVITYCNTR1 0x4c000014
+#define DLB2_CHP_SMON_ACTIVITYCNTR1_RST 0x0
+union dlb2_chp_smon_activitycntr1 {
+ struct {
+ u32 counter1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_SMON_MAX_TMR 0x4c000018
+#define DLB2_CHP_SMON_MAX_TMR_RST 0x0
+union dlb2_chp_smon_max_tmr {
+ struct {
+ u32 maxvalue : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_SMON_TMR 0x4c00001c
+#define DLB2_CHP_SMON_TMR_RST 0x0
+union dlb2_chp_smon_tmr {
+ struct {
+ u32 timer : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CHP_CTRL_DIAG_02 0x4c000028
+#define DLB2_CHP_CTRL_DIAG_02_RST 0x1555
+union dlb2_chp_ctrl_diag_02 {
+ struct {
+ u32 egress_credit_status_empty : 1;
+ u32 egress_credit_status_afull : 1;
+ u32 chp_outbound_hcw_pipe_credit_status_empty : 1;
+ u32 chp_outbound_hcw_pipe_credit_status_afull : 1;
+ u32 chp_lsp_ap_cmp_pipe_credit_status_empty : 1;
+ u32 chp_lsp_ap_cmp_pipe_credit_status_afull : 1;
+ u32 chp_lsp_tok_pipe_credit_status_empty : 1;
+ u32 chp_lsp_tok_pipe_credit_status_afull : 1;
+ u32 chp_rop_pipe_credit_status_empty : 1;
+ u32 chp_rop_pipe_credit_status_afull : 1;
+ u32 qed_to_cq_pipe_credit_status_empty : 1;
+ u32 qed_to_cq_pipe_credit_status_afull : 1;
+ u32 egress_lsp_token_credit_status_empty : 1;
+ u32 egress_lsp_token_credit_status_afull : 1;
+ u32 rsvd0 : 18;
+ } field;
+ u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0 0x54000000
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_0_RST 0xfefcfaf8
+union dlb2_dp_cfg_arb_weights_tqpri_dir_0 {
+ struct {
+ u32 pri0 : 8;
+ u32 pri1 : 8;
+ u32 pri2 : 8;
+ u32 pri3 : 8;
+ } field;
+ u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1 0x54000004
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_DIR_1_RST 0x0
+union dlb2_dp_cfg_arb_weights_tqpri_dir_1 {
+ struct {
+ u32 rsvz0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x54000008
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
+union dlb2_dp_cfg_arb_weights_tqpri_replay_0 {
+ struct {
+ u32 pri0 : 8;
+ u32 pri1 : 8;
+ u32 pri2 : 8;
+ u32 pri3 : 8;
+ } field;
+ u32 val;
+};
+
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x5400000c
+#define DLB2_DP_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
+union dlb2_dp_cfg_arb_weights_tqpri_replay_1 {
+ struct {
+ u32 rsvz0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_DP_DIR_CSR_CTRL 0x54000010
+#define DLB2_DP_DIR_CSR_CTRL_RST 0x0
+union dlb2_dp_dir_csr_ctrl {
+ struct {
+ u32 int_cor_alarm_dis : 1;
+ u32 int_cor_synd_dis : 1;
+ u32 int_uncr_alarm_dis : 1;
+ u32 int_unc_synd_dis : 1;
+ u32 int_inf0_alarm_dis : 1;
+ u32 int_inf0_synd_dis : 1;
+ u32 int_inf1_alarm_dis : 1;
+ u32 int_inf1_synd_dis : 1;
+ u32 int_inf2_alarm_dis : 1;
+ u32 int_inf2_synd_dis : 1;
+ u32 int_inf3_alarm_dis : 1;
+ u32 int_inf3_synd_dis : 1;
+ u32 int_inf4_alarm_dis : 1;
+ u32 int_inf4_synd_dis : 1;
+ u32 int_inf5_alarm_dis : 1;
+ u32 int_inf5_synd_dis : 1;
+ u32 rsvz0 : 16;
+ } field;
+ u32 val;
+};
+
+#define DLB2_DP_SMON_ACTIVITYCNTR0 0x5c000058
+#define DLB2_DP_SMON_ACTIVITYCNTR0_RST 0x0
+union dlb2_dp_smon_activitycntr0 {
+ struct {
+ u32 counter0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_DP_SMON_ACTIVITYCNTR1 0x5c00005c
+#define DLB2_DP_SMON_ACTIVITYCNTR1_RST 0x0
+union dlb2_dp_smon_activitycntr1 {
+ struct {
+ u32 counter1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_DP_SMON_COMPARE0 0x5c000060
+#define DLB2_DP_SMON_COMPARE0_RST 0x0
+union dlb2_dp_smon_compare0 {
+ struct {
+ u32 compare0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_DP_SMON_COMPARE1 0x5c000064
+#define DLB2_DP_SMON_COMPARE1_RST 0x0
+union dlb2_dp_smon_compare1 {
+ struct {
+ u32 compare1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_DP_SMON_CFG0 0x5c000068
+#define DLB2_DP_SMON_CFG0_RST 0x40000000
+union dlb2_dp_smon_cfg0 {
+ struct {
+ u32 smon_enable : 1;
+ u32 smon_0trigger_enable : 1;
+ u32 rsvz0 : 2;
+ u32 smon0_function : 3;
+ u32 smon0_function_compare : 1;
+ u32 smon1_function : 3;
+ u32 smon1_function_compare : 1;
+ u32 smon_mode : 4;
+ u32 stopcounterovfl : 1;
+ u32 intcounterovfl : 1;
+ u32 statcounter0ovfl : 1;
+ u32 statcounter1ovfl : 1;
+ u32 stoptimerovfl : 1;
+ u32 inttimerovfl : 1;
+ u32 stattimerovfl : 1;
+ u32 rsvz1 : 1;
+ u32 timer_prescale : 5;
+ u32 rsvz2 : 1;
+ u32 version : 2;
+ } field;
+ u32 val;
+};
+
+#define DLB2_DP_SMON_CFG1 0x5c00006c
+#define DLB2_DP_SMON_CFG1_RST 0x0
+union dlb2_dp_smon_cfg1 {
+ struct {
+ u32 mode0 : 8;
+ u32 mode1 : 8;
+ u32 rsvz0 : 16;
+ } field;
+ u32 val;
+};
+
+#define DLB2_DP_SMON_MAX_TMR 0x5c000070
+#define DLB2_DP_SMON_MAX_TMR_RST 0x0
+union dlb2_dp_smon_max_tmr {
+ struct {
+ u32 maxvalue : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_DP_SMON_TMR 0x5c000074
+#define DLB2_DP_SMON_TMR_RST 0x0
+union dlb2_dp_smon_tmr {
+ struct {
+ u32 timer : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_DQED_PIPE_SMON_ACTIVITYCNTR0 0x6c000024
+#define DLB2_DQED_PIPE_SMON_ACTIVITYCNTR0_RST 0x0
+union dlb2_dqed_pipe_smon_activitycntr0 {
+ struct {
+ u32 counter0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_DQED_PIPE_SMON_ACTIVITYCNTR1 0x6c000028
+#define DLB2_DQED_PIPE_SMON_ACTIVITYCNTR1_RST 0x0
+union dlb2_dqed_pipe_smon_activitycntr1 {
+ struct {
+ u32 counter1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_DQED_PIPE_SMON_COMPARE0 0x6c00002c
+#define DLB2_DQED_PIPE_SMON_COMPARE0_RST 0x0
+union dlb2_dqed_pipe_smon_compare0 {
+ struct {
+ u32 compare0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_DQED_PIPE_SMON_COMPARE1 0x6c000030
+#define DLB2_DQED_PIPE_SMON_COMPARE1_RST 0x0
+union dlb2_dqed_pipe_smon_compare1 {
+ struct {
+ u32 compare1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_DQED_PIPE_SMON_CFG0 0x6c000034
+#define DLB2_DQED_PIPE_SMON_CFG0_RST 0x40000000
+union dlb2_dqed_pipe_smon_cfg0 {
+ struct {
+ u32 smon_enable : 1;
+ u32 smon_0trigger_enable : 1;
+ u32 rsvz0 : 2;
+ u32 smon0_function : 3;
+ u32 smon0_function_compare : 1;
+ u32 smon1_function : 3;
+ u32 smon1_function_compare : 1;
+ u32 smon_mode : 4;
+ u32 stopcounterovfl : 1;
+ u32 intcounterovfl : 1;
+ u32 statcounter0ovfl : 1;
+ u32 statcounter1ovfl : 1;
+ u32 stoptimerovfl : 1;
+ u32 inttimerovfl : 1;
+ u32 stattimerovfl : 1;
+ u32 rsvz1 : 1;
+ u32 timer_prescale : 5;
+ u32 rsvz2 : 1;
+ u32 version : 2;
+ } field;
+ u32 val;
+};
+
+#define DLB2_DQED_PIPE_SMON_CFG1 0x6c000038
+#define DLB2_DQED_PIPE_SMON_CFG1_RST 0x0
+union dlb2_dqed_pipe_smon_cfg1 {
+ struct {
+ u32 mode0 : 8;
+ u32 mode1 : 8;
+ u32 rsvz0 : 16;
+ } field;
+ u32 val;
+};
+
+#define DLB2_DQED_PIPE_SMON_MAX_TMR 0x6c00003c
+#define DLB2_DQED_PIPE_SMON_MAX_TMR_RST 0x0
+union dlb2_dqed_pipe_smon_max_tmr {
+ struct {
+ u32 maxvalue : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_DQED_PIPE_SMON_TMR 0x6c000040
+#define DLB2_DQED_PIPE_SMON_TMR_RST 0x0
+union dlb2_dqed_pipe_smon_tmr {
+ struct {
+ u32 timer : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_QED_PIPE_SMON_ACTIVITYCNTR0 0x7c000024
+#define DLB2_QED_PIPE_SMON_ACTIVITYCNTR0_RST 0x0
+union dlb2_qed_pipe_smon_activitycntr0 {
+ struct {
+ u32 counter0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_QED_PIPE_SMON_ACTIVITYCNTR1 0x7c000028
+#define DLB2_QED_PIPE_SMON_ACTIVITYCNTR1_RST 0x0
+union dlb2_qed_pipe_smon_activitycntr1 {
+ struct {
+ u32 counter1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_QED_PIPE_SMON_COMPARE0 0x7c00002c
+#define DLB2_QED_PIPE_SMON_COMPARE0_RST 0x0
+union dlb2_qed_pipe_smon_compare0 {
+ struct {
+ u32 compare0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_QED_PIPE_SMON_COMPARE1 0x7c000030
+#define DLB2_QED_PIPE_SMON_COMPARE1_RST 0x0
+union dlb2_qed_pipe_smon_compare1 {
+ struct {
+ u32 compare1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_QED_PIPE_SMON_CFG0 0x7c000034
+#define DLB2_QED_PIPE_SMON_CFG0_RST 0x40000000
+union dlb2_qed_pipe_smon_cfg0 {
+ struct {
+ u32 smon_enable : 1;
+ u32 smon_0trigger_enable : 1;
+ u32 rsvz0 : 2;
+ u32 smon0_function : 3;
+ u32 smon0_function_compare : 1;
+ u32 smon1_function : 3;
+ u32 smon1_function_compare : 1;
+ u32 smon_mode : 4;
+ u32 stopcounterovfl : 1;
+ u32 intcounterovfl : 1;
+ u32 statcounter0ovfl : 1;
+ u32 statcounter1ovfl : 1;
+ u32 stoptimerovfl : 1;
+ u32 inttimerovfl : 1;
+ u32 stattimerovfl : 1;
+ u32 rsvz1 : 1;
+ u32 timer_prescale : 5;
+ u32 rsvz2 : 1;
+ u32 version : 2;
+ } field;
+ u32 val;
+};
+
+#define DLB2_QED_PIPE_SMON_CFG1 0x7c000038
+#define DLB2_QED_PIPE_SMON_CFG1_RST 0x0
+union dlb2_qed_pipe_smon_cfg1 {
+ struct {
+ u32 mode0 : 8;
+ u32 mode1 : 8;
+ u32 rsvz0 : 16;
+ } field;
+ u32 val;
+};
+
+#define DLB2_QED_PIPE_SMON_MAX_TMR 0x7c00003c
+#define DLB2_QED_PIPE_SMON_MAX_TMR_RST 0x0
+union dlb2_qed_pipe_smon_max_tmr {
+ struct {
+ u32 maxvalue : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_QED_PIPE_SMON_TMR 0x7c000040
+#define DLB2_QED_PIPE_SMON_TMR_RST 0x0
+union dlb2_qed_pipe_smon_tmr {
+ struct {
+ u32 timer : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0 0x84000000
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_0 {
+ struct {
+ u32 pri0 : 8;
+ u32 pri1 : 8;
+ u32 pri2 : 8;
+ u32 pri3 : 8;
+ } field;
+ u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1 0x84000004
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_ATQ_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_atq_1 {
+ struct {
+ u32 rsvz0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0 0x84000008
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_0 {
+ struct {
+ u32 pri0 : 8;
+ u32 pri1 : 8;
+ u32 pri2 : 8;
+ u32 pri3 : 8;
+ } field;
+ u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1 0x8400000c
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_NALB_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_nalb_1 {
+ struct {
+ u32 rsvz0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0 0x84000010
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_0_RST 0xfefcfaf8
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_0 {
+ struct {
+ u32 pri0 : 8;
+ u32 pri1 : 8;
+ u32 pri2 : 8;
+ u32 pri3 : 8;
+ } field;
+ u32 val;
+};
+
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1 0x84000014
+#define DLB2_NALB_PIPE_CFG_ARB_WEIGHTS_TQPRI_REPLAY_1_RST 0x0
+union dlb2_nalb_pipe_cfg_arb_weights_tqpri_replay_1 {
+ struct {
+ u32 rsvz0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_NALB_PIPE_SMON_ACTIVITYCNTR0 0x8c000064
+#define DLB2_NALB_PIPE_SMON_ACTIVITYCNTR0_RST 0x0
+union dlb2_nalb_pipe_smon_activitycntr0 {
+ struct {
+ u32 counter0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_NALB_PIPE_SMON_ACTIVITYCNTR1 0x8c000068
+#define DLB2_NALB_PIPE_SMON_ACTIVITYCNTR1_RST 0x0
+union dlb2_nalb_pipe_smon_activitycntr1 {
+ struct {
+ u32 counter1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_NALB_PIPE_SMON_COMPARE0 0x8c00006c
+#define DLB2_NALB_PIPE_SMON_COMPARE0_RST 0x0
+union dlb2_nalb_pipe_smon_compare0 {
+ struct {
+ u32 compare0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_NALB_PIPE_SMON_COMPARE1 0x8c000070
+#define DLB2_NALB_PIPE_SMON_COMPARE1_RST 0x0
+union dlb2_nalb_pipe_smon_compare1 {
+ struct {
+ u32 compare1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_NALB_PIPE_SMON_CFG0 0x8c000074
+#define DLB2_NALB_PIPE_SMON_CFG0_RST 0x40000000
+union dlb2_nalb_pipe_smon_cfg0 {
+ struct {
+ u32 smon_enable : 1;
+ u32 smon_0trigger_enable : 1;
+ u32 rsvz0 : 2;
+ u32 smon0_function : 3;
+ u32 smon0_function_compare : 1;
+ u32 smon1_function : 3;
+ u32 smon1_function_compare : 1;
+ u32 smon_mode : 4;
+ u32 stopcounterovfl : 1;
+ u32 intcounterovfl : 1;
+ u32 statcounter0ovfl : 1;
+ u32 statcounter1ovfl : 1;
+ u32 stoptimerovfl : 1;
+ u32 inttimerovfl : 1;
+ u32 stattimerovfl : 1;
+ u32 rsvz1 : 1;
+ u32 timer_prescale : 5;
+ u32 rsvz2 : 1;
+ u32 version : 2;
+ } field;
+ u32 val;
+};
+
+#define DLB2_NALB_PIPE_SMON_CFG1 0x8c000078
+#define DLB2_NALB_PIPE_SMON_CFG1_RST 0x0
+union dlb2_nalb_pipe_smon_cfg1 {
+ struct {
+ u32 mode0 : 8;
+ u32 mode1 : 8;
+ u32 rsvz0 : 16;
+ } field;
+ u32 val;
+};
+
+#define DLB2_NALB_PIPE_SMON_MAX_TMR 0x8c00007c
+#define DLB2_NALB_PIPE_SMON_MAX_TMR_RST 0x0
+union dlb2_nalb_pipe_smon_max_tmr {
+ struct {
+ u32 maxvalue : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_NALB_PIPE_SMON_TMR 0x8c000080
+#define DLB2_NALB_PIPE_SMON_TMR_RST 0x0
+union dlb2_nalb_pipe_smon_tmr {
+ struct {
+ u32 timer : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_0_SLT_SHFT(x) \
+ (0x96000000 + (x) * 0x4)
+#define DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST 0x0
+union dlb2_ro_pipe_grp_0_slt_shft {
+ struct {
+ u32 change : 10;
+ u32 rsvd0 : 22;
+ } field;
+ u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_1_SLT_SHFT(x) \
+ (0x96010000 + (x) * 0x4)
+#define DLB2_RO_PIPE_GRP_1_SLT_SHFT_RST 0x0
+union dlb2_ro_pipe_grp_1_slt_shft {
+ struct {
+ u32 change : 10;
+ u32 rsvd0 : 22;
+ } field;
+ u32 val;
+};
+
+#define DLB2_RO_PIPE_GRP_SN_MODE 0x94000000
+#define DLB2_RO_PIPE_GRP_SN_MODE_RST 0x0
+union dlb2_ro_pipe_grp_sn_mode {
+ struct {
+ u32 sn_mode_0 : 3;
+ u32 rszv0 : 5;
+ u32 sn_mode_1 : 3;
+ u32 rszv1 : 21;
+ } field;
+ u32 val;
+};
+
+#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0 0x9c000000
+#define DLB2_RO_PIPE_CFG_CTRL_GENERAL_0_RST 0x0
+union dlb2_ro_pipe_cfg_ctrl_general_0 {
+ struct {
+ u32 unit_single_step_mode : 1;
+ u32 rr_en : 1;
+ u32 rszv0 : 30;
+ } field;
+ u32 val;
+};
+
+#define DLB2_RO_PIPE_SMON_ACTIVITYCNTR0 0x9c000030
+#define DLB2_RO_PIPE_SMON_ACTIVITYCNTR0_RST 0x0
+union dlb2_ro_pipe_smon_activitycntr0 {
+ struct {
+ u32 counter0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_RO_PIPE_SMON_ACTIVITYCNTR1 0x9c000034
+#define DLB2_RO_PIPE_SMON_ACTIVITYCNTR1_RST 0x0
+union dlb2_ro_pipe_smon_activitycntr1 {
+ struct {
+ u32 counter1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_RO_PIPE_SMON_COMPARE0 0x9c000038
+#define DLB2_RO_PIPE_SMON_COMPARE0_RST 0x0
+union dlb2_ro_pipe_smon_compare0 {
+ struct {
+ u32 compare0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_RO_PIPE_SMON_COMPARE1 0x9c00003c
+#define DLB2_RO_PIPE_SMON_COMPARE1_RST 0x0
+union dlb2_ro_pipe_smon_compare1 {
+ struct {
+ u32 compare1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_RO_PIPE_SMON_CFG0 0x9c000040
+#define DLB2_RO_PIPE_SMON_CFG0_RST 0x40000000
+union dlb2_ro_pipe_smon_cfg0 {
+ struct {
+ u32 smon_enable : 1;
+ u32 smon_0trigger_enable : 1;
+ u32 rsvz0 : 2;
+ u32 smon0_function : 3;
+ u32 smon0_function_compare : 1;
+ u32 smon1_function : 3;
+ u32 smon1_function_compare : 1;
+ u32 smon_mode : 4;
+ u32 stopcounterovfl : 1;
+ u32 intcounterovfl : 1;
+ u32 statcounter0ovfl : 1;
+ u32 statcounter1ovfl : 1;
+ u32 stoptimerovfl : 1;
+ u32 inttimerovfl : 1;
+ u32 stattimerovfl : 1;
+ u32 rsvz1 : 1;
+ u32 timer_prescale : 5;
+ u32 rsvz2 : 1;
+ u32 version : 2;
+ } field;
+ u32 val;
+};
+
+#define DLB2_RO_PIPE_SMON_CFG1 0x9c000044
+#define DLB2_RO_PIPE_SMON_CFG1_RST 0x0
+union dlb2_ro_pipe_smon_cfg1 {
+ struct {
+ u32 mode0 : 8;
+ u32 mode1 : 8;
+ u32 rsvz0 : 16;
+ } field;
+ u32 val;
+};
+
+#define DLB2_RO_PIPE_SMON_MAX_TMR 0x9c000048
+#define DLB2_RO_PIPE_SMON_MAX_TMR_RST 0x0
+union dlb2_ro_pipe_smon_max_tmr {
+ struct {
+ u32 maxvalue : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_RO_PIPE_SMON_TMR 0x9c00004c
+#define DLB2_RO_PIPE_SMON_TMR_RST 0x0
+union dlb2_ro_pipe_smon_tmr {
+ struct {
+ u32 timer : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_CQ2PRIOV(x) \
+ (0xa0000000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2PRIOV_RST 0x0
+union dlb2_lsp_cq2priov {
+ struct {
+ u32 prio : 24;
+ u32 v : 8;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_CQ2QID0(x) \
+ (0xa0080000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2QID0_RST 0x0
+union dlb2_lsp_cq2qid0 {
+ struct {
+ u32 qid_p0 : 7;
+ u32 rsvd3 : 1;
+ u32 qid_p1 : 7;
+ u32 rsvd2 : 1;
+ u32 qid_p2 : 7;
+ u32 rsvd1 : 1;
+ u32 qid_p3 : 7;
+ u32 rsvd0 : 1;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_CQ2QID1(x) \
+ (0xa0100000 + (x) * 0x1000)
+#define DLB2_LSP_CQ2QID1_RST 0x0
+union dlb2_lsp_cq2qid1 {
+ struct {
+ u32 qid_p4 : 7;
+ u32 rsvd3 : 1;
+ u32 qid_p5 : 7;
+ u32 rsvd2 : 1;
+ u32 qid_p6 : 7;
+ u32 rsvd1 : 1;
+ u32 qid_p7 : 7;
+ u32 rsvd0 : 1;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_DSBL(x) \
+ (0xa0180000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_DSBL_RST 0x1
+union dlb2_lsp_cq_dir_dsbl {
+ struct {
+ u32 disabled : 1;
+ u32 rsvd0 : 31;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TKN_CNT(x) \
+ (0xa0200000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TKN_CNT_RST 0x0
+union dlb2_lsp_cq_dir_tkn_cnt {
+ struct {
+ u32 count : 13;
+ u32 rsvd0 : 19;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(x) \
+ (0xa0280000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST 0x0
+union dlb2_lsp_cq_dir_tkn_depth_sel_dsi {
+ struct {
+ u32 token_depth_select : 4;
+ u32 disable_wb_opt : 1;
+ u32 ignore_depth : 1;
+ u32 rsvd0 : 26;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(x) \
+ (0xa0300000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST 0x0
+union dlb2_lsp_cq_dir_tot_sch_cntl {
+ struct {
+ u32 count : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(x) \
+ (0xa0380000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST 0x0
+union dlb2_lsp_cq_dir_tot_sch_cnth {
+ struct {
+ u32 count : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_DSBL(x) \
+ (0xa0400000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_DSBL_RST 0x1
+union dlb2_lsp_cq_ldb_dsbl {
+ struct {
+ u32 disabled : 1;
+ u32 rsvd0 : 31;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_INFL_CNT(x) \
+ (0xa0480000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_INFL_CNT_RST 0x0
+union dlb2_lsp_cq_ldb_infl_cnt {
+ struct {
+ u32 count : 12;
+ u32 rsvd0 : 20;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_INFL_LIM(x) \
+ (0xa0500000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_INFL_LIM_RST 0x0
+union dlb2_lsp_cq_ldb_infl_lim {
+ struct {
+ u32 limit : 12;
+ u32 rsvd0 : 20;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TKN_CNT(x) \
+ (0xa0580000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TKN_CNT_RST 0x0
+union dlb2_lsp_cq_ldb_tkn_cnt {
+ struct {
+ u32 token_count : 11;
+ u32 rsvd0 : 21;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(x) \
+ (0xa0600000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST 0x0
+union dlb2_lsp_cq_ldb_tkn_depth_sel {
+ struct {
+ u32 token_depth_select : 4;
+ u32 ignore_depth : 1;
+ u32 rsvd0 : 27;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(x) \
+ (0xa0680000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST 0x0
+union dlb2_lsp_cq_ldb_tot_sch_cntl {
+ struct {
+ u32 count : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(x) \
+ (0xa0700000 + (x) * 0x1000)
+#define DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST 0x0
+union dlb2_lsp_cq_ldb_tot_sch_cnth {
+ struct {
+ u32 count : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_MAX_DEPTH(x) \
+ (0xa0780000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_MAX_DEPTH_RST 0x0
+union dlb2_lsp_qid_dir_max_depth {
+ struct {
+ u32 depth : 13;
+ u32 rsvd0 : 19;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(x) \
+ (0xa0800000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_dir_tot_enq_cntl {
+ struct {
+ u32 count : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(x) \
+ (0xa0880000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_dir_tot_enq_cnth {
+ struct {
+ u32 count : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_ENQUEUE_CNT(x) \
+ (0xa0900000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_dir_enqueue_cnt {
+ struct {
+ u32 count : 13;
+ u32 rsvd0 : 19;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_QID_DIR_DEPTH_THRSH(x) \
+ (0xa0980000 + (x) * 0x1000)
+#define DLB2_LSP_QID_DIR_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_dir_depth_thrsh {
+ struct {
+ u32 thresh : 13;
+ u32 rsvd0 : 19;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_QID_AQED_ACTIVE_CNT(x) \
+ (0xa0a00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_AQED_ACTIVE_CNT_RST 0x0
+union dlb2_lsp_qid_aqed_active_cnt {
+ struct {
+ u32 count : 12;
+ u32 rsvd0 : 20;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_QID_AQED_ACTIVE_LIM(x) \
+ (0xa0a80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_AQED_ACTIVE_LIM_RST 0x0
+union dlb2_lsp_qid_aqed_active_lim {
+ struct {
+ u32 limit : 12;
+ u32 rsvd0 : 20;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(x) \
+ (0xa0b00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_atm_tot_enq_cntl {
+ struct {
+ u32 count : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(x) \
+ (0xa0b80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_atm_tot_enq_cnth {
+ struct {
+ u32 count : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT(x) \
+ (0xa0c00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATQ_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_atq_enqueue_cnt {
+ struct {
+ u32 count : 14;
+ u32 rsvd0 : 18;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_ENQUEUE_CNT(x) \
+ (0xa0c80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_ENQUEUE_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_enqueue_cnt {
+ struct {
+ u32 count : 14;
+ u32 rsvd0 : 18;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_INFL_CNT(x) \
+ (0xa0d00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_INFL_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_infl_cnt {
+ struct {
+ u32 count : 12;
+ u32 rsvd0 : 20;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_INFL_LIM(x) \
+ (0xa0d80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_INFL_LIM_RST 0x0
+union dlb2_lsp_qid_ldb_infl_lim {
+ struct {
+ u32 limit : 12;
+ u32 rsvd0 : 20;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_QID2CQIDIX_00(x) \
+ (0xa0e00000 + (x) * 0x1000)
+#define DLB2_LSP_QID2CQIDIX_00_RST 0x0
+#define DLB2_LSP_QID2CQIDIX(x, y) \
+ (DLB2_LSP_QID2CQIDIX_00(x) + 0x80000 * (y))
+#define DLB2_LSP_QID2CQIDIX_NUM 16
+union dlb2_lsp_qid2cqidix_00 {
+ struct {
+ u32 cq_p0 : 8;
+ u32 cq_p1 : 8;
+ u32 cq_p2 : 8;
+ u32 cq_p3 : 8;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_QID2CQIDIX2_00(x) \
+ (0xa1600000 + (x) * 0x1000)
+#define DLB2_LSP_QID2CQIDIX2_00_RST 0x0
+#define DLB2_LSP_QID2CQIDIX2(x, y) \
+ (DLB2_LSP_QID2CQIDIX2_00(x) + 0x80000 * (y))
+#define DLB2_LSP_QID2CQIDIX2_NUM 16
+union dlb2_lsp_qid2cqidix2_00 {
+ struct {
+ u32 cq_p0 : 8;
+ u32 cq_p1 : 8;
+ u32 cq_p2 : 8;
+ u32 cq_p3 : 8;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_QID_LDB_REPLAY_CNT(x) \
+ (0xa1e00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_LDB_REPLAY_CNT_RST 0x0
+union dlb2_lsp_qid_ldb_replay_cnt {
+ struct {
+ u32 count : 14;
+ u32 rsvd0 : 18;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_MAX_DEPTH(x) \
+ (0xa1f00000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_MAX_DEPTH_RST 0x0
+union dlb2_lsp_qid_naldb_max_depth {
+ struct {
+ u32 depth : 14;
+ u32 rsvd0 : 18;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(x) \
+ (0xa1f80000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST 0x0
+union dlb2_lsp_qid_naldb_tot_enq_cntl {
+ struct {
+ u32 count : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(x) \
+ (0xa2000000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST 0x0
+union dlb2_lsp_qid_naldb_tot_enq_cnth {
+ struct {
+ u32 count : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_DEPTH_THRSH(x) \
+ (0xa2080000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_atm_depth_thrsh {
+ struct {
+ u32 thresh : 14;
+ u32 rsvd0 : 18;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_QID_NALDB_DEPTH_THRSH(x) \
+ (0xa2100000 + (x) * 0x1000)
+#define DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST 0x0
+union dlb2_lsp_qid_naldb_depth_thrsh {
+ struct {
+ u32 thresh : 14;
+ u32 rsvd0 : 18;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_QID_ATM_ACTIVE(x) \
+ (0xa2180000 + (x) * 0x1000)
+#define DLB2_LSP_QID_ATM_ACTIVE_RST 0x0
+union dlb2_lsp_qid_atm_active {
+ struct {
+ u32 count : 14;
+ u32 rsvd0 : 18;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0 0xa4000008
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_0_RST 0x0
+union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_0 {
+ struct {
+ u32 pri0_weight : 8;
+ u32 pri1_weight : 8;
+ u32 pri2_weight : 8;
+ u32 pri3_weight : 8;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1 0xa400000c
+#define DLB2_LSP_CFG_ARB_WEIGHT_ATM_NALB_QID_1_RST 0x0
+union dlb2_lsp_cfg_arb_weight_atm_nalb_qid_1 {
+ struct {
+ u32 rsvz0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0 0xa4000014
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_0_RST 0x0
+union dlb2_lsp_cfg_arb_weight_ldb_qid_0 {
+ struct {
+ u32 pri0_weight : 8;
+ u32 pri1_weight : 8;
+ u32 pri2_weight : 8;
+ u32 pri3_weight : 8;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1 0xa4000018
+#define DLB2_LSP_CFG_ARB_WEIGHT_LDB_QID_1_RST 0x0
+union dlb2_lsp_cfg_arb_weight_ldb_qid_1 {
+ struct {
+ u32 rsvz0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_LDB_SCHED_CTRL 0xa400002c
+#define DLB2_LSP_LDB_SCHED_CTRL_RST 0x0
+union dlb2_lsp_ldb_sched_ctrl {
+ struct {
+ u32 cq : 8;
+ u32 qidix : 3;
+ u32 value : 1;
+ u32 nalb_haswork_v : 1;
+ u32 rlist_haswork_v : 1;
+ u32 slist_haswork_v : 1;
+ u32 inflight_ok_v : 1;
+ u32 aqed_nfull_v : 1;
+ u32 rsvz0 : 15;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_DIR_SCH_CNT_L 0xa4000034
+#define DLB2_LSP_DIR_SCH_CNT_L_RST 0x0
+union dlb2_lsp_dir_sch_cnt_l {
+ struct {
+ u32 count : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_DIR_SCH_CNT_H 0xa4000038
+#define DLB2_LSP_DIR_SCH_CNT_H_RST 0x0
+union dlb2_lsp_dir_sch_cnt_h {
+ struct {
+ u32 count : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_LDB_SCH_CNT_L 0xa400003c
+#define DLB2_LSP_LDB_SCH_CNT_L_RST 0x0
+union dlb2_lsp_ldb_sch_cnt_l {
+ struct {
+ u32 count : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_LDB_SCH_CNT_H 0xa4000040
+#define DLB2_LSP_LDB_SCH_CNT_H_RST 0x0
+union dlb2_lsp_ldb_sch_cnt_h {
+ struct {
+ u32 count : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_CFG_SHDW_CTRL 0xa4000070
+#define DLB2_LSP_CFG_SHDW_CTRL_RST 0x0
+union dlb2_lsp_cfg_shdw_ctrl {
+ struct {
+ u32 transfer : 1;
+ u32 rsvd0 : 31;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_CFG_SHDW_RANGE_COS(x) \
+ (0xa4000074 + (x) * 4)
+#define DLB2_LSP_CFG_SHDW_RANGE_COS_RST 0x40
+union dlb2_lsp_cfg_shdw_range_cos {
+ struct {
+ u32 bw_range : 9;
+ u32 rsvz0 : 22;
+ u32 no_extra_credit : 1;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_CFG_CTRL_GENERAL_0 0xac000000
+#define DLB2_LSP_CFG_CTRL_GENERAL_0_RST 0x0
+union dlb2_lsp_cfg_ctrl_general_0 {
+ struct {
+ u32 disab_atq_empty_arb : 1;
+ u32 inc_tok_unit_idle : 1;
+ u32 disab_rlist_pri : 1;
+ u32 inc_cmp_unit_idle : 1;
+ u32 rsvz0 : 2;
+ u32 dir_single_op : 1;
+ u32 dir_half_bw : 1;
+ u32 dir_single_out : 1;
+ u32 dir_disab_multi : 1;
+ u32 atq_single_op : 1;
+ u32 atq_half_bw : 1;
+ u32 atq_single_out : 1;
+ u32 atq_disab_multi : 1;
+ u32 dirrpl_single_op : 1;
+ u32 dirrpl_half_bw : 1;
+ u32 dirrpl_single_out : 1;
+ u32 lbrpl_single_op : 1;
+ u32 lbrpl_half_bw : 1;
+ u32 lbrpl_single_out : 1;
+ u32 ldb_single_op : 1;
+ u32 ldb_half_bw : 1;
+ u32 ldb_disab_multi : 1;
+ u32 atm_single_sch : 1;
+ u32 atm_single_cmp : 1;
+ u32 ldb_ce_tog_arb : 1;
+ u32 rsvz1 : 1;
+ u32 smon0_valid_sel : 2;
+ u32 smon0_value_sel : 1;
+ u32 smon0_compare_sel : 2;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_SMON_COMPARE0 0xac000048
+#define DLB2_LSP_SMON_COMPARE0_RST 0x0
+union dlb2_lsp_smon_compare0 {
+ struct {
+ u32 compare0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_SMON_COMPARE1 0xac00004c
+#define DLB2_LSP_SMON_COMPARE1_RST 0x0
+union dlb2_lsp_smon_compare1 {
+ struct {
+ u32 compare1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_SMON_CFG0 0xac000050
+#define DLB2_LSP_SMON_CFG0_RST 0x40000000
+union dlb2_lsp_smon_cfg0 {
+ struct {
+ u32 smon_enable : 1;
+ u32 smon_0trigger_enable : 1;
+ u32 rsvz0 : 2;
+ u32 smon0_function : 3;
+ u32 smon0_function_compare : 1;
+ u32 smon1_function : 3;
+ u32 smon1_function_compare : 1;
+ u32 smon_mode : 4;
+ u32 stopcounterovfl : 1;
+ u32 intcounterovfl : 1;
+ u32 statcounter0ovfl : 1;
+ u32 statcounter1ovfl : 1;
+ u32 stoptimerovfl : 1;
+ u32 inttimerovfl : 1;
+ u32 stattimerovfl : 1;
+ u32 rsvz1 : 1;
+ u32 timer_prescale : 5;
+ u32 rsvz2 : 1;
+ u32 version : 2;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_SMON_CFG1 0xac000054
+#define DLB2_LSP_SMON_CFG1_RST 0x0
+union dlb2_lsp_smon_cfg1 {
+ struct {
+ u32 mode0 : 8;
+ u32 mode1 : 8;
+ u32 rsvz0 : 16;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_SMON_ACTIVITYCNTR0 0xac000058
+#define DLB2_LSP_SMON_ACTIVITYCNTR0_RST 0x0
+union dlb2_lsp_smon_activitycntr0 {
+ struct {
+ u32 counter0 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_SMON_ACTIVITYCNTR1 0xac00005c
+#define DLB2_LSP_SMON_ACTIVITYCNTR1_RST 0x0
+union dlb2_lsp_smon_activitycntr1 {
+ struct {
+ u32 counter1 : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_SMON_MAX_TMR 0xac000060
+#define DLB2_LSP_SMON_MAX_TMR_RST 0x0
+union dlb2_lsp_smon_max_tmr {
+ struct {
+ u32 maxvalue : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_LSP_SMON_TMR 0xac000064
+#define DLB2_LSP_SMON_TMR_RST 0x0
+union dlb2_lsp_smon_tmr {
+ struct {
+ u32 timer : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CFG_MSTR_DIAG_RESET_STS 0xb4000000
+#define DLB2_CFG_MSTR_DIAG_RESET_STS_RST 0x80000bff
+union dlb2_cfg_mstr_diag_reset_sts {
+ struct {
+ u32 chp_pf_reset_done : 1;
+ u32 rop_pf_reset_done : 1;
+ u32 lsp_pf_reset_done : 1;
+ u32 nalb_pf_reset_done : 1;
+ u32 ap_pf_reset_done : 1;
+ u32 dp_pf_reset_done : 1;
+ u32 qed_pf_reset_done : 1;
+ u32 dqed_pf_reset_done : 1;
+ u32 aqed_pf_reset_done : 1;
+ u32 sys_pf_reset_done : 1;
+ u32 pf_reset_active : 1;
+ u32 flrsm_state : 7;
+ u32 rsvd0 : 13;
+ u32 dlb_proc_reset_done : 1;
+ } field;
+ u32 val;
+};
+
#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS 0xb4000004
#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS_RST 0x9d0fffff
union dlb2_cfg_mstr_cfg_diagnostic_idle_status {
@@ -80,4 +3628,75 @@ union dlb2_cfg_mstr_cfg_pm_pmcsr_disable {
u32 val;
};
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_BYTES 256
+#define DLB2_FUNC_VF_VF2PF_MAILBOX(x) \
+ (0x1000 + (x) * 0x4)
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_RST 0x0
+union dlb2_func_vf_vf2pf_mailbox {
+ struct {
+ u32 msg : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR 0x1f00
+#define DLB2_FUNC_VF_VF2PF_MAILBOX_ISR_RST 0x0
+#define DLB2_FUNC_VF_SIOV_VF2PF_MAILBOX_ISR_TRIGGER 0x8000
+union dlb2_func_vf_vf2pf_mailbox_isr {
+ struct {
+ u32 isr : 1;
+ u32 rsvd0 : 31;
+ } field;
+ u32 val;
+};
+
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_BYTES 64
+#define DLB2_FUNC_VF_PF2VF_MAILBOX(x) \
+ (0x2000 + (x) * 0x4)
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_RST 0x0
+union dlb2_func_vf_pf2vf_mailbox {
+ struct {
+ u32 msg : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR 0x2f00
+#define DLB2_FUNC_VF_PF2VF_MAILBOX_ISR_RST 0x0
+union dlb2_func_vf_pf2vf_mailbox_isr {
+ struct {
+ u32 pf_isr : 1;
+ u32 rsvd0 : 31;
+ } field;
+ u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_MSI_ISR_PEND 0x2f10
+#define DLB2_FUNC_VF_VF_MSI_ISR_PEND_RST 0x0
+union dlb2_func_vf_vf_msi_isr_pend {
+ struct {
+ u32 isr_pend : 32;
+ } field;
+ u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS 0x3000
+#define DLB2_FUNC_VF_VF_RESET_IN_PROGRESS_RST 0x1
+union dlb2_func_vf_vf_reset_in_progress {
+ struct {
+ u32 reset_in_progress : 1;
+ u32 rsvd0 : 31;
+ } field;
+ u32 val;
+};
+
+#define DLB2_FUNC_VF_VF_MSI_ISR 0x4000
+#define DLB2_FUNC_VF_VF_MSI_ISR_RST 0x0
+union dlb2_func_vf_vf_msi_isr {
+ struct {
+ u32 vf_msi_isr : 32;
+ } field;
+ u32 val;
+};
+
#endif /* __DLB2_REGS_H */
diff --git a/drivers/misc/dlb2/dlb2_resource.c b/drivers/misc/dlb2/dlb2_resource.c
index 3b36bfef1e9b..2cc744cea2d5 100644
--- a/drivers/misc/dlb2/dlb2_resource.c
+++ b/drivers/misc/dlb2/dlb2_resource.c
@@ -1,11 +1,40 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright(c) 2016-2020 Intel Corporation */
+#include <linux/frame.h>
+
#include "dlb2_bitmap.h"
#include "dlb2_hw_types.h"
+#include "dlb2_main.h"
#include "dlb2_regs.h"
#include "dlb2_resource.h"
+#define DLB2_DOM_LIST_HEAD(head, type) \
+ list_first_entry_or_null(&(head), type, domain_list)
+
+#define DLB2_FUNC_LIST_HEAD(head, type) \
+ list_first_entry_or_null(&(head), type, func_list)
+
+#define DLB2_DOM_LIST_FOR(head, ptr) \
+ list_for_each_entry(ptr, &(head), domain_list)
+
+#define DLB2_FUNC_LIST_FOR(head, ptr) \
+ list_for_each_entry(ptr, &(head), func_list)
+
+#define DLB2_DOM_LIST_FOR_SAFE(head, ptr, ptr_tmp) \
+ list_for_each_entry_safe(ptr, ptr_tmp, &(head), domain_list)
+
+/*
+ * The PF driver cannot assume that a register write will affect subsequent HCW
+ * writes. To ensure a write completes, the driver must read back a CSR. This
+ * function only need be called for configuration that can occur after the
+ * domain has started; prior to starting, applications can't send HCWs.
+ */
+static inline void dlb2_flush_csr(struct dlb2_hw *hw)
+{
+ DLB2_CSR_RD(hw, DLB2_SYS_TOTAL_VAS);
+}
+
static void dlb2_init_fn_rsrc_lists(struct dlb2_function_resources *rsrc)
{
int i;
@@ -185,6 +214,3109 @@ int dlb2_resource_init(struct dlb2_hw *hw)
return ret;
}
+static struct dlb2_hw_domain *dlb2_get_domain_from_id(struct dlb2_hw *hw,
+ u32 id,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_function_resources *rsrcs;
+ struct dlb2_hw_domain *domain;
+
+ if (id >= DLB2_MAX_NUM_DOMAINS)
+ return NULL;
+
+ if (!vdev_req)
+ return &hw->domains[id];
+
+ rsrcs = &hw->vdev[vdev_id];
+
+ DLB2_FUNC_LIST_FOR(rsrcs->used_domains, domain)
+ if (domain->id.virt_id == id)
+ return domain;
+
+ return NULL;
+}
+
+static struct dlb2_ldb_queue *
+dlb2_get_ldb_queue_from_id(struct dlb2_hw *hw,
+ u32 id,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_function_resources *rsrcs;
+ struct dlb2_hw_domain *domain;
+ struct dlb2_ldb_queue *queue;
+
+ if (id >= DLB2_MAX_NUM_LDB_QUEUES)
+ return NULL;
+
+ rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+ if (!vdev_req)
+ return &hw->rsrcs.ldb_queues[id];
+
+ DLB2_FUNC_LIST_FOR(rsrcs->used_domains, domain) {
+ DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue)
+ if (queue->id.virt_id == id)
+ return queue;
+ }
+
+ DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_queues, queue)
+ if (queue->id.virt_id == id)
+ return queue;
+
+ return NULL;
+}
+
+static int dlb2_attach_ldb_queues(struct dlb2_hw *hw,
+ struct dlb2_function_resources *rsrcs,
+ struct dlb2_hw_domain *domain,
+ u32 num_queues,
+ struct dlb2_cmd_response *resp)
+{
+ unsigned int i;
+
+ if (rsrcs->num_avail_ldb_queues < num_queues) {
+ resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ for (i = 0; i < num_queues; i++) {
+ struct dlb2_ldb_queue *queue;
+
+ queue = DLB2_FUNC_LIST_HEAD(rsrcs->avail_ldb_queues,
+ typeof(*queue));
+ if (!queue) {
+ DLB2_HW_ERR(hw,
+ "[%s()] Internal error: domain validation failed\n",
+ __func__);
+ return -EFAULT;
+ }
+
+ list_del(&queue->func_list);
+
+ queue->domain_id = domain->id;
+ queue->owned = true;
+
+ list_add(&queue->domain_list, &domain->avail_ldb_queues);
+ }
+
+ rsrcs->num_avail_ldb_queues -= num_queues;
+
+ return 0;
+}
+
+static struct dlb2_ldb_port *
+dlb2_get_next_ldb_port(struct dlb2_hw *hw,
+ struct dlb2_function_resources *rsrcs,
+ u32 domain_id,
+ u32 cos_id)
+{
+ struct dlb2_ldb_port *port;
+
+ /*
+ * To reduce the odds of consecutive load-balanced ports mapping to the
+ * same queue(s), the driver attempts to allocate ports whose neighbors
+ * are owned by a different domain.
+ */
+ DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port) {
+ u32 next, prev;
+ u32 phys_id;
+
+ phys_id = port->id.phys_id;
+ next = phys_id + 1;
+ prev = phys_id - 1;
+
+ if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+ next = 0;
+ if (phys_id == 0)
+ prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+ if (!hw->rsrcs.ldb_ports[next].owned ||
+ hw->rsrcs.ldb_ports[next].domain_id.phys_id == domain_id)
+ continue;
+
+ if (!hw->rsrcs.ldb_ports[prev].owned ||
+ hw->rsrcs.ldb_ports[prev].domain_id.phys_id == domain_id)
+ continue;
+
+ return port;
+ }
+
+ /*
+ * Failing that, the driver looks for a port with one neighbor owned by
+ * a different domain and the other unallocated.
+ */
+ DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port) {
+ u32 next, prev;
+ u32 phys_id;
+
+ phys_id = port->id.phys_id;
+ next = phys_id + 1;
+ prev = phys_id - 1;
+
+ if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+ next = 0;
+ if (phys_id == 0)
+ prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+ if (!hw->rsrcs.ldb_ports[prev].owned &&
+ hw->rsrcs.ldb_ports[next].owned &&
+ hw->rsrcs.ldb_ports[next].domain_id.phys_id != domain_id)
+ return port;
+
+ if (!hw->rsrcs.ldb_ports[next].owned &&
+ hw->rsrcs.ldb_ports[prev].owned &&
+ hw->rsrcs.ldb_ports[prev].domain_id.phys_id != domain_id)
+ return port;
+ }
+
+ /*
+ * Failing that, the driver looks for a port with both neighbors
+ * unallocated.
+ */
+ DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[cos_id], port) {
+ u32 next, prev;
+ u32 phys_id;
+
+ phys_id = port->id.phys_id;
+ next = phys_id + 1;
+ prev = phys_id - 1;
+
+ if (phys_id == DLB2_MAX_NUM_LDB_PORTS - 1)
+ next = 0;
+ if (phys_id == 0)
+ prev = DLB2_MAX_NUM_LDB_PORTS - 1;
+
+ if (!hw->rsrcs.ldb_ports[prev].owned &&
+ !hw->rsrcs.ldb_ports[next].owned)
+ return port;
+ }
+
+ /* If all else fails, the driver returns the next available port. */
+ return DLB2_FUNC_LIST_HEAD(rsrcs->avail_ldb_ports[cos_id],
+ typeof(*port));
+}
+
+static int __dlb2_attach_ldb_ports(struct dlb2_hw *hw,
+ struct dlb2_function_resources *rsrcs,
+ struct dlb2_hw_domain *domain,
+ u32 num_ports,
+ u32 cos_id,
+ struct dlb2_cmd_response *resp)
+{
+ unsigned int i;
+
+ if (rsrcs->num_avail_ldb_ports[cos_id] < num_ports) {
+ resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ for (i = 0; i < num_ports; i++) {
+ struct dlb2_ldb_port *port;
+
+ port = dlb2_get_next_ldb_port(hw, rsrcs,
+ domain->id.phys_id, cos_id);
+ if (!port) {
+ DLB2_HW_ERR(hw,
+ "[%s()] Internal error: domain validation failed\n",
+ __func__);
+ return -EFAULT;
+ }
+
+ list_del(&port->func_list);
+
+ port->domain_id = domain->id;
+ port->owned = true;
+
+ list_add(&port->domain_list,
+ &domain->avail_ldb_ports[cos_id]);
+ }
+
+ rsrcs->num_avail_ldb_ports[cos_id] -= num_ports;
+
+ return 0;
+}
+
+static int dlb2_attach_ldb_ports(struct dlb2_hw *hw,
+ struct dlb2_function_resources *rsrcs,
+ struct dlb2_hw_domain *domain,
+ struct dlb2_create_sched_domain_args *args,
+ struct dlb2_cmd_response *resp)
+{
+ unsigned int i;
+ int ret, j;
+
+ if (args->cos_strict) {
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ u32 num = args->num_cos_ldb_ports[i];
+
+ /* Allocate ports from specific classes-of-service */
+ ret = __dlb2_attach_ldb_ports(hw,
+ rsrcs,
+ domain,
+ num,
+ i,
+ resp);
+ if (ret)
+ return ret;
+ }
+ } else {
+ unsigned int k;
+ u32 cos_id;
+
+ /*
+ * Attempt to allocate from specific class-of-service, but
+ * fallback to the other classes if that fails.
+ */
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ for (j = 0; j < args->num_cos_ldb_ports[i]; j++) {
+ for (k = 0; k < DLB2_NUM_COS_DOMAINS; k++) {
+ cos_id = (i + k) % DLB2_NUM_COS_DOMAINS;
+
+ ret = __dlb2_attach_ldb_ports(hw,
+ rsrcs,
+ domain,
+ 1,
+ cos_id,
+ resp);
+ if (ret == 0)
+ break;
+ }
+
+ if (ret < 0)
+ return ret;
+ }
+ }
+ }
+
+ /* Allocate num_ldb_ports from any class-of-service */
+ for (i = 0; i < args->num_ldb_ports; i++) {
+ for (j = 0; j < DLB2_NUM_COS_DOMAINS; j++) {
+ ret = __dlb2_attach_ldb_ports(hw,
+ rsrcs,
+ domain,
+ 1,
+ j,
+ resp);
+ if (ret == 0)
+ break;
+ }
+
+ if (ret < 0)
+ return ret;
+ }
+
+ return 0;
+}
+
+static int dlb2_attach_dir_ports(struct dlb2_hw *hw,
+ struct dlb2_function_resources *rsrcs,
+ struct dlb2_hw_domain *domain,
+ u32 num_ports,
+ struct dlb2_cmd_response *resp)
+{
+ unsigned int i;
+
+ if (rsrcs->num_avail_dir_pq_pairs < num_ports) {
+ resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ for (i = 0; i < num_ports; i++) {
+ struct dlb2_dir_pq_pair *port;
+
+ port = DLB2_FUNC_LIST_HEAD(rsrcs->avail_dir_pq_pairs,
+ typeof(*port));
+ if (!port) {
+ DLB2_HW_ERR(hw,
+ "[%s()] Internal error: domain validation failed\n",
+ __func__);
+ return -EFAULT;
+ }
+
+ list_del(&port->func_list);
+
+ port->domain_id = domain->id;
+ port->owned = true;
+
+ list_add(&port->domain_list, &domain->avail_dir_pq_pairs);
+ }
+
+ rsrcs->num_avail_dir_pq_pairs -= num_ports;
+
+ return 0;
+}
+
+static int dlb2_attach_ldb_credits(struct dlb2_function_resources *rsrcs,
+ struct dlb2_hw_domain *domain,
+ u32 num_credits,
+ struct dlb2_cmd_response *resp)
+{
+ if (rsrcs->num_avail_qed_entries < num_credits) {
+ resp->status = DLB2_ST_LDB_CREDITS_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ rsrcs->num_avail_qed_entries -= num_credits;
+ domain->num_ldb_credits += num_credits;
+ return 0;
+}
+
+static int dlb2_attach_dir_credits(struct dlb2_function_resources *rsrcs,
+ struct dlb2_hw_domain *domain,
+ u32 num_credits,
+ struct dlb2_cmd_response *resp)
+{
+ if (rsrcs->num_avail_dqed_entries < num_credits) {
+ resp->status = DLB2_ST_DIR_CREDITS_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ rsrcs->num_avail_dqed_entries -= num_credits;
+ domain->num_dir_credits += num_credits;
+ return 0;
+}
+
+static int dlb2_attach_atomic_inflights(struct dlb2_function_resources *rsrcs,
+ struct dlb2_hw_domain *domain,
+ u32 num_atomic_inflights,
+ struct dlb2_cmd_response *resp)
+{
+ if (rsrcs->num_avail_aqed_entries < num_atomic_inflights) {
+ resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ rsrcs->num_avail_aqed_entries -= num_atomic_inflights;
+ domain->num_avail_aqed_entries += num_atomic_inflights;
+ return 0;
+}
+
+static int
+dlb2_attach_domain_hist_list_entries(struct dlb2_function_resources *rsrcs,
+ struct dlb2_hw_domain *domain,
+ u32 num_hist_list_entries,
+ struct dlb2_cmd_response *resp)
+{
+ struct dlb2_bitmap *bitmap;
+ int base;
+
+ if (num_hist_list_entries) {
+ bitmap = rsrcs->avail_hist_list_entries;
+
+ base = dlb2_bitmap_find_set_bit_range(bitmap,
+ num_hist_list_entries);
+ if (base < 0)
+ goto error;
+
+ domain->total_hist_list_entries = num_hist_list_entries;
+ domain->avail_hist_list_entries = num_hist_list_entries;
+ domain->hist_list_entry_base = base;
+ domain->hist_list_entry_offset = 0;
+
+ dlb2_bitmap_clear_range(bitmap, base, num_hist_list_entries);
+ }
+ return 0;
+
+error:
+ resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+ return -EINVAL;
+}
+
+static int
+dlb2_verify_create_sched_dom_args(struct dlb2_function_resources *rsrcs,
+ struct dlb2_create_sched_domain_args *args,
+ struct dlb2_cmd_response *resp)
+{
+ u32 num_avail_ldb_ports, req_ldb_ports;
+ struct dlb2_bitmap *avail_hl_entries;
+ unsigned int max_contig_hl_range;
+ int i;
+
+ avail_hl_entries = rsrcs->avail_hist_list_entries;
+
+ max_contig_hl_range = dlb2_bitmap_longest_set_range(avail_hl_entries);
+
+ num_avail_ldb_ports = 0;
+ req_ldb_ports = 0;
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ num_avail_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+ req_ldb_ports += args->num_cos_ldb_ports[i];
+ }
+
+ req_ldb_ports += args->num_ldb_ports;
+
+ if (rsrcs->num_avail_domains < 1) {
+ resp->status = DLB2_ST_DOMAIN_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ if (rsrcs->num_avail_ldb_queues < args->num_ldb_queues) {
+ resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ if (req_ldb_ports > num_avail_ldb_ports) {
+ resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ for (i = 0; args->cos_strict && i < DLB2_NUM_COS_DOMAINS; i++) {
+ if (args->num_cos_ldb_ports[i] >
+ rsrcs->num_avail_ldb_ports[i]) {
+ resp->status = DLB2_ST_LDB_PORTS_UNAVAILABLE;
+ return -EINVAL;
+ }
+ }
+
+ if (args->num_ldb_queues > 0 && req_ldb_ports == 0) {
+ resp->status = DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES;
+ return -EINVAL;
+ }
+
+ if (rsrcs->num_avail_dir_pq_pairs < args->num_dir_ports) {
+ resp->status = DLB2_ST_DIR_PORTS_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ if (rsrcs->num_avail_qed_entries < args->num_ldb_credits) {
+ resp->status = DLB2_ST_LDB_CREDITS_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ if (rsrcs->num_avail_dqed_entries < args->num_dir_credits) {
+ resp->status = DLB2_ST_DIR_CREDITS_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ if (rsrcs->num_avail_aqed_entries < args->num_atomic_inflights) {
+ resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ if (max_contig_hl_range < args->num_hist_list_entries) {
+ resp->status = DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ if (resp->status)
+ return -EINVAL;
+
+ return 0;
+}
+
+static bool dlb2_port_find_slot(struct dlb2_ldb_port *port,
+ enum dlb2_qid_map_state state,
+ int *slot)
+{
+ int i;
+
+ for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+ if (port->qid_map[i].state == state)
+ break;
+ }
+
+ *slot = i;
+
+ return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static bool dlb2_port_find_slot_queue(struct dlb2_ldb_port *port,
+ enum dlb2_qid_map_state state,
+ struct dlb2_ldb_queue *queue,
+ int *slot)
+{
+ int i;
+
+ for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+ if (port->qid_map[i].state == state &&
+ port->qid_map[i].qid == queue->id.phys_id)
+ break;
+ }
+
+ *slot = i;
+
+ return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static int dlb2_port_slot_state_transition(struct dlb2_hw *hw,
+ struct dlb2_ldb_port *port,
+ struct dlb2_ldb_queue *queue,
+ int slot,
+ enum dlb2_qid_map_state new_state)
+{
+ enum dlb2_qid_map_state curr_state = port->qid_map[slot].state;
+ struct dlb2_hw_domain *domain;
+ int domain_id;
+
+ domain_id = port->domain_id.phys_id;
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, false, 0);
+ if (!domain) {
+ DLB2_HW_ERR(hw,
+ "[%s()] Internal error: unable to find domain %d\n",
+ __func__, domain_id);
+ return -EINVAL;
+ }
+
+ switch (curr_state) {
+ case DLB2_QUEUE_UNMAPPED:
+ switch (new_state) {
+ case DLB2_QUEUE_MAPPED:
+ queue->num_mappings++;
+ port->num_mappings++;
+ break;
+ case DLB2_QUEUE_MAP_IN_PROG:
+ queue->num_pending_additions++;
+ domain->num_pending_additions++;
+ break;
+ default:
+ goto error;
+ }
+ break;
+ case DLB2_QUEUE_MAPPED:
+ switch (new_state) {
+ case DLB2_QUEUE_UNMAPPED:
+ queue->num_mappings--;
+ port->num_mappings--;
+ break;
+ case DLB2_QUEUE_UNMAP_IN_PROG:
+ port->num_pending_removals++;
+ domain->num_pending_removals++;
+ break;
+ case DLB2_QUEUE_MAPPED:
+ /* Priority change, nothing to update */
+ break;
+ default:
+ goto error;
+ }
+ break;
+ case DLB2_QUEUE_MAP_IN_PROG:
+ switch (new_state) {
+ case DLB2_QUEUE_UNMAPPED:
+ queue->num_pending_additions--;
+ domain->num_pending_additions--;
+ break;
+ case DLB2_QUEUE_MAPPED:
+ queue->num_mappings++;
+ port->num_mappings++;
+ queue->num_pending_additions--;
+ domain->num_pending_additions--;
+ break;
+ default:
+ goto error;
+ }
+ break;
+ case DLB2_QUEUE_UNMAP_IN_PROG:
+ switch (new_state) {
+ case DLB2_QUEUE_UNMAPPED:
+ port->num_pending_removals--;
+ domain->num_pending_removals--;
+ queue->num_mappings--;
+ port->num_mappings--;
+ break;
+ case DLB2_QUEUE_MAPPED:
+ port->num_pending_removals--;
+ domain->num_pending_removals--;
+ break;
+ case DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+ /* Nothing to update */
+ break;
+ default:
+ goto error;
+ }
+ break;
+ case DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+ switch (new_state) {
+ case DLB2_QUEUE_UNMAP_IN_PROG:
+ /* Nothing to update */
+ break;
+ case DLB2_QUEUE_UNMAPPED:
+ /*
+ * An UNMAP_IN_PROG_PENDING_MAP slot briefly
+ * becomes UNMAPPED before it transitions to
+ * MAP_IN_PROG.
+ */
+ queue->num_mappings--;
+ port->num_mappings--;
+ port->num_pending_removals--;
+ domain->num_pending_removals--;
+ break;
+ default:
+ goto error;
+ }
+ break;
+ default:
+ goto error;
+ }
+
+ port->qid_map[slot].state = new_state;
+
+ DLB2_HW_DBG(hw,
+ "[%s()] queue %d -> port %d state transition (%d -> %d)\n",
+ __func__, queue->id.phys_id, port->id.phys_id,
+ curr_state, new_state);
+ return 0;
+
+error:
+ DLB2_HW_ERR(hw,
+ "[%s()] Internal error: invalid queue %d -> port %d state transition (%d -> %d)\n",
+ __func__, queue->id.phys_id, port->id.phys_id,
+ curr_state, new_state);
+ return -EFAULT;
+}
+
+static void dlb2_configure_domain_credits(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain)
+{
+ union dlb2_chp_cfg_ldb_vas_crd r0 = { {0} };
+ union dlb2_chp_cfg_dir_vas_crd r1 = { {0} };
+
+ r0.field.count = domain->num_ldb_credits;
+
+ DLB2_CSR_WR(hw, DLB2_CHP_CFG_LDB_VAS_CRD(domain->id.phys_id), r0.val);
+
+ r1.field.count = domain->num_dir_credits;
+
+ DLB2_CSR_WR(hw, DLB2_CHP_CFG_DIR_VAS_CRD(domain->id.phys_id), r1.val);
+}
+
+static int
+dlb2_domain_attach_resources(struct dlb2_hw *hw,
+ struct dlb2_function_resources *rsrcs,
+ struct dlb2_hw_domain *domain,
+ struct dlb2_create_sched_domain_args *args,
+ struct dlb2_cmd_response *resp)
+{
+ int ret;
+
+ ret = dlb2_attach_ldb_queues(hw,
+ rsrcs,
+ domain,
+ args->num_ldb_queues,
+ resp);
+ if (ret < 0)
+ return ret;
+
+ ret = dlb2_attach_ldb_ports(hw,
+ rsrcs,
+ domain,
+ args,
+ resp);
+ if (ret < 0)
+ return ret;
+
+ ret = dlb2_attach_dir_ports(hw,
+ rsrcs,
+ domain,
+ args->num_dir_ports,
+ resp);
+ if (ret < 0)
+ return ret;
+
+ ret = dlb2_attach_ldb_credits(rsrcs,
+ domain,
+ args->num_ldb_credits,
+ resp);
+ if (ret < 0)
+ return ret;
+
+ ret = dlb2_attach_dir_credits(rsrcs,
+ domain,
+ args->num_dir_credits,
+ resp);
+ if (ret < 0)
+ return ret;
+
+ ret = dlb2_attach_domain_hist_list_entries(rsrcs,
+ domain,
+ args->num_hist_list_entries,
+ resp);
+ if (ret < 0)
+ return ret;
+
+ ret = dlb2_attach_atomic_inflights(rsrcs,
+ domain,
+ args->num_atomic_inflights,
+ resp);
+ if (ret < 0)
+ return ret;
+
+ dlb2_configure_domain_credits(hw, domain);
+
+ domain->configured = true;
+
+ domain->started = false;
+
+ rsrcs->num_avail_domains--;
+
+ return 0;
+}
+
+static void dlb2_ldb_port_cq_enable(struct dlb2_hw *hw,
+ struct dlb2_ldb_port *port)
+{
+ union dlb2_lsp_cq_ldb_dsbl reg;
+
+ /*
+ * Don't re-enable the port if a removal is pending. The caller should
+ * mark this port as enabled (if it isn't already), and when the
+ * removal completes the port will be enabled.
+ */
+ if (port->num_pending_removals)
+ return;
+
+ reg.field.disabled = 0;
+
+ DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id), reg.val);
+
+ dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_port_cq_disable(struct dlb2_hw *hw,
+ struct dlb2_ldb_port *port)
+{
+ union dlb2_lsp_cq_ldb_dsbl reg;
+
+ reg.field.disabled = 1;
+
+ DLB2_CSR_WR(hw, DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id), reg.val);
+
+ dlb2_flush_csr(hw);
+}
+
+static void dlb2_dir_port_cq_enable(struct dlb2_hw *hw,
+ struct dlb2_dir_pq_pair *port)
+{
+ union dlb2_lsp_cq_dir_dsbl reg;
+
+ reg.field.disabled = 0;
+
+ DLB2_CSR_WR(hw, DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id), reg.val);
+
+ dlb2_flush_csr(hw);
+}
+
+static void dlb2_dir_port_cq_disable(struct dlb2_hw *hw,
+ struct dlb2_dir_pq_pair *port)
+{
+ union dlb2_lsp_cq_dir_dsbl reg;
+
+ reg.field.disabled = 1;
+
+ DLB2_CSR_WR(hw, DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id), reg.val);
+
+ dlb2_flush_csr(hw);
+}
+
+static int dlb2_ldb_port_map_qid_static(struct dlb2_hw *hw,
+ struct dlb2_ldb_port *p,
+ struct dlb2_ldb_queue *q,
+ u8 priority)
+{
+ union dlb2_lsp_cq2priov r0;
+ union dlb2_lsp_cq2qid0 r1;
+ union dlb2_atm_qid2cqidix_00 r2;
+ union dlb2_lsp_qid2cqidix_00 r3;
+ union dlb2_lsp_qid2cqidix2_00 r4;
+ enum dlb2_qid_map_state state;
+ int i;
+
+ /* Look for a pending or already mapped slot, else an unused slot */
+ if (!dlb2_port_find_slot_queue(p, DLB2_QUEUE_MAP_IN_PROG, q, &i) &&
+ !dlb2_port_find_slot_queue(p, DLB2_QUEUE_MAPPED, q, &i) &&
+ !dlb2_port_find_slot(p, DLB2_QUEUE_UNMAPPED, &i)) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: CQ has no available QID mapping slots\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: port slot tracking failed\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ /* Read-modify-write the priority and valid bit register */
+ r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(p->id.phys_id));
+
+ r0.field.v |= 1 << i;
+ r0.field.prio |= (priority & 0x7) << i * 3;
+
+ DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(p->id.phys_id), r0.val);
+
+ /* Read-modify-write the QID map register */
+ if (i < 4)
+ r1.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2QID0(p->id.phys_id));
+ else
+ r1.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2QID1(p->id.phys_id));
+
+ if (i == 0 || i == 4)
+ r1.field.qid_p0 = q->id.phys_id;
+ if (i == 1 || i == 5)
+ r1.field.qid_p1 = q->id.phys_id;
+ if (i == 2 || i == 6)
+ r1.field.qid_p2 = q->id.phys_id;
+ if (i == 3 || i == 7)
+ r1.field.qid_p3 = q->id.phys_id;
+
+ if (i < 4)
+ DLB2_CSR_WR(hw, DLB2_LSP_CQ2QID0(p->id.phys_id), r1.val);
+ else
+ DLB2_CSR_WR(hw, DLB2_LSP_CQ2QID1(p->id.phys_id), r1.val);
+
+ r2.val = DLB2_CSR_RD(hw,
+ DLB2_ATM_QID2CQIDIX(q->id.phys_id,
+ p->id.phys_id / 4));
+
+ r3.val = DLB2_CSR_RD(hw,
+ DLB2_LSP_QID2CQIDIX(q->id.phys_id,
+ p->id.phys_id / 4));
+
+ r4.val = DLB2_CSR_RD(hw,
+ DLB2_LSP_QID2CQIDIX2(q->id.phys_id,
+ p->id.phys_id / 4));
+
+ switch (p->id.phys_id % 4) {
+ case 0:
+ r2.field.cq_p0 |= 1 << i;
+ r3.field.cq_p0 |= 1 << i;
+ r4.field.cq_p0 |= 1 << i;
+ break;
+
+ case 1:
+ r2.field.cq_p1 |= 1 << i;
+ r3.field.cq_p1 |= 1 << i;
+ r4.field.cq_p1 |= 1 << i;
+ break;
+
+ case 2:
+ r2.field.cq_p2 |= 1 << i;
+ r3.field.cq_p2 |= 1 << i;
+ r4.field.cq_p2 |= 1 << i;
+ break;
+
+ case 3:
+ r2.field.cq_p3 |= 1 << i;
+ r3.field.cq_p3 |= 1 << i;
+ r4.field.cq_p3 |= 1 << i;
+ break;
+ }
+
+ DLB2_CSR_WR(hw,
+ DLB2_ATM_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+ r2.val);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+ r3.val);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_QID2CQIDIX2(q->id.phys_id, p->id.phys_id / 4),
+ r4.val);
+
+ dlb2_flush_csr(hw);
+
+ p->qid_map[i].qid = q->id.phys_id;
+ p->qid_map[i].priority = priority;
+
+ state = DLB2_QUEUE_MAPPED;
+
+ return dlb2_port_slot_state_transition(hw, p, q, i, state);
+}
+
+static int dlb2_ldb_port_set_has_work_bits(struct dlb2_hw *hw,
+ struct dlb2_ldb_port *port,
+ struct dlb2_ldb_queue *queue,
+ int slot)
+{
+ union dlb2_lsp_qid_aqed_active_cnt r0;
+ union dlb2_lsp_qid_ldb_enqueue_cnt r1;
+ union dlb2_lsp_ldb_sched_ctrl r2 = { {0} };
+
+ /* Set the atomic scheduling haswork bit */
+ r0.val = DLB2_CSR_RD(hw,
+ DLB2_LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+
+ r2.field.cq = port->id.phys_id;
+ r2.field.qidix = slot;
+ r2.field.value = 1;
+ r2.field.rlist_haswork_v = r0.field.count > 0;
+
+ /* Set the non-atomic scheduling haswork bit */
+ DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+ r1.val = DLB2_CSR_RD(hw,
+ DLB2_LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+
+ memset(&r2, 0, sizeof(r2));
+
+ r2.field.cq = port->id.phys_id;
+ r2.field.qidix = slot;
+ r2.field.value = 1;
+ r2.field.nalb_haswork_v = (r1.field.count > 0);
+
+ DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+ dlb2_flush_csr(hw);
+
+ return 0;
+}
+
+static void dlb2_ldb_port_clear_has_work_bits(struct dlb2_hw *hw,
+ struct dlb2_ldb_port *port,
+ u8 slot)
+{
+ union dlb2_lsp_ldb_sched_ctrl r2 = { {0} };
+
+ r2.field.cq = port->id.phys_id;
+ r2.field.qidix = slot;
+ r2.field.value = 0;
+ r2.field.rlist_haswork_v = 1;
+
+ DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+ memset(&r2, 0, sizeof(r2));
+
+ r2.field.cq = port->id.phys_id;
+ r2.field.qidix = slot;
+ r2.field.value = 0;
+ r2.field.nalb_haswork_v = 1;
+
+ DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r2.val);
+
+ dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_port_clear_queue_if_status(struct dlb2_hw *hw,
+ struct dlb2_ldb_port *port,
+ int slot)
+{
+ union dlb2_lsp_ldb_sched_ctrl r0 = { {0} };
+
+ r0.field.cq = port->id.phys_id;
+ r0.field.qidix = slot;
+ r0.field.value = 0;
+ r0.field.inflight_ok_v = 1;
+
+ DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r0.val);
+
+ dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_port_set_queue_if_status(struct dlb2_hw *hw,
+ struct dlb2_ldb_port *port,
+ int slot)
+{
+ union dlb2_lsp_ldb_sched_ctrl r0 = { {0} };
+
+ r0.field.cq = port->id.phys_id;
+ r0.field.qidix = slot;
+ r0.field.value = 1;
+ r0.field.inflight_ok_v = 1;
+
+ DLB2_CSR_WR(hw, DLB2_LSP_LDB_SCHED_CTRL, r0.val);
+
+ dlb2_flush_csr(hw);
+}
+
+static void dlb2_ldb_queue_set_inflight_limit(struct dlb2_hw *hw,
+ struct dlb2_ldb_queue *queue)
+{
+ union dlb2_lsp_qid_ldb_infl_lim r0 = { {0} };
+
+ r0.field.limit = queue->num_qid_inflights;
+
+ DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), r0.val);
+}
+
+static void dlb2_ldb_queue_clear_inflight_limit(struct dlb2_hw *hw,
+ struct dlb2_ldb_queue *queue)
+{
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id),
+ DLB2_LSP_QID_LDB_INFL_LIM_RST);
+}
+
+/*
+ * dlb2_ldb_queue_{enable, disable}_mapped_cqs() don't operate exactly as
+ * their function names imply, and should only be called by the dynamic CQ
+ * mapping code.
+ */
+static void dlb2_ldb_queue_disable_mapped_cqs(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain,
+ struct dlb2_ldb_queue *queue)
+{
+ struct dlb2_ldb_port *port;
+ int slot, i;
+
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port) {
+ enum dlb2_qid_map_state state = DLB2_QUEUE_MAPPED;
+
+ if (!dlb2_port_find_slot_queue(port, state,
+ queue, &slot))
+ continue;
+
+ if (port->enabled)
+ dlb2_ldb_port_cq_disable(hw, port);
+ }
+ }
+}
+
+static void dlb2_ldb_queue_enable_mapped_cqs(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain,
+ struct dlb2_ldb_queue *queue)
+{
+ struct dlb2_ldb_port *port;
+ int slot, i;
+
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port) {
+ enum dlb2_qid_map_state state = DLB2_QUEUE_MAPPED;
+
+ if (!dlb2_port_find_slot_queue(port, state,
+ queue, &slot))
+ continue;
+
+ if (port->enabled)
+ dlb2_ldb_port_cq_enable(hw, port);
+ }
+ }
+}
+
+static int dlb2_ldb_port_finish_map_qid_dynamic(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain,
+ struct dlb2_ldb_port *port,
+ struct dlb2_ldb_queue *queue)
+{
+ union dlb2_lsp_qid_ldb_infl_cnt r0;
+ enum dlb2_qid_map_state state;
+ int slot, ret, i;
+ u8 prio;
+
+ r0.val = DLB2_CSR_RD(hw,
+ DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+ if (r0.field.count) {
+ DLB2_HW_ERR(hw,
+ "[%s()] Internal error: non-zero QID inflight count\n",
+ __func__);
+ return -EINVAL;
+ }
+
+ /*
+ * Static map the port and set its corresponding has_work bits.
+ */
+ state = DLB2_QUEUE_MAP_IN_PROG;
+ if (!dlb2_port_find_slot_queue(port, state, queue, &slot))
+ return -EINVAL;
+
+ if (slot >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: port slot tracking failed\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ prio = port->qid_map[slot].priority;
+
+ /*
+ * Update the CQ2QID, CQ2PRIOV, and QID2CQIDX registers, and
+ * the port's qid_map state.
+ */
+ ret = dlb2_ldb_port_map_qid_static(hw, port, queue, prio);
+ if (ret)
+ return ret;
+
+ ret = dlb2_ldb_port_set_has_work_bits(hw, port, queue, slot);
+ if (ret)
+ return ret;
+
+ /*
+ * Ensure IF_status(cq,qid) is 0 before enabling the port to
+ * prevent spurious schedules to cause the queue's inflight
+ * count to increase.
+ */
+ dlb2_ldb_port_clear_queue_if_status(hw, port, slot);
+
+ /* Reset the queue's inflight status */
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port) {
+ state = DLB2_QUEUE_MAPPED;
+ if (!dlb2_port_find_slot_queue(port, state,
+ queue, &slot))
+ continue;
+
+ dlb2_ldb_port_set_queue_if_status(hw, port, slot);
+ }
+ }
+
+ dlb2_ldb_queue_set_inflight_limit(hw, queue);
+
+ /* Re-enable CQs mapped to this queue */
+ dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+ /* If this queue has other mappings pending, clear its inflight limit */
+ if (queue->num_pending_additions > 0)
+ dlb2_ldb_queue_clear_inflight_limit(hw, queue);
+
+ return 0;
+}
+
+/**
+ * dlb2_ldb_port_map_qid_dynamic() - perform a "dynamic" QID->CQ mapping
+ * @hw: dlb2_hw handle for a particular device.
+ * @port: load-balanced port
+ * @queue: load-balanced queue
+ * @priority: queue servicing priority
+ *
+ * Returns 0 if the queue was mapped, 1 if the mapping is scheduled to occur
+ * at a later point, and <0 if an error occurred.
+ */
+static int dlb2_ldb_port_map_qid_dynamic(struct dlb2_hw *hw,
+ struct dlb2_ldb_port *port,
+ struct dlb2_ldb_queue *queue,
+ u8 priority)
+{
+ union dlb2_lsp_qid_ldb_infl_cnt r0 = { {0} };
+ enum dlb2_qid_map_state state;
+ struct dlb2_hw_domain *domain;
+ int domain_id, slot, ret;
+
+ domain_id = port->domain_id.phys_id;
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, false, 0);
+ if (!domain) {
+ DLB2_HW_ERR(hw,
+ "[%s()] Internal error: unable to find domain %d\n",
+ __func__, port->domain_id.phys_id);
+ return -EINVAL;
+ }
+
+ /*
+ * Set the QID inflight limit to 0 to prevent further scheduling of the
+ * queue.
+ */
+ DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), 0);
+
+ if (!dlb2_port_find_slot(port, DLB2_QUEUE_UNMAPPED, &slot)) {
+ DLB2_HW_ERR(hw,
+ "Internal error: No available unmapped slots\n");
+ return -EFAULT;
+ }
+
+ if (slot >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: port slot tracking failed\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ port->qid_map[slot].qid = queue->id.phys_id;
+ port->qid_map[slot].priority = priority;
+
+ state = DLB2_QUEUE_MAP_IN_PROG;
+ ret = dlb2_port_slot_state_transition(hw, port, queue, slot, state);
+ if (ret)
+ return ret;
+
+ /*
+ * Disable the affected CQ, and the CQs already mapped to the QID,
+ * before reading the QID's inflight count a second time. There is an
+ * unlikely race in which the QID may schedule one more QE after we
+ * read an inflight count of 0, and disabling the CQs guarantees that
+ * the race will not occur after a re-read of the inflight count
+ * register.
+ */
+ if (port->enabled)
+ dlb2_ldb_port_cq_disable(hw, port);
+
+ dlb2_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+ r0.val = DLB2_CSR_RD(hw,
+ DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+ if (r0.field.count) {
+ if (port->enabled)
+ dlb2_ldb_port_cq_enable(hw, port);
+
+ dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+ return 1;
+ }
+
+ return dlb2_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+}
+
+static int dlb2_ldb_port_map_qid(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain,
+ struct dlb2_ldb_port *port,
+ struct dlb2_ldb_queue *queue,
+ u8 prio)
+{
+ if (domain->started)
+ return dlb2_ldb_port_map_qid_dynamic(hw, port, queue, prio);
+ else
+ return dlb2_ldb_port_map_qid_static(hw, port, queue, prio);
+}
+
+static int dlb2_ldb_port_unmap_qid(struct dlb2_hw *hw,
+ struct dlb2_ldb_port *port,
+ struct dlb2_ldb_queue *queue)
+{
+ enum dlb2_qid_map_state mapped, in_progress, pending_map, unmapped;
+ union dlb2_lsp_cq2priov r0;
+ union dlb2_atm_qid2cqidix_00 r1;
+ union dlb2_lsp_qid2cqidix_00 r2;
+ union dlb2_lsp_qid2cqidix2_00 r3;
+ u32 queue_id;
+ u32 port_id;
+ int i;
+
+ /* Find the queue's slot */
+ mapped = DLB2_QUEUE_MAPPED;
+ in_progress = DLB2_QUEUE_UNMAP_IN_PROG;
+ pending_map = DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+ if (!dlb2_port_find_slot_queue(port, mapped, queue, &i) &&
+ !dlb2_port_find_slot_queue(port, in_progress, queue, &i) &&
+ !dlb2_port_find_slot_queue(port, pending_map, queue, &i)) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: QID %d isn't mapped\n",
+ __func__, __LINE__, queue->id.phys_id);
+ return -EFAULT;
+ }
+
+ if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: port slot tracking failed\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ port_id = port->id.phys_id;
+ queue_id = queue->id.phys_id;
+
+ /* Read-modify-write the priority and valid bit register */
+ r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(port_id));
+
+ r0.field.v &= ~(1 << i);
+
+ DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port_id), r0.val);
+
+ r1.val = DLB2_CSR_RD(hw,
+ DLB2_ATM_QID2CQIDIX(queue_id, port_id / 4));
+
+ r2.val = DLB2_CSR_RD(hw,
+ DLB2_LSP_QID2CQIDIX(queue_id, port_id / 4));
+
+ r3.val = DLB2_CSR_RD(hw,
+ DLB2_LSP_QID2CQIDIX2(queue_id, port_id / 4));
+
+ switch (port_id % 4) {
+ case 0:
+ r1.field.cq_p0 &= ~(1 << i);
+ r2.field.cq_p0 &= ~(1 << i);
+ r3.field.cq_p0 &= ~(1 << i);
+ break;
+
+ case 1:
+ r1.field.cq_p1 &= ~(1 << i);
+ r2.field.cq_p1 &= ~(1 << i);
+ r3.field.cq_p1 &= ~(1 << i);
+ break;
+
+ case 2:
+ r1.field.cq_p2 &= ~(1 << i);
+ r2.field.cq_p2 &= ~(1 << i);
+ r3.field.cq_p2 &= ~(1 << i);
+ break;
+
+ case 3:
+ r1.field.cq_p3 &= ~(1 << i);
+ r2.field.cq_p3 &= ~(1 << i);
+ r3.field.cq_p3 &= ~(1 << i);
+ break;
+ }
+
+ DLB2_CSR_WR(hw,
+ DLB2_ATM_QID2CQIDIX(queue_id, port_id / 4),
+ r1.val);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_QID2CQIDIX(queue_id, port_id / 4),
+ r2.val);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_QID2CQIDIX2(queue_id, port_id / 4),
+ r3.val);
+
+ dlb2_flush_csr(hw);
+
+ unmapped = DLB2_QUEUE_UNMAPPED;
+
+ return dlb2_port_slot_state_transition(hw, port, queue, i, unmapped);
+}
+
+static void
+dlb2_log_create_sched_domain_args(struct dlb2_hw *hw,
+ struct dlb2_create_sched_domain_args *args,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ DLB2_HW_DBG(hw, "DLB2 create sched domain arguments:\n");
+ if (vdev_req)
+ DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+ DLB2_HW_DBG(hw, "\tNumber of LDB queues: %d\n",
+ args->num_ldb_queues);
+ DLB2_HW_DBG(hw, "\tNumber of LDB ports (any CoS): %d\n",
+ args->num_ldb_ports);
+ DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 0): %d\n",
+ args->num_cos_ldb_ports[0]);
+ DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 1): %d\n",
+ args->num_cos_ldb_ports[1]);
+ DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 2): %d\n",
+ args->num_cos_ldb_ports[2]);
+ DLB2_HW_DBG(hw, "\tNumber of LDB ports (CoS 3): %d\n",
+ args->num_cos_ldb_ports[3]);
+ DLB2_HW_DBG(hw, "\tStrict CoS allocation: %d\n",
+ args->cos_strict);
+ DLB2_HW_DBG(hw, "\tNumber of DIR ports: %d\n",
+ args->num_dir_ports);
+ DLB2_HW_DBG(hw, "\tNumber of ATM inflights: %d\n",
+ args->num_atomic_inflights);
+ DLB2_HW_DBG(hw, "\tNumber of hist list entries: %d\n",
+ args->num_hist_list_entries);
+ DLB2_HW_DBG(hw, "\tNumber of LDB credits: %d\n",
+ args->num_ldb_credits);
+ DLB2_HW_DBG(hw, "\tNumber of DIR credits: %d\n",
+ args->num_dir_credits);
+}
+
+/**
+ * dlb2_hw_create_sched_domain() - Allocate and initialize a DLB scheduling
+ * domain and its resources.
+ * @hw: Contains the current state of the DLB2 hardware.
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
+ struct dlb2_create_sched_domain_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_function_resources *rsrcs;
+ struct dlb2_hw_domain *domain;
+ int ret;
+
+ rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+ dlb2_log_create_sched_domain_args(hw, args, vdev_req, vdev_id);
+
+ /*
+ * Verify that hardware resources are available before attempting to
+ * satisfy the request. This simplifies the error unwinding code.
+ */
+ ret = dlb2_verify_create_sched_dom_args(rsrcs, args, resp);
+ if (ret)
+ return ret;
+
+ domain = DLB2_FUNC_LIST_HEAD(rsrcs->avail_domains, typeof(*domain));
+
+ if (!domain) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: no available domains\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ if (domain->configured) {
+ DLB2_HW_ERR(hw,
+ "[%s()] Internal error: avail_domains contains configured domains.\n",
+ __func__);
+ return -EFAULT;
+ }
+
+ dlb2_init_domain_rsrc_lists(domain);
+
+ ret = dlb2_domain_attach_resources(hw, rsrcs, domain, args, resp);
+ if (ret < 0) {
+ DLB2_HW_ERR(hw,
+ "[%s()] Internal error: failed to verify args.\n",
+ __func__);
+
+ return ret;
+ }
+
+ list_del(&domain->func_list);
+
+ list_add(&domain->func_list, &rsrcs->used_domains);
+
+ resp->id = (vdev_req) ? domain->id.virt_id : domain->id.phys_id;
+ resp->status = 0;
+
+ return 0;
+}
+
+static void
+dlb2_domain_finish_unmap_port_slot(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain,
+ struct dlb2_ldb_port *port,
+ int slot)
+{
+ enum dlb2_qid_map_state state;
+ struct dlb2_ldb_queue *queue;
+
+ queue = &hw->rsrcs.ldb_queues[port->qid_map[slot].qid];
+
+ state = port->qid_map[slot].state;
+
+ /* Update the QID2CQIDX and CQ2QID vectors */
+ dlb2_ldb_port_unmap_qid(hw, port, queue);
+
+ /*
+ * Ensure the QID will not be serviced by this {CQ, slot} by clearing
+ * the has_work bits
+ */
+ dlb2_ldb_port_clear_has_work_bits(hw, port, slot);
+
+ /* Reset the {CQ, slot} to its default state */
+ dlb2_ldb_port_set_queue_if_status(hw, port, slot);
+
+ /* Re-enable the CQ if it wasn't manually disabled by the user */
+ if (port->enabled)
+ dlb2_ldb_port_cq_enable(hw, port);
+
+ /*
+ * If there is a mapping that is pending this slot's removal, perform
+ * the mapping now.
+ */
+ if (state == DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP) {
+ struct dlb2_ldb_port_qid_map *map;
+ struct dlb2_ldb_queue *map_queue;
+ u8 prio;
+
+ map = &port->qid_map[slot];
+
+ map->qid = map->pending_qid;
+ map->priority = map->pending_priority;
+
+ map_queue = &hw->rsrcs.ldb_queues[map->qid];
+ prio = map->priority;
+
+ dlb2_ldb_port_map_qid(hw, domain, port, map_queue, prio);
+ }
+}
+
+static bool dlb2_domain_finish_unmap_port(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain,
+ struct dlb2_ldb_port *port)
+{
+ union dlb2_lsp_cq_ldb_infl_cnt r0;
+ int i;
+
+ if (port->num_pending_removals == 0)
+ return false;
+
+ /*
+ * The unmap requires all the CQ's outstanding inflights to be
+ * completed.
+ */
+ r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+ if (r0.field.count > 0)
+ return false;
+
+ for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+ struct dlb2_ldb_port_qid_map *map;
+
+ map = &port->qid_map[i];
+
+ if (map->state != DLB2_QUEUE_UNMAP_IN_PROG &&
+ map->state != DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP)
+ continue;
+
+ dlb2_domain_finish_unmap_port_slot(hw, domain, port, i);
+ }
+
+ return true;
+}
+
+static unsigned int
+dlb2_domain_finish_unmap_qid_procedures(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain)
+{
+ struct dlb2_ldb_port *port;
+ int i;
+
+ if (!domain->configured || domain->num_pending_removals == 0)
+ return 0;
+
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port)
+ dlb2_domain_finish_unmap_port(hw, domain, port);
+ }
+
+ return domain->num_pending_removals;
+}
+
+static void dlb2_domain_finish_map_port(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain,
+ struct dlb2_ldb_port *port)
+{
+ int i;
+
+ for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+ union dlb2_lsp_qid_ldb_infl_cnt r0;
+ struct dlb2_ldb_queue *queue;
+ int qid;
+
+ if (port->qid_map[i].state != DLB2_QUEUE_MAP_IN_PROG)
+ continue;
+
+ qid = port->qid_map[i].qid;
+
+ queue = dlb2_get_ldb_queue_from_id(hw, qid, false, 0);
+
+ if (!queue) {
+ DLB2_HW_ERR(hw,
+ "[%s()] Internal error: unable to find queue %d\n",
+ __func__, qid);
+ continue;
+ }
+
+ r0.val = DLB2_CSR_RD(hw, DLB2_LSP_QID_LDB_INFL_CNT(qid));
+
+ if (r0.field.count)
+ continue;
+
+ /*
+ * Disable the affected CQ, and the CQs already mapped to the
+ * QID, before reading the QID's inflight count a second time.
+ * There is an unlikely race in which the QID may schedule one
+ * more QE after we read an inflight count of 0, and disabling
+ * the CQs guarantees that the race will not occur after a
+ * re-read of the inflight count register.
+ */
+ if (port->enabled)
+ dlb2_ldb_port_cq_disable(hw, port);
+
+ dlb2_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+ r0.val = DLB2_CSR_RD(hw, DLB2_LSP_QID_LDB_INFL_CNT(qid));
+
+ if (r0.field.count) {
+ if (port->enabled)
+ dlb2_ldb_port_cq_enable(hw, port);
+
+ dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+ continue;
+ }
+
+ dlb2_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+ }
+}
+
+static unsigned int
+dlb2_domain_finish_map_qid_procedures(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain)
+{
+ struct dlb2_ldb_port *port;
+ int i;
+
+ if (!domain->configured || domain->num_pending_additions == 0)
+ return 0;
+
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port)
+ dlb2_domain_finish_map_port(hw, domain, port);
+ }
+
+ return domain->num_pending_additions;
+}
+
+static u32 dlb2_ldb_cq_inflight_count(struct dlb2_hw *hw,
+ struct dlb2_ldb_port *port)
+{
+ union dlb2_lsp_cq_ldb_infl_cnt r0;
+
+ r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+
+ return r0.field.count;
+}
+
+static u32 dlb2_ldb_cq_token_count(struct dlb2_hw *hw,
+ struct dlb2_ldb_port *port)
+{
+ union dlb2_lsp_cq_ldb_tkn_cnt r0;
+
+ r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id));
+
+ /*
+ * Account for the initial token count, which is used in order to
+ * provide a CQ with depth less than 8.
+ */
+
+ return r0.field.token_count - port->init_tkn_cnt;
+}
+
+static void __iomem *dlb2_producer_port_addr(struct dlb2_hw *hw,
+ u8 port_id,
+ bool is_ldb)
+{
+ struct dlb2_dev *dlb2_dev;
+ unsigned long size;
+ uintptr_t address;
+
+ dlb2_dev = container_of(hw, struct dlb2_dev, hw);
+
+ address = (uintptr_t)dlb2_dev->hw.func_kva;
+
+ if (is_ldb) {
+ size = DLB2_LDB_PP_STRIDE;
+ address += DLB2_DRV_LDB_PP_BASE + size * port_id;
+ } else {
+ size = DLB2_DIR_PP_STRIDE;
+ address += DLB2_DRV_DIR_PP_BASE + size * port_id;
+ }
+
+ return (void __iomem *)address;
+}
+
+static void dlb2_fence_hcw(void __iomem *addr)
+{
+ /*
+ * To ensure outstanding HCWs reach the device before subsequent device
+ * accesses, fence them.
+ */
+ mb();
+}
+
+static int dlb2_drain_ldb_cq(struct dlb2_hw *hw, struct dlb2_ldb_port *port)
+{
+ u32 infl_cnt, tkn_cnt;
+ unsigned int i;
+
+ infl_cnt = dlb2_ldb_cq_inflight_count(hw, port);
+ tkn_cnt = dlb2_ldb_cq_token_count(hw, port);
+
+ if (infl_cnt || tkn_cnt) {
+ struct dlb2_hcw hcw_mem[8], *hcw;
+ void __iomem *pp_addr;
+
+ pp_addr = dlb2_producer_port_addr(hw, port->id.phys_id, true);
+
+ /* Point hcw to a 64B-aligned location */
+ hcw = (struct dlb2_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+ /*
+ * Program the first HCW for a completion and token return and
+ * the other HCWs as NOOPS
+ */
+
+ memset(hcw, 0, 4 * sizeof(*hcw));
+ hcw->qe_comp = (infl_cnt > 0);
+ hcw->cq_token = (tkn_cnt > 0);
+ hcw->lock_id = tkn_cnt - 1;
+
+ /* Return tokens in the first HCW */
+ iosubmit_cmds512(pp_addr, hcw, 1);
+
+ hcw->cq_token = 0;
+
+ /* Issue remaining completions (if any) */
+ for (i = 1; i < infl_cnt; i++)
+ iosubmit_cmds512(pp_addr, hcw, 1);
+
+ dlb2_fence_hcw(pp_addr);
+ }
+
+ return 0;
+}
+
+static int dlb2_domain_wait_for_ldb_cqs_to_empty(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain)
+{
+ struct dlb2_ldb_port *port;
+ int i;
+
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port) {
+ int i;
+
+ for (i = 0; i < DLB2_MAX_CQ_COMP_CHECK_LOOPS; i++) {
+ if (dlb2_ldb_cq_inflight_count(hw, port) == 0)
+ break;
+ }
+
+ if (i == DLB2_MAX_CQ_COMP_CHECK_LOOPS) {
+ DLB2_HW_ERR(hw,
+ "[%s()] Internal error: failed to flush load-balanced port %d's completions.\n",
+ __func__, port->id.phys_id);
+ return -EFAULT;
+ }
+ }
+ }
+
+ return 0;
+}
+
+static int dlb2_domain_reset_software_state(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain)
+{
+ struct dlb2_dir_pq_pair *tmp_dir_port;
+ struct dlb2_function_resources *rsrcs;
+ struct dlb2_ldb_queue *tmp_ldb_queue;
+ struct dlb2_ldb_port *tmp_ldb_port;
+ struct dlb2_dir_pq_pair *dir_port;
+ struct dlb2_ldb_queue *ldb_queue;
+ struct dlb2_ldb_port *ldb_port;
+ struct list_head *list;
+ int ret, i;
+
+ rsrcs = domain->parent_func;
+
+ /* Move the domain's ldb queues to the function's avail list */
+ list = &domain->used_ldb_queues;
+ DLB2_DOM_LIST_FOR_SAFE(*list, ldb_queue, tmp_ldb_queue) {
+ if (ldb_queue->sn_cfg_valid) {
+ struct dlb2_sn_group *grp;
+
+ grp = &hw->rsrcs.sn_groups[ldb_queue->sn_group];
+
+ dlb2_sn_group_free_slot(grp, ldb_queue->sn_slot);
+ ldb_queue->sn_cfg_valid = false;
+ }
+
+ ldb_queue->owned = false;
+ ldb_queue->num_mappings = 0;
+ ldb_queue->num_pending_additions = 0;
+
+ list_del(&ldb_queue->domain_list);
+ list_add(&ldb_queue->func_list, &rsrcs->avail_ldb_queues);
+ rsrcs->num_avail_ldb_queues++;
+ }
+
+ list = &domain->avail_ldb_queues;
+ DLB2_DOM_LIST_FOR_SAFE(*list, ldb_queue, tmp_ldb_queue) {
+ ldb_queue->owned = false;
+
+ list_del(&ldb_queue->domain_list);
+ list_add(&ldb_queue->func_list, &rsrcs->avail_ldb_queues);
+ rsrcs->num_avail_ldb_queues++;
+ }
+
+ /* Move the domain's ldb ports to the function's avail list */
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ list = &domain->used_ldb_ports[i];
+ DLB2_DOM_LIST_FOR_SAFE(*list, ldb_port, tmp_ldb_port) {
+ int j;
+
+ ldb_port->owned = false;
+ ldb_port->configured = false;
+ ldb_port->num_pending_removals = 0;
+ ldb_port->num_mappings = 0;
+ ldb_port->init_tkn_cnt = 0;
+ for (j = 0; j < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; j++)
+ ldb_port->qid_map[j].state =
+ DLB2_QUEUE_UNMAPPED;
+
+ list_del(&ldb_port->domain_list);
+ list_add(&ldb_port->func_list,
+ &rsrcs->avail_ldb_ports[i]);
+ rsrcs->num_avail_ldb_ports[i]++;
+ }
+
+ list = &domain->avail_ldb_ports[i];
+ DLB2_DOM_LIST_FOR_SAFE(*list, ldb_port, tmp_ldb_port) {
+ ldb_port->owned = false;
+
+ list_del(&ldb_port->domain_list);
+ list_add(&ldb_port->func_list,
+ &rsrcs->avail_ldb_ports[i]);
+ rsrcs->num_avail_ldb_ports[i]++;
+ }
+ }
+
+ /* Move the domain's dir ports to the function's avail list */
+ list = &domain->used_dir_pq_pairs;
+ DLB2_DOM_LIST_FOR_SAFE(*list, dir_port, tmp_dir_port) {
+ dir_port->owned = false;
+ dir_port->port_configured = false;
+ dir_port->init_tkn_cnt = 0;
+
+ list_del(&dir_port->domain_list);
+
+ list_add(&dir_port->func_list, &rsrcs->avail_dir_pq_pairs);
+ rsrcs->num_avail_dir_pq_pairs++;
+ }
+
+ list = &domain->avail_dir_pq_pairs;
+ DLB2_DOM_LIST_FOR_SAFE(*list, dir_port, tmp_dir_port) {
+ dir_port->owned = false;
+
+ list_del(&dir_port->domain_list);
+
+ list_add(&dir_port->func_list, &rsrcs->avail_dir_pq_pairs);
+ rsrcs->num_avail_dir_pq_pairs++;
+ }
+
+ /* Return hist list entries to the function */
+ ret = dlb2_bitmap_set_range(rsrcs->avail_hist_list_entries,
+ domain->hist_list_entry_base,
+ domain->total_hist_list_entries);
+ if (ret) {
+ DLB2_HW_ERR(hw,
+ "[%s()] Internal error: domain hist list base doesn't match the function's bitmap.\n",
+ __func__);
+ return ret;
+ }
+
+ domain->total_hist_list_entries = 0;
+ domain->avail_hist_list_entries = 0;
+ domain->hist_list_entry_base = 0;
+ domain->hist_list_entry_offset = 0;
+
+ rsrcs->num_avail_qed_entries += domain->num_ldb_credits;
+ domain->num_ldb_credits = 0;
+
+ rsrcs->num_avail_dqed_entries += domain->num_dir_credits;
+ domain->num_dir_credits = 0;
+
+ rsrcs->num_avail_aqed_entries += domain->num_avail_aqed_entries;
+ rsrcs->num_avail_aqed_entries += domain->num_used_aqed_entries;
+ domain->num_avail_aqed_entries = 0;
+ domain->num_used_aqed_entries = 0;
+
+ domain->num_pending_removals = 0;
+ domain->num_pending_additions = 0;
+ domain->configured = false;
+ domain->started = false;
+
+ /*
+ * Move the domain out of the used_domains list and back to the
+ * function's avail_domains list.
+ */
+ list_del(&domain->func_list);
+ list_add(&domain->func_list, &rsrcs->avail_domains);
+ rsrcs->num_avail_domains++;
+
+ return 0;
+}
+
+static u32 dlb2_dir_queue_depth(struct dlb2_hw *hw,
+ struct dlb2_dir_pq_pair *queue)
+{
+ union dlb2_lsp_qid_dir_enqueue_cnt r0;
+
+ r0.val = DLB2_CSR_RD(hw,
+ DLB2_LSP_QID_DIR_ENQUEUE_CNT(queue->id.phys_id));
+
+ return r0.field.count;
+}
+
+static bool dlb2_dir_queue_is_empty(struct dlb2_hw *hw,
+ struct dlb2_dir_pq_pair *queue)
+{
+ return dlb2_dir_queue_depth(hw, queue) == 0;
+}
+
+static u32 dlb2_ldb_queue_depth(struct dlb2_hw *hw,
+ struct dlb2_ldb_queue *queue)
+{
+ union dlb2_lsp_qid_aqed_active_cnt r0;
+ union dlb2_lsp_qid_atm_active r1;
+ union dlb2_lsp_qid_ldb_enqueue_cnt r2;
+
+ r0.val = DLB2_CSR_RD(hw,
+ DLB2_LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+ r1.val = DLB2_CSR_RD(hw,
+ DLB2_LSP_QID_ATM_ACTIVE(queue->id.phys_id));
+
+ r2.val = DLB2_CSR_RD(hw,
+ DLB2_LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+
+ return r0.field.count + r1.field.count + r2.field.count;
+}
+
+static bool dlb2_ldb_queue_is_empty(struct dlb2_hw *hw,
+ struct dlb2_ldb_queue *queue)
+{
+ return dlb2_ldb_queue_depth(hw, queue) == 0;
+}
+
+static void __dlb2_domain_reset_ldb_port_registers(struct dlb2_hw *hw,
+ struct dlb2_ldb_port *port)
+{
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_LDB_PP2VAS(port->id.phys_id),
+ DLB2_SYS_LDB_PP2VAS_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_LDB_CQ2VAS(port->id.phys_id),
+ DLB2_CHP_LDB_CQ2VAS_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_LDB_PP2VDEV(port->id.phys_id),
+ DLB2_SYS_LDB_PP2VDEV_RST);
+
+ if (port->id.vdev_owned) {
+ unsigned int offs;
+ u32 virt_id;
+
+ /*
+ * DLB uses producer port address bits 17:12 to determine the
+ * producer port ID. In Scalable IOV mode, PP accesses come
+ * through the PF MMIO window for the physical producer port,
+ * so for translation purposes the virtual and physical port
+ * IDs are equal.
+ */
+ if (hw->virt_mode == DLB2_VIRT_SRIOV)
+ virt_id = port->id.virt_id;
+ else
+ virt_id = port->id.phys_id;
+
+ offs = port->id.vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_VF_LDB_VPP2PP(offs),
+ DLB2_SYS_VF_LDB_VPP2PP_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_VF_LDB_VPP_V(offs),
+ DLB2_SYS_VF_LDB_VPP_V_RST);
+ }
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_LDB_PP_V(port->id.phys_id),
+ DLB2_SYS_LDB_PP_V_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_CQ_LDB_DSBL(port->id.phys_id),
+ DLB2_LSP_CQ_LDB_DSBL_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_LDB_CQ_DEPTH(port->id.phys_id),
+ DLB2_CHP_LDB_CQ_DEPTH_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_CQ_LDB_INFL_LIM(port->id.phys_id),
+ DLB2_LSP_CQ_LDB_INFL_LIM_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_HIST_LIST_LIM(port->id.phys_id),
+ DLB2_CHP_HIST_LIST_LIM_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_HIST_LIST_BASE(port->id.phys_id),
+ DLB2_CHP_HIST_LIST_BASE_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_HIST_LIST_POP_PTR(port->id.phys_id),
+ DLB2_CHP_HIST_LIST_POP_PTR_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_HIST_LIST_PUSH_PTR(port->id.phys_id),
+ DLB2_CHP_HIST_LIST_PUSH_PTR_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+ DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_LDB_CQ_TMR_THRSH(port->id.phys_id),
+ DLB2_CHP_LDB_CQ_TMR_THRSH_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_LDB_CQ_INT_ENB(port->id.phys_id),
+ DLB2_CHP_LDB_CQ_INT_ENB_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_LDB_CQ_ISR(port->id.phys_id),
+ DLB2_SYS_LDB_CQ_ISR_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL(port->id.phys_id),
+ DLB2_LSP_CQ_LDB_TKN_DEPTH_SEL_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+ DLB2_CHP_LDB_CQ_TKN_DEPTH_SEL_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_LDB_CQ_WPTR(port->id.phys_id),
+ DLB2_CHP_LDB_CQ_WPTR_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_CQ_LDB_TKN_CNT(port->id.phys_id),
+ DLB2_LSP_CQ_LDB_TKN_CNT_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_LDB_CQ_ADDR_L(port->id.phys_id),
+ DLB2_SYS_LDB_CQ_ADDR_L_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_LDB_CQ_ADDR_U(port->id.phys_id),
+ DLB2_SYS_LDB_CQ_ADDR_U_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_LDB_CQ_AT(port->id.phys_id),
+ DLB2_SYS_LDB_CQ_AT_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_LDB_CQ_PASID(port->id.phys_id),
+ DLB2_SYS_LDB_CQ_PASID_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_LDB_CQ2VF_PF_RO(port->id.phys_id),
+ DLB2_SYS_LDB_CQ2VF_PF_RO_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_CQ_LDB_TOT_SCH_CNTL(port->id.phys_id),
+ DLB2_LSP_CQ_LDB_TOT_SCH_CNTL_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_CQ_LDB_TOT_SCH_CNTH(port->id.phys_id),
+ DLB2_LSP_CQ_LDB_TOT_SCH_CNTH_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_CQ2QID0(port->id.phys_id),
+ DLB2_LSP_CQ2QID0_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_CQ2QID1(port->id.phys_id),
+ DLB2_LSP_CQ2QID1_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_CQ2PRIOV(port->id.phys_id),
+ DLB2_LSP_CQ2PRIOV_RST);
+}
+
+static void dlb2_domain_reset_ldb_port_registers(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain)
+{
+ struct dlb2_ldb_port *port;
+ int i;
+
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port)
+ __dlb2_domain_reset_ldb_port_registers(hw, port);
+ }
+}
+
+static void
+__dlb2_domain_reset_dir_port_registers(struct dlb2_hw *hw,
+ struct dlb2_dir_pq_pair *port)
+{
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_DIR_CQ2VAS(port->id.phys_id),
+ DLB2_CHP_DIR_CQ2VAS_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_CQ_DIR_DSBL(port->id.phys_id),
+ DLB2_LSP_CQ_DIR_DSBL_RST);
+
+ DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_OPT_CLR, port->id.phys_id);
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_DIR_CQ_DEPTH(port->id.phys_id),
+ DLB2_CHP_DIR_CQ_DEPTH_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+ DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_DIR_CQ_TMR_THRSH(port->id.phys_id),
+ DLB2_CHP_DIR_CQ_TMR_THRSH_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_DIR_CQ_INT_ENB(port->id.phys_id),
+ DLB2_CHP_DIR_CQ_INT_ENB_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_DIR_CQ_ISR(port->id.phys_id),
+ DLB2_SYS_DIR_CQ_ISR_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI(port->id.phys_id),
+ DLB2_LSP_CQ_DIR_TKN_DEPTH_SEL_DSI_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL(port->id.phys_id),
+ DLB2_CHP_DIR_CQ_TKN_DEPTH_SEL_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_DIR_CQ_WPTR(port->id.phys_id),
+ DLB2_CHP_DIR_CQ_WPTR_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id),
+ DLB2_LSP_CQ_DIR_TKN_CNT_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_DIR_CQ_ADDR_L(port->id.phys_id),
+ DLB2_SYS_DIR_CQ_ADDR_L_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_DIR_CQ_ADDR_U(port->id.phys_id),
+ DLB2_SYS_DIR_CQ_ADDR_U_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_DIR_CQ_AT(port->id.phys_id),
+ DLB2_SYS_DIR_CQ_AT_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_DIR_CQ_PASID(port->id.phys_id),
+ DLB2_SYS_DIR_CQ_PASID_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_DIR_CQ_FMT(port->id.phys_id),
+ DLB2_SYS_DIR_CQ_FMT_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_DIR_CQ2VF_PF_RO(port->id.phys_id),
+ DLB2_SYS_DIR_CQ2VF_PF_RO_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_CQ_DIR_TOT_SCH_CNTL(port->id.phys_id),
+ DLB2_LSP_CQ_DIR_TOT_SCH_CNTL_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_CQ_DIR_TOT_SCH_CNTH(port->id.phys_id),
+ DLB2_LSP_CQ_DIR_TOT_SCH_CNTH_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_DIR_PP2VAS(port->id.phys_id),
+ DLB2_SYS_DIR_PP2VAS_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_DIR_CQ2VAS(port->id.phys_id),
+ DLB2_CHP_DIR_CQ2VAS_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_DIR_PP2VDEV(port->id.phys_id),
+ DLB2_SYS_DIR_PP2VDEV_RST);
+
+ if (port->id.vdev_owned) {
+ unsigned int offs;
+ u32 virt_id;
+
+ /*
+ * DLB uses producer port address bits 17:12 to determine the
+ * producer port ID. In Scalable IOV mode, PP accesses come
+ * through the PF MMIO window for the physical producer port,
+ * so for translation purposes the virtual and physical port
+ * IDs are equal.
+ */
+ if (hw->virt_mode == DLB2_VIRT_SRIOV)
+ virt_id = port->id.virt_id;
+ else
+ virt_id = port->id.phys_id;
+
+ offs = port->id.vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_VF_DIR_VPP2PP(offs),
+ DLB2_SYS_VF_DIR_VPP2PP_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_VF_DIR_VPP_V(offs),
+ DLB2_SYS_VF_DIR_VPP_V_RST);
+ }
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_DIR_PP_V(port->id.phys_id),
+ DLB2_SYS_DIR_PP_V_RST);
+}
+
+static void dlb2_domain_reset_dir_port_registers(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain)
+{
+ struct dlb2_dir_pq_pair *port;
+
+ DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port)
+ __dlb2_domain_reset_dir_port_registers(hw, port);
+}
+
+static void dlb2_domain_reset_ldb_queue_registers(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain)
+{
+ struct dlb2_ldb_queue *queue;
+
+ DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue) {
+ unsigned int queue_id = queue->id.phys_id;
+ int i;
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL(queue_id),
+ DLB2_LSP_QID_NALDB_TOT_ENQ_CNTL_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH(queue_id),
+ DLB2_LSP_QID_NALDB_TOT_ENQ_CNTH_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_QID_ATM_TOT_ENQ_CNTL(queue_id),
+ DLB2_LSP_QID_ATM_TOT_ENQ_CNTL_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_QID_ATM_TOT_ENQ_CNTH(queue_id),
+ DLB2_LSP_QID_ATM_TOT_ENQ_CNTH_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_QID_NALDB_MAX_DEPTH(queue_id),
+ DLB2_LSP_QID_NALDB_MAX_DEPTH_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_QID_LDB_INFL_LIM(queue_id),
+ DLB2_LSP_QID_LDB_INFL_LIM_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_QID_AQED_ACTIVE_LIM(queue_id),
+ DLB2_LSP_QID_AQED_ACTIVE_LIM_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_QID_ATM_DEPTH_THRSH(queue_id),
+ DLB2_LSP_QID_ATM_DEPTH_THRSH_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_QID_NALDB_DEPTH_THRSH(queue_id),
+ DLB2_LSP_QID_NALDB_DEPTH_THRSH_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_LDB_QID_ITS(queue_id),
+ DLB2_SYS_LDB_QID_ITS_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_ORD_QID_SN(queue_id),
+ DLB2_CHP_ORD_QID_SN_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_ORD_QID_SN_MAP(queue_id),
+ DLB2_CHP_ORD_QID_SN_MAP_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_LDB_QID_V(queue_id),
+ DLB2_SYS_LDB_QID_V_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_LDB_QID_CFG_V(queue_id),
+ DLB2_SYS_LDB_QID_CFG_V_RST);
+
+ if (queue->sn_cfg_valid) {
+ u32 offs[2];
+
+ offs[0] = DLB2_RO_PIPE_GRP_0_SLT_SHFT(queue->sn_slot);
+ offs[1] = DLB2_RO_PIPE_GRP_1_SLT_SHFT(queue->sn_slot);
+
+ DLB2_CSR_WR(hw,
+ offs[queue->sn_group],
+ DLB2_RO_PIPE_GRP_0_SLT_SHFT_RST);
+ }
+
+ for (i = 0; i < DLB2_LSP_QID2CQIDIX_NUM; i++) {
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_QID2CQIDIX(queue_id, i),
+ DLB2_LSP_QID2CQIDIX_00_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_QID2CQIDIX2(queue_id, i),
+ DLB2_LSP_QID2CQIDIX2_00_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_ATM_QID2CQIDIX(queue_id, i),
+ DLB2_ATM_QID2CQIDIX_00_RST);
+ }
+ }
+}
+
+static void dlb2_domain_reset_dir_queue_registers(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain)
+{
+ struct dlb2_dir_pq_pair *queue;
+
+ DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue) {
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_QID_DIR_MAX_DEPTH(queue->id.phys_id),
+ DLB2_LSP_QID_DIR_MAX_DEPTH_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_QID_DIR_TOT_ENQ_CNTL(queue->id.phys_id),
+ DLB2_LSP_QID_DIR_TOT_ENQ_CNTL_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_QID_DIR_TOT_ENQ_CNTH(queue->id.phys_id),
+ DLB2_LSP_QID_DIR_TOT_ENQ_CNTH_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id),
+ DLB2_LSP_QID_DIR_DEPTH_THRSH_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_DIR_QID_ITS(queue->id.phys_id),
+ DLB2_SYS_DIR_QID_ITS_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_DIR_QID_V(queue->id.phys_id),
+ DLB2_SYS_DIR_QID_V_RST);
+ }
+}
+
+static u32 dlb2_dir_cq_token_count(struct dlb2_hw *hw,
+ struct dlb2_dir_pq_pair *port)
+{
+ union dlb2_lsp_cq_dir_tkn_cnt r0;
+
+ r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ_DIR_TKN_CNT(port->id.phys_id));
+
+ /*
+ * Account for the initial token count, which is used in order to
+ * provide a CQ with depth less than 8.
+ */
+
+ return r0.field.count - port->init_tkn_cnt;
+}
+
+static int dlb2_domain_verify_reset_success(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain)
+{
+ struct dlb2_dir_pq_pair *dir_port;
+ struct dlb2_ldb_port *ldb_port;
+ struct dlb2_ldb_queue *queue;
+ int i;
+
+ /*
+ * Confirm that all the domain's queue's inflight counts and AQED
+ * active counts are 0.
+ */
+ DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue) {
+ if (!dlb2_ldb_queue_is_empty(hw, queue)) {
+ DLB2_HW_ERR(hw,
+ "[%s()] Internal error: failed to empty ldb queue %d\n",
+ __func__, queue->id.phys_id);
+ return -EFAULT;
+ }
+ }
+
+ /* Confirm that all the domain's CQs inflight and token counts are 0. */
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], ldb_port) {
+ if (dlb2_ldb_cq_inflight_count(hw, ldb_port) ||
+ dlb2_ldb_cq_token_count(hw, ldb_port)) {
+ DLB2_HW_ERR(hw,
+ "[%s()] Internal error: failed to empty ldb port %d\n",
+ __func__, ldb_port->id.phys_id);
+ return -EFAULT;
+ }
+ }
+ }
+
+ DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, dir_port) {
+ if (!dlb2_dir_queue_is_empty(hw, dir_port)) {
+ DLB2_HW_ERR(hw,
+ "[%s()] Internal error: failed to empty dir queue %d\n",
+ __func__, dir_port->id.phys_id);
+ return -EFAULT;
+ }
+
+ if (dlb2_dir_cq_token_count(hw, dir_port)) {
+ DLB2_HW_ERR(hw,
+ "[%s()] Internal error: failed to empty dir port %d\n",
+ __func__, dir_port->id.phys_id);
+ return -EFAULT;
+ }
+ }
+
+ return 0;
+}
+
+static void dlb2_domain_reset_registers(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain)
+{
+ dlb2_domain_reset_ldb_port_registers(hw, domain);
+
+ dlb2_domain_reset_dir_port_registers(hw, domain);
+
+ dlb2_domain_reset_ldb_queue_registers(hw, domain);
+
+ dlb2_domain_reset_dir_queue_registers(hw, domain);
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_CFG_LDB_VAS_CRD(domain->id.phys_id),
+ DLB2_CHP_CFG_LDB_VAS_CRD_RST);
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_CFG_DIR_VAS_CRD(domain->id.phys_id),
+ DLB2_CHP_CFG_DIR_VAS_CRD_RST);
+}
+
+static int dlb2_domain_drain_ldb_cqs(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain,
+ bool toggle_port)
+{
+ struct dlb2_ldb_port *port;
+ int ret, i;
+
+ /* If the domain hasn't been started, there's no traffic to drain */
+ if (!domain->started)
+ return 0;
+
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port) {
+ if (toggle_port)
+ dlb2_ldb_port_cq_disable(hw, port);
+
+ ret = dlb2_drain_ldb_cq(hw, port);
+ if (ret < 0)
+ return ret;
+
+ if (toggle_port)
+ dlb2_ldb_port_cq_enable(hw, port);
+ }
+ }
+
+ return 0;
+}
+
+static bool dlb2_domain_mapped_queues_empty(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain)
+{
+ struct dlb2_ldb_queue *queue;
+
+ DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue) {
+ if (queue->num_mappings == 0)
+ continue;
+
+ if (!dlb2_ldb_queue_is_empty(hw, queue))
+ return false;
+ }
+
+ return true;
+}
+
+static int dlb2_domain_drain_mapped_queues(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain)
+{
+ int i, ret;
+
+ /* If the domain hasn't been started, there's no traffic to drain */
+ if (!domain->started)
+ return 0;
+
+ if (domain->num_pending_removals > 0) {
+ DLB2_HW_ERR(hw,
+ "[%s()] Internal error: failed to unmap domain queues\n",
+ __func__);
+ return -EFAULT;
+ }
+
+ for (i = 0; i < DLB2_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+ ret = dlb2_domain_drain_ldb_cqs(hw, domain, true);
+ if (ret < 0)
+ return ret;
+
+ if (dlb2_domain_mapped_queues_empty(hw, domain))
+ break;
+ }
+
+ if (i == DLB2_MAX_QID_EMPTY_CHECK_LOOPS) {
+ DLB2_HW_ERR(hw,
+ "[%s()] Internal error: failed to empty queues\n",
+ __func__);
+ return -EFAULT;
+ }
+
+ /*
+ * Drain the CQs one more time. For the queues to go empty, they would
+ * have scheduled one or more QEs.
+ */
+ ret = dlb2_domain_drain_ldb_cqs(hw, domain, true);
+ if (ret < 0)
+ return ret;
+
+ return 0;
+}
+
+static int dlb2_domain_drain_unmapped_queue(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain,
+ struct dlb2_ldb_queue *queue)
+{
+ struct dlb2_ldb_port *port;
+ int ret, i;
+
+ /* If a domain has LDB queues, it must have LDB ports */
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ if (!list_empty(&domain->used_ldb_ports[i]))
+ break;
+ }
+
+ if (i == DLB2_NUM_COS_DOMAINS) {
+ DLB2_HW_ERR(hw,
+ "[%s()] Internal error: No configured LDB ports\n",
+ __func__);
+ return -EFAULT;
+ }
+
+ port = DLB2_DOM_LIST_HEAD(domain->used_ldb_ports[i], typeof(*port));
+
+ /* If necessary, free up a QID slot in this CQ */
+ if (port->num_mappings == DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+ struct dlb2_ldb_queue *mapped_queue;
+
+ mapped_queue = &hw->rsrcs.ldb_queues[port->qid_map[0].qid];
+
+ ret = dlb2_ldb_port_unmap_qid(hw, port, mapped_queue);
+ if (ret)
+ return ret;
+ }
+
+ ret = dlb2_ldb_port_map_qid_dynamic(hw, port, queue, 0);
+ if (ret)
+ return ret;
+
+ return dlb2_domain_drain_mapped_queues(hw, domain);
+}
+
+static int dlb2_domain_drain_unmapped_queues(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain)
+{
+ struct dlb2_ldb_queue *queue;
+ int ret;
+
+ /* If the domain hasn't been started, there's no traffic to drain */
+ if (!domain->started)
+ return 0;
+
+ /*
+ * Pre-condition: the unattached queue must not have any outstanding
+ * completions. This is ensured by calling dlb2_domain_drain_ldb_cqs()
+ * prior to this in dlb2_domain_drain_mapped_queues().
+ */
+ DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue) {
+ if (queue->num_mappings != 0 ||
+ dlb2_ldb_queue_is_empty(hw, queue))
+ continue;
+
+ ret = dlb2_domain_drain_unmapped_queue(hw, domain, queue);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+static int dlb2_drain_dir_cq(struct dlb2_hw *hw,
+ struct dlb2_dir_pq_pair *port)
+{
+ unsigned int port_id = port->id.phys_id;
+ u32 cnt;
+
+ /* Return any outstanding tokens */
+ cnt = dlb2_dir_cq_token_count(hw, port);
+
+ if (cnt != 0) {
+ struct dlb2_hcw hcw_mem[8], *hcw;
+ void __iomem *pp_addr;
+
+ pp_addr = dlb2_producer_port_addr(hw, port_id, false);
+
+ /* Point hcw to a 64B-aligned location */
+ hcw = (struct dlb2_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+ /*
+ * Program the first HCW for a batch token return and
+ * the rest as NOOPS
+ */
+ memset(hcw, 0, 4 * sizeof(*hcw));
+ hcw->cq_token = 1;
+ hcw->lock_id = cnt - 1;
+
+ iosubmit_cmds512(pp_addr, hcw, 1);
+
+ dlb2_fence_hcw(pp_addr);
+ }
+
+ return 0;
+}
+
+static int dlb2_domain_drain_dir_cqs(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain,
+ bool toggle_port)
+{
+ struct dlb2_dir_pq_pair *port;
+ int ret;
+
+ DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port) {
+ /*
+ * Can't drain a port if it's not configured, and there's
+ * nothing to drain if its queue is unconfigured.
+ */
+ if (!port->port_configured || !port->queue_configured)
+ continue;
+
+ if (toggle_port)
+ dlb2_dir_port_cq_disable(hw, port);
+
+ ret = dlb2_drain_dir_cq(hw, port);
+ if (ret < 0)
+ return ret;
+
+ if (toggle_port)
+ dlb2_dir_port_cq_enable(hw, port);
+ }
+
+ return 0;
+}
+
+static bool dlb2_domain_dir_queues_empty(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain)
+{
+ struct dlb2_dir_pq_pair *queue;
+
+ DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue) {
+ if (!dlb2_dir_queue_is_empty(hw, queue))
+ return false;
+ }
+
+ return true;
+}
+
+static int dlb2_domain_drain_dir_queues(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain)
+{
+ int i, ret;
+
+ /* If the domain hasn't been started, there's no traffic to drain */
+ if (!domain->started)
+ return 0;
+
+ for (i = 0; i < DLB2_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+ ret = dlb2_domain_drain_dir_cqs(hw, domain, true);
+ if (ret < 0)
+ return ret;
+
+ if (dlb2_domain_dir_queues_empty(hw, domain))
+ break;
+ }
+
+ if (i == DLB2_MAX_QID_EMPTY_CHECK_LOOPS) {
+ DLB2_HW_ERR(hw,
+ "[%s()] Internal error: failed to empty queues\n",
+ __func__);
+ return -EFAULT;
+ }
+
+ /*
+ * Drain the CQs one more time. For the queues to go empty, they would
+ * have scheduled one or more QEs.
+ */
+ ret = dlb2_domain_drain_dir_cqs(hw, domain, true);
+ if (ret < 0)
+ return ret;
+
+ return 0;
+}
+
+static void
+dlb2_domain_disable_dir_producer_ports(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain)
+{
+ struct dlb2_dir_pq_pair *port;
+ union dlb2_sys_dir_pp_v r1;
+
+ r1.field.pp_v = 0;
+
+ DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port)
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_DIR_PP_V(port->id.phys_id),
+ r1.val);
+}
+
+static void
+dlb2_domain_disable_ldb_producer_ports(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain)
+{
+ union dlb2_sys_ldb_pp_v r1;
+ struct dlb2_ldb_port *port;
+ int i;
+
+ r1.field.pp_v = 0;
+
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port)
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_LDB_PP_V(port->id.phys_id),
+ r1.val);
+ }
+}
+
+static void dlb2_domain_disable_dir_vpps(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain,
+ unsigned int vdev_id)
+{
+ union dlb2_sys_vf_dir_vpp_v r1;
+ struct dlb2_dir_pq_pair *port;
+
+ r1.field.vpp_v = 0;
+
+ DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port) {
+ unsigned int offs;
+ u32 virt_id;
+
+ if (hw->virt_mode == DLB2_VIRT_SRIOV)
+ virt_id = port->id.virt_id;
+ else
+ virt_id = port->id.phys_id;
+
+ offs = vdev_id * DLB2_MAX_NUM_DIR_PORTS + virt_id;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VPP_V(offs), r1.val);
+ }
+}
+
+static void dlb2_domain_disable_ldb_vpps(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain,
+ unsigned int vdev_id)
+{
+ union dlb2_sys_vf_ldb_vpp_v r1;
+ struct dlb2_ldb_port *port;
+ int i;
+
+ r1.field.vpp_v = 0;
+
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port) {
+ unsigned int offs;
+ u32 virt_id;
+
+ if (hw->virt_mode == DLB2_VIRT_SRIOV)
+ virt_id = port->id.virt_id;
+ else
+ virt_id = port->id.phys_id;
+
+ offs = vdev_id * DLB2_MAX_NUM_LDB_PORTS + virt_id;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VPP_V(offs), r1.val);
+ }
+ }
+}
+
+static void dlb2_domain_disable_ldb_seq_checks(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain)
+{
+ union dlb2_chp_sn_chk_enbl r1;
+ struct dlb2_ldb_port *port;
+ int i;
+
+ r1.field.en = 0;
+
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port)
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_SN_CHK_ENBL(port->id.phys_id),
+ r1.val);
+ }
+}
+
+static void
+dlb2_domain_disable_ldb_port_interrupts(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain)
+{
+ union dlb2_chp_ldb_cq_int_enb r0 = { {0} };
+ union dlb2_chp_ldb_cq_wd_enb r1 = { {0} };
+ struct dlb2_ldb_port *port;
+ int i;
+
+ r0.field.en_tim = 0;
+ r0.field.en_depth = 0;
+
+ r1.field.wd_enable = 0;
+
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port) {
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_LDB_CQ_INT_ENB(port->id.phys_id),
+ r0.val);
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_LDB_CQ_WD_ENB(port->id.phys_id),
+ r1.val);
+ }
+ }
+}
+
+static void
+dlb2_domain_disable_dir_port_interrupts(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain)
+{
+ union dlb2_chp_dir_cq_int_enb r0 = { {0} };
+ union dlb2_chp_dir_cq_wd_enb r1 = { {0} };
+ struct dlb2_dir_pq_pair *port;
+
+ r0.field.en_tim = 0;
+ r0.field.en_depth = 0;
+
+ r1.field.wd_enable = 0;
+
+ DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port) {
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_DIR_CQ_INT_ENB(port->id.phys_id),
+ r0.val);
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_DIR_CQ_WD_ENB(port->id.phys_id),
+ r1.val);
+ }
+}
+
+static void
+dlb2_domain_disable_ldb_queue_write_perms(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain)
+{
+ int domain_offset = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES;
+ struct dlb2_ldb_queue *queue;
+
+ DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue) {
+ union dlb2_sys_ldb_vasqid_v r0 = { {0} };
+ union dlb2_sys_ldb_qid2vqid r1 = { {0} };
+ union dlb2_sys_vf_ldb_vqid_v r2 = { {0} };
+ union dlb2_sys_vf_ldb_vqid2qid r3 = { {0} };
+ int idx;
+
+ idx = domain_offset + queue->id.phys_id;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(idx), r0.val);
+
+ if (queue->id.vdev_owned) {
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_LDB_QID2VQID(queue->id.phys_id),
+ r1.val);
+
+ idx = queue->id.vdev_id * DLB2_MAX_NUM_LDB_QUEUES +
+ queue->id.virt_id;
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_VF_LDB_VQID_V(idx),
+ r2.val);
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_VF_LDB_VQID2QID(idx),
+ r3.val);
+ }
+ }
+}
+
+static void
+dlb2_domain_disable_dir_queue_write_perms(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain)
+{
+ int domain_offset = domain->id.phys_id * DLB2_MAX_NUM_DIR_PORTS;
+ struct dlb2_dir_pq_pair *queue;
+
+ DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, queue) {
+ union dlb2_sys_dir_vasqid_v r0 = { {0} };
+ union dlb2_sys_vf_dir_vqid_v r1 = { {0} };
+ union dlb2_sys_vf_dir_vqid2qid r2 = { {0} };
+ int idx;
+
+ idx = domain_offset + queue->id.phys_id;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(idx), r0.val);
+
+ if (queue->id.vdev_owned) {
+ idx = queue->id.vdev_id * DLB2_MAX_NUM_DIR_PORTS +
+ queue->id.virt_id;
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_VF_DIR_VQID_V(idx),
+ r1.val);
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_VF_DIR_VQID2QID(idx),
+ r2.val);
+ }
+ }
+}
+
+static void dlb2_domain_disable_dir_cqs(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain)
+{
+ struct dlb2_dir_pq_pair *port;
+
+ DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port) {
+ port->enabled = false;
+
+ dlb2_dir_port_cq_disable(hw, port);
+ }
+}
+
+static void dlb2_domain_disable_ldb_cqs(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain)
+{
+ struct dlb2_ldb_port *port;
+ int i;
+
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port) {
+ port->enabled = false;
+
+ dlb2_ldb_port_cq_disable(hw, port);
+ }
+ }
+}
+
+static void dlb2_domain_enable_ldb_cqs(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain)
+{
+ struct dlb2_ldb_port *port;
+ int i;
+
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port) {
+ port->enabled = true;
+
+ dlb2_ldb_port_cq_enable(hw, port);
+ }
+ }
+}
+
+static void dlb2_log_reset_domain(struct dlb2_hw *hw,
+ u32 domain_id,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ DLB2_HW_DBG(hw, "DLB2 reset domain:\n");
+ if (vdev_req)
+ DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+ DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+}
+
+/**
+ * dlb2_reset_domain() - Reset a DLB scheduling domain and its associated
+ * hardware resources.
+ * @hw: Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Note: User software *must* stop sending to this domain's producer ports
+ * before invoking this function, otherwise undefined behavior will result.
+ *
+ * Return: returns < 0 on error, 0 otherwise.
+ */
+int dlb2_reset_domain(struct dlb2_hw *hw,
+ u32 domain_id,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_hw_domain *domain;
+ int ret;
+
+ dlb2_log_reset_domain(hw, domain_id, vdev_req, vdev_id);
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+ if (!domain || !domain->configured)
+ return -EINVAL;
+
+ /* Disable VPPs */
+ if (vdev_req) {
+ dlb2_domain_disable_dir_vpps(hw, domain, vdev_id);
+
+ dlb2_domain_disable_ldb_vpps(hw, domain, vdev_id);
+ }
+
+ /* Disable CQ interrupts */
+ dlb2_domain_disable_dir_port_interrupts(hw, domain);
+
+ dlb2_domain_disable_ldb_port_interrupts(hw, domain);
+
+ /*
+ * For each queue owned by this domain, disable its write permissions to
+ * cause any traffic sent to it to be dropped. Well-behaved software
+ * should not be sending QEs at this point.
+ */
+ dlb2_domain_disable_dir_queue_write_perms(hw, domain);
+
+ dlb2_domain_disable_ldb_queue_write_perms(hw, domain);
+
+ /* Turn off completion tracking on all the domain's PPs. */
+ dlb2_domain_disable_ldb_seq_checks(hw, domain);
+
+ /*
+ * Disable the LDB CQs and drain them in order to complete the map and
+ * unmap procedures, which require zero CQ inflights and zero QID
+ * inflights respectively.
+ */
+ dlb2_domain_disable_ldb_cqs(hw, domain);
+
+ ret = dlb2_domain_drain_ldb_cqs(hw, domain, false);
+ if (ret < 0)
+ return ret;
+
+ ret = dlb2_domain_wait_for_ldb_cqs_to_empty(hw, domain);
+ if (ret < 0)
+ return ret;
+
+ ret = dlb2_domain_finish_unmap_qid_procedures(hw, domain);
+ if (ret < 0)
+ return ret;
+
+ ret = dlb2_domain_finish_map_qid_procedures(hw, domain);
+ if (ret < 0)
+ return ret;
+
+ /* Re-enable the CQs in order to drain the mapped queues. */
+ dlb2_domain_enable_ldb_cqs(hw, domain);
+
+ ret = dlb2_domain_drain_mapped_queues(hw, domain);
+ if (ret < 0)
+ return ret;
+
+ ret = dlb2_domain_drain_unmapped_queues(hw, domain);
+ if (ret < 0)
+ return ret;
+
+ /* Done draining LDB QEs, so disable the CQs. */
+ dlb2_domain_disable_ldb_cqs(hw, domain);
+
+ dlb2_domain_drain_dir_queues(hw, domain);
+
+ /* Done draining DIR QEs, so disable the CQs. */
+ dlb2_domain_disable_dir_cqs(hw, domain);
+
+ /* Disable PPs */
+ dlb2_domain_disable_dir_producer_ports(hw, domain);
+
+ dlb2_domain_disable_ldb_producer_ports(hw, domain);
+
+ ret = dlb2_domain_verify_reset_success(hw, domain);
+ if (ret)
+ return ret;
+
+ /* Reset the QID and port state. */
+ dlb2_domain_reset_registers(hw, domain);
+
+ /* Hardware reset complete. Reset the domain's software state */
+ ret = dlb2_domain_reset_software_state(hw, domain);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
struct dlb2_get_num_resources_args *arg,
bool vdev_req,
diff --git a/drivers/misc/dlb2/dlb2_resource.h b/drivers/misc/dlb2/dlb2_resource.h
index b23676df22a6..7dc50865681b 100644
--- a/drivers/misc/dlb2/dlb2_resource.h
+++ b/drivers/misc/dlb2/dlb2_resource.h
@@ -7,6 +7,8 @@
#include <linux/types.h>
+#include <uapi/linux/dlb2_user.h>
+
#include "dlb2_hw_types.h"
/**
@@ -35,6 +37,69 @@ int dlb2_resource_init(struct dlb2_hw *hw);
void dlb2_resource_free(struct dlb2_hw *hw);
/**
+ * dlb2_hw_create_sched_domain() - create a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @args: scheduling domain creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a scheduling domain containing the resources specified
+ * in args. The individual resources (queues, ports, credits) can be configured
+ * after creating a scheduling domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the domain ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, or the requested domain name
+ * is already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
+ struct dlb2_create_sched_domain_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_request,
+ unsigned int vdev_id);
+
+/**
+ * dlb2_reset_domain() - reset a scheduling domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function resets and frees a DLB 2.0 scheduling domain and its associated
+ * resources.
+ *
+ * Pre-condition: the driver must ensure software has stopped sending QEs
+ * through this domain's producer ports before invoking this function, or
+ * undefined behavior will result.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -1 otherwise.
+ *
+ * EINVAL - Invalid domain ID, or the domain is not configured.
+ * EFAULT - Internal error. (Possibly caused if software is the pre-condition
+ * is not met.)
+ * ETIMEDOUT - Hardware component didn't reset in the expected time.
+ */
+int dlb2_reset_domain(struct dlb2_hw *hw,
+ u32 domain_id,
+ bool vdev_request,
+ unsigned int vdev_id);
+
+/**
* dlb2_hw_get_num_resources() - query the PCI function's available resources
* @hw: dlb2_hw handle for a particular device.
* @arg: pointer to resource counts.
@@ -65,4 +130,13 @@ int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
*/
void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw);
+enum dlb2_virt_mode {
+ DLB2_VIRT_NONE,
+ DLB2_VIRT_SRIOV,
+ DLB2_VIRT_SIOV,
+
+ /* NUM_DLB2_VIRT_MODES must be last */
+ NUM_DLB2_VIRT_MODES,
+};
+
#endif /* __DLB2_RESOURCE_H */
diff --git a/include/uapi/linux/dlb2_user.h b/include/uapi/linux/dlb2_user.h
index 49a8a39f79a2..02d380b51c2c 100644
--- a/include/uapi/linux/dlb2_user.h
+++ b/include/uapi/linux/dlb2_user.h
@@ -7,6 +7,92 @@
#include <linux/types.h>
+enum dlb2_error {
+ DLB2_ST_SUCCESS = 0,
+ DLB2_ST_NAME_EXISTS,
+ DLB2_ST_DOMAIN_UNAVAILABLE,
+ DLB2_ST_LDB_PORTS_UNAVAILABLE,
+ DLB2_ST_DIR_PORTS_UNAVAILABLE,
+ DLB2_ST_LDB_QUEUES_UNAVAILABLE,
+ DLB2_ST_LDB_CREDITS_UNAVAILABLE,
+ DLB2_ST_DIR_CREDITS_UNAVAILABLE,
+ DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE,
+ DLB2_ST_INVALID_DOMAIN_ID,
+ DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION,
+ DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE,
+ DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE,
+ DLB2_ST_INVALID_LDB_QUEUE_ID,
+ DLB2_ST_INVALID_CQ_DEPTH,
+ DLB2_ST_INVALID_CQ_VIRT_ADDR,
+ DLB2_ST_INVALID_PORT_ID,
+ DLB2_ST_INVALID_QID,
+ DLB2_ST_INVALID_PRIORITY,
+ DLB2_ST_NO_QID_SLOTS_AVAILABLE,
+ DLB2_ST_INVALID_DIR_QUEUE_ID,
+ DLB2_ST_DIR_QUEUES_UNAVAILABLE,
+ DLB2_ST_DOMAIN_NOT_CONFIGURED,
+ DLB2_ST_INTERNAL_ERROR,
+ DLB2_ST_DOMAIN_IN_USE,
+ DLB2_ST_DOMAIN_NOT_FOUND,
+ DLB2_ST_QUEUE_NOT_FOUND,
+ DLB2_ST_DOMAIN_STARTED,
+ DLB2_ST_DOMAIN_NOT_STARTED,
+ DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES,
+ DLB2_ST_DOMAIN_RESET_FAILED,
+ DLB2_ST_MBOX_ERROR,
+ DLB2_ST_INVALID_HIST_LIST_DEPTH,
+ DLB2_ST_NO_MEMORY,
+ DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL,
+ DLB2_ST_INVALID_COS_ID,
+ DLB2_ST_INVALID_SMON_ID,
+ DLB2_ST_INVALID_SMON_MODE,
+ DLB2_ST_INVALID_SMON_COMP_MODE,
+ DLB2_ST_INVALID_SMON_CAP_MODE,
+};
+
+static const char dlb2_error_strings[][128] = {
+ "DLB2_ST_SUCCESS",
+ "DLB2_ST_NAME_EXISTS",
+ "DLB2_ST_DOMAIN_UNAVAILABLE",
+ "DLB2_ST_LDB_PORTS_UNAVAILABLE",
+ "DLB2_ST_DIR_PORTS_UNAVAILABLE",
+ "DLB2_ST_LDB_QUEUES_UNAVAILABLE",
+ "DLB2_ST_LDB_CREDITS_UNAVAILABLE",
+ "DLB2_ST_DIR_CREDITS_UNAVAILABLE",
+ "DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE",
+ "DLB2_ST_INVALID_DOMAIN_ID",
+ "DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION",
+ "DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE",
+ "DLB2_ST_HIST_LIST_ENTRIES_UNAVAILABLE",
+ "DLB2_ST_INVALID_LDB_QUEUE_ID",
+ "DLB2_ST_INVALID_CQ_DEPTH",
+ "DLB2_ST_INVALID_CQ_VIRT_ADDR",
+ "DLB2_ST_INVALID_PORT_ID",
+ "DLB2_ST_INVALID_QID",
+ "DLB2_ST_INVALID_PRIORITY",
+ "DLB2_ST_NO_QID_SLOTS_AVAILABLE",
+ "DLB2_ST_INVALID_DIR_QUEUE_ID",
+ "DLB2_ST_DIR_QUEUES_UNAVAILABLE",
+ "DLB2_ST_DOMAIN_NOT_CONFIGURED",
+ "DLB2_ST_INTERNAL_ERROR",
+ "DLB2_ST_DOMAIN_IN_USE",
+ "DLB2_ST_DOMAIN_NOT_FOUND",
+ "DLB2_ST_QUEUE_NOT_FOUND",
+ "DLB2_ST_DOMAIN_STARTED",
+ "DLB2_ST_DOMAIN_NOT_STARTED",
+ "DLB2_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES",
+ "DLB2_ST_DOMAIN_RESET_FAILED",
+ "DLB2_ST_MBOX_ERROR",
+ "DLB2_ST_INVALID_HIST_LIST_DEPTH",
+ "DLB2_ST_NO_MEMORY",
+ "DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL",
+ "DLB2_ST_INVALID_COS_ID",
+ "DLB2_ST_INVALID_SMON_ID",
+ "DLB2_ST_INVALID_SMON_MODE",
+ "DLB2_ST_INVALID_SMON_COMP_MODE",
+ "DLB2_ST_INVALID_SMON_CAP_MODE",
+};
+
struct dlb2_cmd_response {
__u32 status; /* Interpret using enum dlb2_error */
__u32 id;
--
2.13.6
When a CPU enqueues a queue entry (QE) to DLB 2.0, the QE entry is sent to
a DLB 2.0 queue. These queues hold queue entries (QEs) that have not yet
been scheduled to a destination port. The queue's depth is the number of
QEs residing in a queue.
Each queue supports multiple priority levels, and while a directed queue
has a 1:1 mapping with a directed port, load-balanced queues can be
configured with a set of load-balanced ports that software desires the
queue's QEs to be scheduled to.
Signed-off-by: Gage Eads <[email protected]>
Reviewed-by: Magnus Karlsson <[email protected]>
---
drivers/misc/dlb2/dlb2_ioctl.c | 61 ++++
drivers/misc/dlb2/dlb2_ioctl.h | 2 +
drivers/misc/dlb2/dlb2_main.c | 2 +
drivers/misc/dlb2/dlb2_main.h | 16 +
drivers/misc/dlb2/dlb2_pf_ops.c | 40 +++
drivers/misc/dlb2/dlb2_resource.c | 732 ++++++++++++++++++++++++++++++++++++++
drivers/misc/dlb2/dlb2_resource.h | 125 +++++++
include/uapi/linux/dlb2_user.h | 138 +++++++
8 files changed, 1116 insertions(+)
diff --git a/drivers/misc/dlb2/dlb2_ioctl.c b/drivers/misc/dlb2/dlb2_ioctl.c
index 56ebd9387e31..f787fd19689f 100644
--- a/drivers/misc/dlb2/dlb2_ioctl.c
+++ b/drivers/misc/dlb2/dlb2_ioctl.c
@@ -9,6 +9,67 @@
#include "dlb2_ioctl.h"
#include "dlb2_main.h"
+/*
+ * The DLB domain ioctl callback template minimizes replication of boilerplate
+ * code to copy arguments, acquire and release the resource lock, and execute
+ * the command. The arguments and response structure name should have the
+ * format dlb2_<lower_name>_args.
+ */
+#define DLB2_DOMAIN_IOCTL_CALLBACK_TEMPLATE(lower_name) \
+static int dlb2_domain_ioctl_##lower_name(struct dlb2_dev *dev, \
+ struct dlb2_domain *domain, \
+ unsigned long user_arg) \
+{ \
+ struct dlb2_cmd_response response = {0}; \
+ struct dlb2_##lower_name##_args arg; \
+ int ret; \
+ \
+ if (copy_from_user(&arg, (void __user *)user_arg, sizeof(arg))) \
+ return -EFAULT; \
+ \
+ mutex_lock(&dev->resource_mutex); \
+ \
+ ret = dev->ops->lower_name(&dev->hw, \
+ domain->id, \
+ &arg, \
+ &response); \
+ \
+ mutex_unlock(&dev->resource_mutex); \
+ \
+ BUILD_BUG_ON(offsetof(typeof(arg), response) != 0); \
+ \
+ if (copy_to_user((void __user *)user_arg, \
+ &response, \
+ sizeof(response))) \
+ return -EFAULT; \
+ \
+ return ret; \
+}
+
+DLB2_DOMAIN_IOCTL_CALLBACK_TEMPLATE(create_ldb_queue)
+DLB2_DOMAIN_IOCTL_CALLBACK_TEMPLATE(create_dir_queue)
+DLB2_DOMAIN_IOCTL_CALLBACK_TEMPLATE(get_ldb_queue_depth)
+DLB2_DOMAIN_IOCTL_CALLBACK_TEMPLATE(get_dir_queue_depth)
+
+long dlb2_domain_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
+{
+ struct dlb2_domain *dom = f->private_data;
+ struct dlb2_dev *dev = dom->dlb2_dev;
+
+ switch (cmd) {
+ case DLB2_IOC_CREATE_LDB_QUEUE:
+ return dlb2_domain_ioctl_create_ldb_queue(dev, dom, arg);
+ case DLB2_IOC_CREATE_DIR_QUEUE:
+ return dlb2_domain_ioctl_create_dir_queue(dev, dom, arg);
+ case DLB2_IOC_GET_LDB_QUEUE_DEPTH:
+ return dlb2_domain_ioctl_get_ldb_queue_depth(dev, dom, arg);
+ case DLB2_IOC_GET_DIR_QUEUE_DEPTH:
+ return dlb2_domain_ioctl_get_dir_queue_depth(dev, dom, arg);
+ default:
+ return -ENOTTY;
+ }
+}
+
/* [7:0]: device revision, [15:8]: device version */
#define DLB2_SET_DEVICE_VERSION(ver, rev) (((ver) << 8) | (rev))
diff --git a/drivers/misc/dlb2/dlb2_ioctl.h b/drivers/misc/dlb2/dlb2_ioctl.h
index 367c93b98414..548cff477cf5 100644
--- a/drivers/misc/dlb2/dlb2_ioctl.h
+++ b/drivers/misc/dlb2/dlb2_ioctl.h
@@ -9,4 +9,6 @@
long dlb2_ioctl(struct file *f, unsigned int cmd, unsigned long arg);
+long dlb2_domain_ioctl(struct file *f, unsigned int cmd, unsigned long arg);
+
#endif /* __DLB2_IOCTL_H */
diff --git a/drivers/misc/dlb2/dlb2_main.c b/drivers/misc/dlb2/dlb2_main.c
index 17a26e6856d1..b331e096d206 100644
--- a/drivers/misc/dlb2/dlb2_main.c
+++ b/drivers/misc/dlb2/dlb2_main.c
@@ -154,6 +154,8 @@ static int dlb2_domain_close(struct inode *i, struct file *f)
const struct file_operations dlb2_domain_fops = {
.owner = THIS_MODULE,
.release = dlb2_domain_close,
+ .unlocked_ioctl = dlb2_domain_ioctl,
+ .compat_ioctl = compat_ptr_ioctl,
};
/**********************************/
diff --git a/drivers/misc/dlb2/dlb2_main.h b/drivers/misc/dlb2/dlb2_main.h
index 86344f223649..41a3254a78ac 100644
--- a/drivers/misc/dlb2/dlb2_main.h
+++ b/drivers/misc/dlb2/dlb2_main.h
@@ -60,9 +60,25 @@ struct dlb2_device_ops {
int (*create_sched_domain)(struct dlb2_hw *hw,
struct dlb2_create_sched_domain_args *args,
struct dlb2_cmd_response *resp);
+ int (*create_ldb_queue)(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_create_ldb_queue_args *args,
+ struct dlb2_cmd_response *resp);
+ int (*create_dir_queue)(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_create_dir_queue_args *args,
+ struct dlb2_cmd_response *resp);
int (*get_num_resources)(struct dlb2_hw *hw,
struct dlb2_get_num_resources_args *args);
int (*reset_domain)(struct dlb2_hw *hw, u32 domain_id);
+ int (*get_ldb_queue_depth)(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_get_ldb_queue_depth_args *args,
+ struct dlb2_cmd_response *resp);
+ int (*get_dir_queue_depth)(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_get_dir_queue_depth_args *args,
+ struct dlb2_cmd_response *resp);
void (*init_hardware)(struct dlb2_dev *dev);
};
diff --git a/drivers/misc/dlb2/dlb2_pf_ops.c b/drivers/misc/dlb2/dlb2_pf_ops.c
index 6ca06406b0f2..89c0e66b3c6b 100644
--- a/drivers/misc/dlb2/dlb2_pf_ops.c
+++ b/drivers/misc/dlb2/dlb2_pf_ops.c
@@ -216,6 +216,24 @@ dlb2_pf_create_sched_domain(struct dlb2_hw *hw,
}
static int
+dlb2_pf_create_ldb_queue(struct dlb2_hw *hw,
+ u32 id,
+ struct dlb2_create_ldb_queue_args *args,
+ struct dlb2_cmd_response *resp)
+{
+ return dlb2_hw_create_ldb_queue(hw, id, args, resp, false, 0);
+}
+
+static int
+dlb2_pf_create_dir_queue(struct dlb2_hw *hw,
+ u32 id,
+ struct dlb2_create_dir_queue_args *args,
+ struct dlb2_cmd_response *resp)
+{
+ return dlb2_hw_create_dir_queue(hw, id, args, resp, false, 0);
+}
+
+static int
dlb2_pf_get_num_resources(struct dlb2_hw *hw,
struct dlb2_get_num_resources_args *args)
{
@@ -228,6 +246,24 @@ dlb2_pf_reset_domain(struct dlb2_hw *hw, u32 id)
return dlb2_reset_domain(hw, id, false, 0);
}
+static int
+dlb2_pf_get_ldb_queue_depth(struct dlb2_hw *hw,
+ u32 id,
+ struct dlb2_get_ldb_queue_depth_args *args,
+ struct dlb2_cmd_response *resp)
+{
+ return dlb2_hw_get_ldb_queue_depth(hw, id, args, resp, false, 0);
+}
+
+static int
+dlb2_pf_get_dir_queue_depth(struct dlb2_hw *hw,
+ u32 id,
+ struct dlb2_get_dir_queue_depth_args *args,
+ struct dlb2_cmd_response *resp)
+{
+ return dlb2_hw_get_dir_queue_depth(hw, id, args, resp, false, 0);
+}
+
/********************************/
/****** DLB2 PF Device Ops ******/
/********************************/
@@ -246,7 +282,11 @@ struct dlb2_device_ops dlb2_pf_ops = {
.enable_pm = dlb2_pf_enable_pm,
.wait_for_device_ready = dlb2_pf_wait_for_device_ready,
.create_sched_domain = dlb2_pf_create_sched_domain,
+ .create_ldb_queue = dlb2_pf_create_ldb_queue,
+ .create_dir_queue = dlb2_pf_create_dir_queue,
.get_num_resources = dlb2_pf_get_num_resources,
.reset_domain = dlb2_pf_reset_domain,
+ .get_ldb_queue_depth = dlb2_pf_get_ldb_queue_depth,
+ .get_dir_queue_depth = dlb2_pf_get_dir_queue_depth,
.init_hardware = dlb2_pf_init_hardware,
};
diff --git a/drivers/misc/dlb2/dlb2_resource.c b/drivers/misc/dlb2/dlb2_resource.c
index 2cc744cea2d5..2531c6508994 100644
--- a/drivers/misc/dlb2/dlb2_resource.c
+++ b/drivers/misc/dlb2/dlb2_resource.c
@@ -237,6 +237,24 @@ static struct dlb2_hw_domain *dlb2_get_domain_from_id(struct dlb2_hw *hw,
return NULL;
}
+static struct dlb2_dir_pq_pair *
+dlb2_get_domain_used_dir_pq(u32 id,
+ bool vdev_req,
+ struct dlb2_hw_domain *domain)
+{
+ struct dlb2_dir_pq_pair *port;
+
+ if (id >= DLB2_MAX_NUM_DIR_PORTS)
+ return NULL;
+
+ DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port)
+ if ((!vdev_req && port->id.phys_id == id) ||
+ (vdev_req && port->id.virt_id == id))
+ return port;
+
+ return NULL;
+}
+
static struct dlb2_ldb_queue *
dlb2_get_ldb_queue_from_id(struct dlb2_hw *hw,
u32 id,
@@ -268,6 +286,24 @@ dlb2_get_ldb_queue_from_id(struct dlb2_hw *hw,
return NULL;
}
+static struct dlb2_ldb_queue *
+dlb2_get_domain_ldb_queue(u32 id,
+ bool vdev_req,
+ struct dlb2_hw_domain *domain)
+{
+ struct dlb2_ldb_queue *queue;
+
+ if (id >= DLB2_MAX_NUM_LDB_QUEUES)
+ return NULL;
+
+ DLB2_DOM_LIST_FOR(domain->used_ldb_queues, queue)
+ if ((!vdev_req && queue->id.phys_id == id) ||
+ (vdev_req && queue->id.virt_id == id))
+ return queue;
+
+ return NULL;
+}
+
static int dlb2_attach_ldb_queues(struct dlb2_hw *hw,
struct dlb2_function_resources *rsrcs,
struct dlb2_hw_domain *domain,
@@ -707,6 +743,352 @@ dlb2_verify_create_sched_dom_args(struct dlb2_function_resources *rsrcs,
return 0;
}
+static int
+dlb2_verify_create_ldb_queue_args(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_create_ldb_queue_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_hw_domain *domain;
+ int i;
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+ if (!domain) {
+ resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+ return -EINVAL;
+ }
+
+ if (!domain->configured) {
+ resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+ return -EINVAL;
+ }
+
+ if (domain->started) {
+ resp->status = DLB2_ST_DOMAIN_STARTED;
+ return -EINVAL;
+ }
+
+ if (list_empty(&domain->avail_ldb_queues)) {
+ resp->status = DLB2_ST_LDB_QUEUES_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ if (args->num_sequence_numbers) {
+ for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+ struct dlb2_sn_group *group = &hw->rsrcs.sn_groups[i];
+
+ if (group->sequence_numbers_per_queue ==
+ args->num_sequence_numbers &&
+ !dlb2_sn_group_full(group))
+ break;
+ }
+
+ if (i == DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS) {
+ resp->status = DLB2_ST_SEQUENCE_NUMBERS_UNAVAILABLE;
+ return -EINVAL;
+ }
+ }
+
+ if (args->num_qid_inflights > 4096) {
+ resp->status = DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION;
+ return -EINVAL;
+ }
+
+ /* Inflights must be <= number of sequence numbers if ordered */
+ if (args->num_sequence_numbers != 0 &&
+ args->num_qid_inflights > args->num_sequence_numbers) {
+ resp->status = DLB2_ST_INVALID_QID_INFLIGHT_ALLOCATION;
+ return -EINVAL;
+ }
+
+ if (domain->num_avail_aqed_entries < args->num_atomic_inflights) {
+ resp->status = DLB2_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ if (args->num_atomic_inflights &&
+ args->lock_id_comp_level != 0 &&
+ args->lock_id_comp_level != 64 &&
+ args->lock_id_comp_level != 128 &&
+ args->lock_id_comp_level != 256 &&
+ args->lock_id_comp_level != 512 &&
+ args->lock_id_comp_level != 1024 &&
+ args->lock_id_comp_level != 2048 &&
+ args->lock_id_comp_level != 4096 &&
+ args->lock_id_comp_level != 65536) {
+ resp->status = DLB2_ST_INVALID_LOCK_ID_COMP_LEVEL;
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int
+dlb2_verify_create_dir_queue_args(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_create_dir_queue_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_hw_domain *domain;
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+ if (!domain) {
+ resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+ return -EINVAL;
+ }
+
+ if (!domain->configured) {
+ resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+ return -EINVAL;
+ }
+
+ if (domain->started) {
+ resp->status = DLB2_ST_DOMAIN_STARTED;
+ return -EINVAL;
+ }
+
+ /*
+ * If the user claims the port is already configured, validate the port
+ * ID, its domain, and whether the port is configured.
+ */
+ if (args->port_id != -1) {
+ struct dlb2_dir_pq_pair *port;
+
+ port = dlb2_get_domain_used_dir_pq(args->port_id,
+ vdev_req,
+ domain);
+
+ if (!port || port->domain_id.phys_id != domain->id.phys_id ||
+ !port->port_configured) {
+ resp->status = DLB2_ST_INVALID_PORT_ID;
+ return -EINVAL;
+ }
+ }
+
+ /*
+ * If the queue's port is not configured, validate that a free
+ * port-queue pair is available.
+ */
+ if (args->port_id == -1 && list_empty(&domain->avail_dir_pq_pairs)) {
+ resp->status = DLB2_ST_DIR_QUEUES_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static void dlb2_configure_ldb_queue(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain,
+ struct dlb2_ldb_queue *queue,
+ struct dlb2_create_ldb_queue_args *args,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ union dlb2_sys_vf_ldb_vqid_v r0 = { {0} };
+ union dlb2_sys_vf_ldb_vqid2qid r1 = { {0} };
+ union dlb2_sys_ldb_qid2vqid r2 = { {0} };
+ union dlb2_sys_ldb_vasqid_v r3 = { {0} };
+ union dlb2_lsp_qid_ldb_infl_lim r4 = { {0} };
+ union dlb2_lsp_qid_aqed_active_lim r5 = { {0} };
+ union dlb2_aqed_pipe_qid_hid_width r6 = { {0} };
+ union dlb2_sys_ldb_qid_its r7 = { {0} };
+ union dlb2_lsp_qid_atm_depth_thrsh r8 = { {0} };
+ union dlb2_lsp_qid_naldb_depth_thrsh r9 = { {0} };
+ union dlb2_aqed_pipe_qid_fid_lim r10 = { {0} };
+ union dlb2_chp_ord_qid_sn_map r11 = { {0} };
+ union dlb2_sys_ldb_qid_cfg_v r12 = { {0} };
+ union dlb2_sys_ldb_qid_v r13 = { {0} };
+
+ struct dlb2_sn_group *sn_group;
+ unsigned int offs;
+
+ /* QID write permissions are turned on when the domain is started */
+ r3.field.vasqid_v = 0;
+
+ offs = domain->id.phys_id * DLB2_MAX_NUM_LDB_QUEUES +
+ queue->id.phys_id;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_LDB_VASQID_V(offs), r3.val);
+
+ /*
+ * Unordered QIDs get 4K inflights, ordered get as many as the number
+ * of sequence numbers.
+ */
+ r4.field.limit = args->num_qid_inflights;
+
+ DLB2_CSR_WR(hw, DLB2_LSP_QID_LDB_INFL_LIM(queue->id.phys_id), r4.val);
+
+ r5.field.limit = queue->aqed_limit;
+
+ if (r5.field.limit > DLB2_MAX_NUM_AQED_ENTRIES)
+ r5.field.limit = DLB2_MAX_NUM_AQED_ENTRIES;
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_QID_AQED_ACTIVE_LIM(queue->id.phys_id),
+ r5.val);
+
+ switch (args->lock_id_comp_level) {
+ case 64:
+ r6.field.compress_code = 1;
+ break;
+ case 128:
+ r6.field.compress_code = 2;
+ break;
+ case 256:
+ r6.field.compress_code = 3;
+ break;
+ case 512:
+ r6.field.compress_code = 4;
+ break;
+ case 1024:
+ r6.field.compress_code = 5;
+ break;
+ case 2048:
+ r6.field.compress_code = 6;
+ break;
+ case 4096:
+ r6.field.compress_code = 7;
+ break;
+ case 0:
+ case 65536:
+ r6.field.compress_code = 0;
+ }
+
+ DLB2_CSR_WR(hw,
+ DLB2_AQED_PIPE_QID_HID_WIDTH(queue->id.phys_id),
+ r6.val);
+
+ /* Don't timestamp QEs that pass through this queue */
+ r7.field.qid_its = 0;
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_LDB_QID_ITS(queue->id.phys_id),
+ r7.val);
+
+ r8.field.thresh = args->depth_threshold;
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_QID_ATM_DEPTH_THRSH(queue->id.phys_id),
+ r8.val);
+
+ r9.field.thresh = args->depth_threshold;
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_QID_NALDB_DEPTH_THRSH(queue->id.phys_id),
+ r9.val);
+
+ /*
+ * This register limits the number of inflight flows a queue can have
+ * at one time. It has an upper bound of 2048, but can be
+ * over-subscribed. 512 is chosen so that a single queue doesn't use
+ * the entire atomic storage, but can use a substantial portion if
+ * needed.
+ */
+ r10.field.qid_fid_limit = 512;
+
+ DLB2_CSR_WR(hw,
+ DLB2_AQED_PIPE_QID_FID_LIM(queue->id.phys_id),
+ r10.val);
+
+ /* Configure SNs */
+ sn_group = &hw->rsrcs.sn_groups[queue->sn_group];
+ r11.field.mode = sn_group->mode;
+ r11.field.slot = queue->sn_slot;
+ r11.field.grp = sn_group->id;
+
+ DLB2_CSR_WR(hw, DLB2_CHP_ORD_QID_SN_MAP(queue->id.phys_id), r11.val);
+
+ r12.field.sn_cfg_v = (args->num_sequence_numbers != 0);
+ r12.field.fid_cfg_v = (args->num_atomic_inflights != 0);
+
+ DLB2_CSR_WR(hw, DLB2_SYS_LDB_QID_CFG_V(queue->id.phys_id), r12.val);
+
+ if (vdev_req) {
+ offs = vdev_id * DLB2_MAX_NUM_LDB_QUEUES + queue->id.virt_id;
+
+ r0.field.vqid_v = 1;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VQID_V(offs), r0.val);
+
+ r1.field.qid = queue->id.phys_id;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_VF_LDB_VQID2QID(offs), r1.val);
+
+ r2.field.vqid = queue->id.virt_id;
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_LDB_QID2VQID(queue->id.phys_id),
+ r2.val);
+ }
+
+ r13.field.qid_v = 1;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_LDB_QID_V(queue->id.phys_id), r13.val);
+}
+
+static void dlb2_configure_dir_queue(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain,
+ struct dlb2_dir_pq_pair *queue,
+ struct dlb2_create_dir_queue_args *args,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ union dlb2_sys_dir_vasqid_v r0 = { {0} };
+ union dlb2_sys_dir_qid_its r1 = { {0} };
+ union dlb2_lsp_qid_dir_depth_thrsh r2 = { {0} };
+ union dlb2_sys_dir_qid_v r5 = { {0} };
+
+ unsigned int offs;
+
+ /* QID write permissions are turned on when the domain is started */
+ r0.field.vasqid_v = 0;
+
+ offs = domain->id.phys_id * DLB2_MAX_NUM_DIR_QUEUES +
+ queue->id.phys_id;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_DIR_VASQID_V(offs), r0.val);
+
+ /* Don't timestamp QEs that pass through this queue */
+ r1.field.qid_its = 0;
+
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_DIR_QID_ITS(queue->id.phys_id),
+ r1.val);
+
+ r2.field.thresh = args->depth_threshold;
+
+ DLB2_CSR_WR(hw,
+ DLB2_LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id),
+ r2.val);
+
+ if (vdev_req) {
+ union dlb2_sys_vf_dir_vqid_v r3 = { {0} };
+ union dlb2_sys_vf_dir_vqid2qid r4 = { {0} };
+
+ offs = vdev_id * DLB2_MAX_NUM_DIR_QUEUES + queue->id.virt_id;
+
+ r3.field.vqid_v = 1;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VQID_V(offs), r3.val);
+
+ r4.field.qid = queue->id.phys_id;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_VF_DIR_VQID2QID(offs), r4.val);
+ }
+
+ r5.field.qid_v = 1;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_DIR_QID_V(queue->id.phys_id), r5.val);
+
+ queue->queue_configured = true;
+}
+
static bool dlb2_port_find_slot(struct dlb2_ldb_port *port,
enum dlb2_qid_map_state state,
int *slot)
@@ -955,6 +1337,68 @@ dlb2_domain_attach_resources(struct dlb2_hw *hw,
return 0;
}
+static int
+dlb2_ldb_queue_attach_to_sn_group(struct dlb2_hw *hw,
+ struct dlb2_ldb_queue *queue,
+ struct dlb2_create_ldb_queue_args *args)
+{
+ int slot = -1;
+ int i;
+
+ queue->sn_cfg_valid = false;
+
+ if (args->num_sequence_numbers == 0)
+ return 0;
+
+ for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+ struct dlb2_sn_group *group = &hw->rsrcs.sn_groups[i];
+
+ if (group->sequence_numbers_per_queue ==
+ args->num_sequence_numbers &&
+ !dlb2_sn_group_full(group)) {
+ slot = dlb2_sn_group_alloc_slot(group);
+ if (slot >= 0)
+ break;
+ }
+ }
+
+ if (slot == -1) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: no sequence number slots available\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ queue->sn_cfg_valid = true;
+ queue->sn_group = i;
+ queue->sn_slot = slot;
+ return 0;
+}
+
+static int
+dlb2_ldb_queue_attach_resources(struct dlb2_hw *hw,
+ struct dlb2_hw_domain *domain,
+ struct dlb2_ldb_queue *queue,
+ struct dlb2_create_ldb_queue_args *args)
+{
+ int ret;
+
+ ret = dlb2_ldb_queue_attach_to_sn_group(hw, queue, args);
+ if (ret)
+ return ret;
+
+ /* Attach QID inflights */
+ queue->num_qid_inflights = args->num_qid_inflights;
+
+ /* Attach atomic inflights */
+ queue->aqed_limit = args->num_atomic_inflights;
+
+ domain->num_avail_aqed_entries -= args->num_atomic_inflights;
+ domain->num_used_aqed_entries += args->num_atomic_inflights;
+
+ return 0;
+}
+
static void dlb2_ldb_port_cq_enable(struct dlb2_hw *hw,
struct dlb2_ldb_port *port)
{
@@ -1670,6 +2114,203 @@ int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
}
static void
+dlb2_log_create_ldb_queue_args(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_create_ldb_queue_args *args,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ DLB2_HW_DBG(hw, "DLB2 create load-balanced queue arguments:\n");
+ if (vdev_req)
+ DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+ DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+ domain_id);
+ DLB2_HW_DBG(hw, "\tNumber of sequence numbers: %d\n",
+ args->num_sequence_numbers);
+ DLB2_HW_DBG(hw, "\tNumber of QID inflights: %d\n",
+ args->num_qid_inflights);
+ DLB2_HW_DBG(hw, "\tNumber of ATM inflights: %d\n",
+ args->num_atomic_inflights);
+}
+
+/**
+ * dlb2_hw_create_ldb_queue() - Allocate and initialize a DLB LDB queue.
+ * @hw: Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_create_ldb_queue_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_hw_domain *domain;
+ struct dlb2_ldb_queue *queue;
+ int ret;
+
+ dlb2_log_create_ldb_queue_args(hw, domain_id, args, vdev_req, vdev_id);
+
+ /*
+ * Verify that hardware resources are available before attempting to
+ * satisfy the request. This simplifies the error unwinding code.
+ */
+ ret = dlb2_verify_create_ldb_queue_args(hw,
+ domain_id,
+ args,
+ resp,
+ vdev_req,
+ vdev_id);
+ if (ret)
+ return ret;
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+ if (!domain) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: domain not found\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ queue = DLB2_DOM_LIST_HEAD(domain->avail_ldb_queues, typeof(*queue));
+
+ if (!queue) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: no available ldb queues\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ ret = dlb2_ldb_queue_attach_resources(hw, domain, queue, args);
+ if (ret < 0) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: failed to attach the ldb queue resources\n",
+ __func__, __LINE__);
+ return ret;
+ }
+
+ dlb2_configure_ldb_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
+ queue->num_mappings = 0;
+
+ queue->configured = true;
+
+ /*
+ * Configuration succeeded, so move the resource from the 'avail' to
+ * the 'used' list.
+ */
+ list_del(&queue->domain_list);
+
+ list_add(&queue->domain_list, &domain->used_ldb_queues);
+
+ resp->status = 0;
+ resp->id = (vdev_req) ? queue->id.virt_id : queue->id.phys_id;
+
+ return 0;
+}
+
+static void
+dlb2_log_create_dir_queue_args(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_create_dir_queue_args *args,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ DLB2_HW_DBG(hw, "DLB2 create directed queue arguments:\n");
+ if (vdev_req)
+ DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+ DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+ DLB2_HW_DBG(hw, "\tPort ID: %d\n", args->port_id);
+}
+
+/**
+ * dlb2_hw_create_dir_queue() - Allocate and initialize a DLB DIR queue.
+ * @hw: Contains the current state of the DLB2 hardware.
+ * @domain_id: Domain ID
+ * @args: User-provided arguments.
+ * @resp: Response to user.
+ * @vdev_req: Request came from a virtual device.
+ * @vdev_id: If vdev_req is true, this contains the virtual device's ID.
+ *
+ * Return: returns < 0 on error, 0 otherwise. If the driver is unable to
+ * satisfy a request, resp->status will be set accordingly.
+ */
+int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_create_dir_queue_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_dir_pq_pair *queue;
+ struct dlb2_hw_domain *domain;
+ int ret;
+
+ dlb2_log_create_dir_queue_args(hw, domain_id, args, vdev_req, vdev_id);
+
+ /*
+ * Verify that hardware resources are available before attempting to
+ * satisfy the request. This simplifies the error unwinding code.
+ */
+ ret = dlb2_verify_create_dir_queue_args(hw,
+ domain_id,
+ args,
+ resp,
+ vdev_req,
+ vdev_id);
+ if (ret)
+ return ret;
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+ if (!domain) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: domain not found\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ if (args->port_id != -1)
+ queue = dlb2_get_domain_used_dir_pq(args->port_id,
+ vdev_req,
+ domain);
+ else
+ queue = DLB2_DOM_LIST_HEAD(domain->avail_dir_pq_pairs,
+ typeof(*queue));
+
+ if (!queue) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: no available dir queues\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ dlb2_configure_dir_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
+ /*
+ * Configuration succeeded, so move the resource from the 'avail' to
+ * the 'used' list (if it's not already there).
+ */
+ if (args->port_id == -1) {
+ list_del(&queue->domain_list);
+
+ list_add(&queue->domain_list, &domain->used_dir_pq_pairs);
+ }
+
+ resp->status = 0;
+
+ resp->id = (vdev_req) ? queue->id.virt_id : queue->id.phys_id;
+
+ return 0;
+}
+
+static void
dlb2_domain_finish_unmap_port_slot(struct dlb2_hw *hw,
struct dlb2_hw_domain *domain,
struct dlb2_ldb_port *port,
@@ -2132,6 +2773,54 @@ static bool dlb2_dir_queue_is_empty(struct dlb2_hw *hw,
return dlb2_dir_queue_depth(hw, queue) == 0;
}
+static void dlb2_log_get_dir_queue_depth(struct dlb2_hw *hw,
+ u32 domain_id,
+ u32 queue_id,
+ bool vdev_req,
+ unsigned int vf_id)
+{
+ DLB2_HW_DBG(hw, "DLB get directed queue depth:\n");
+ if (vdev_req)
+ DLB2_HW_DBG(hw, "(Request from VF %d)\n", vf_id);
+ DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+ DLB2_HW_DBG(hw, "\tQueue ID: %d\n", queue_id);
+}
+
+int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_get_dir_queue_depth_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_dir_pq_pair *queue;
+ struct dlb2_hw_domain *domain;
+ int id;
+
+ id = domain_id;
+
+ dlb2_log_get_dir_queue_depth(hw, domain_id, args->queue_id,
+ vdev_req, vdev_id);
+
+ domain = dlb2_get_domain_from_id(hw, id, vdev_req, vdev_id);
+ if (!domain) {
+ resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+ return -EINVAL;
+ }
+
+ id = args->queue_id;
+
+ queue = dlb2_get_domain_used_dir_pq(id, vdev_req, domain);
+ if (!queue) {
+ resp->status = DLB2_ST_INVALID_QID;
+ return -EINVAL;
+ }
+
+ resp->id = dlb2_dir_queue_depth(hw, queue);
+
+ return 0;
+}
+
static u32 dlb2_ldb_queue_depth(struct dlb2_hw *hw,
struct dlb2_ldb_queue *queue)
{
@@ -2156,6 +2845,49 @@ static bool dlb2_ldb_queue_is_empty(struct dlb2_hw *hw,
return dlb2_ldb_queue_depth(hw, queue) == 0;
}
+static void dlb2_log_get_ldb_queue_depth(struct dlb2_hw *hw,
+ u32 domain_id,
+ u32 queue_id,
+ bool vdev_req,
+ unsigned int vf_id)
+{
+ DLB2_HW_DBG(hw, "DLB get load-balanced queue depth:\n");
+ if (vdev_req)
+ DLB2_HW_DBG(hw, "(Request from VF %d)\n", vf_id);
+ DLB2_HW_DBG(hw, "\tDomain ID: %d\n", domain_id);
+ DLB2_HW_DBG(hw, "\tQueue ID: %d\n", queue_id);
+}
+
+int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_get_ldb_queue_depth_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_hw_domain *domain;
+ struct dlb2_ldb_queue *queue;
+
+ dlb2_log_get_ldb_queue_depth(hw, domain_id, args->queue_id,
+ vdev_req, vdev_id);
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+ if (!domain) {
+ resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+ return -EINVAL;
+ }
+
+ queue = dlb2_get_domain_ldb_queue(args->queue_id, vdev_req, domain);
+ if (!queue) {
+ resp->status = DLB2_ST_INVALID_QID;
+ return -EINVAL;
+ }
+
+ resp->id = dlb2_ldb_queue_depth(hw, queue);
+
+ return 0;
+}
+
static void __dlb2_domain_reset_ldb_port_registers(struct dlb2_hw *hw,
struct dlb2_ldb_port *port)
{
diff --git a/drivers/misc/dlb2/dlb2_resource.h b/drivers/misc/dlb2/dlb2_resource.h
index 7dc50865681b..dfb42e0e13de 100644
--- a/drivers/misc/dlb2/dlb2_resource.h
+++ b/drivers/misc/dlb2/dlb2_resource.h
@@ -70,6 +70,73 @@ int dlb2_hw_create_sched_domain(struct dlb2_hw *hw,
unsigned int vdev_id);
/**
+ * dlb2_hw_create_ldb_queue() - create a load-balanced queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a load-balanced queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the queue ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, the domain is not configured,
+ * the domain has already been started, or the requested queue name is
+ * already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_ldb_queue(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_create_ldb_queue_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_request,
+ unsigned int vdev_id);
+
+/**
+ * dlb2_hw_create_dir_queue() - create a directed queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue creation arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function creates a directed queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the queue ID.
+ *
+ * resp->id contains a virtual ID if vdev_request is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, the domain is not configured,
+ * or the domain has already been started.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_create_dir_queue(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_create_dir_queue_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_request,
+ unsigned int vdev_id);
+
+/**
* dlb2_reset_domain() - reset a scheduling domain
* @hw: dlb2_hw handle for a particular device.
* @domain_id: domain ID.
@@ -130,6 +197,64 @@ int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
*/
void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw);
+/**
+ * dlb2_hw_get_ldb_queue_depth() - returns the depth of a load-balanced queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue depth args
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the depth of a load-balanced queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the depth.
+ *
+ * Errors:
+ * EINVAL - Invalid domain ID or queue ID.
+ */
+int dlb2_hw_get_ldb_queue_depth(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_get_ldb_queue_depth_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_request,
+ unsigned int vdev_id);
+
+/**
+ * dlb2_hw_get_dir_queue_depth() - returns the depth of a directed queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: queue depth args
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns the depth of a directed queue.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the depth.
+ *
+ * Errors:
+ * EINVAL - Invalid domain ID or queue ID.
+ */
+int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_get_dir_queue_depth_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_request,
+ unsigned int vdev_id);
+
enum dlb2_virt_mode {
DLB2_VIRT_NONE,
DLB2_VIRT_SRIOV,
diff --git a/include/uapi/linux/dlb2_user.h b/include/uapi/linux/dlb2_user.h
index 02d380b51c2c..ee8f5a970065 100644
--- a/include/uapi/linux/dlb2_user.h
+++ b/include/uapi/linux/dlb2_user.h
@@ -227,6 +227,128 @@ enum dlb2_user_interface_commands {
NUM_DLB2_CMD,
};
+/*********************************/
+/* 'domain' device file commands */
+/*********************************/
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_LDB_QUEUE: Configure a load-balanced queue.
+ * Input parameters:
+ * - num_atomic_inflights: This specifies the amount of temporary atomic QE
+ * storage for this queue. If zero, the queue will not support atomic
+ * scheduling.
+ * - num_sequence_numbers: This specifies the number of sequence numbers used
+ * by this queue. If zero, the queue will not support ordered scheduling.
+ * If non-zero, the queue will not support unordered scheduling.
+ * - num_qid_inflights: The maximum number of QEs that can be inflight
+ * (scheduled to a CQ but not completed) at any time. If
+ * num_sequence_numbers is non-zero, num_qid_inflights must be set equal
+ * to num_sequence_numbers.
+ * - lock_id_comp_level: Lock ID compression level. Specifies the number of
+ * unique lock IDs the queue should compress down to. Valid compression
+ * levels: 0, 64, 128, 256, 512, 1k, 2k, 4k, 64k. If lock_id_comp_level is
+ * 0, the queue won't compress its lock IDs.
+ * - depth_threshold: DLB sets two bits in the received QE to indicate the
+ * depth of the queue relative to the threshold before scheduling the
+ * QE to a CQ:
+ * - 2’b11: depth > threshold
+ * - 2’b10: threshold >= depth > 0.75 * threshold
+ * - 2’b01: 0.75 * threshold >= depth > 0.5 * threshold
+ * - 2’b00: depth <= 0.5 * threshold
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Queue ID.
+ */
+struct dlb2_create_ldb_queue_args {
+ /* Output parameters */
+ struct dlb2_cmd_response response;
+ /* Input parameters */
+ __u32 num_sequence_numbers;
+ __u32 num_qid_inflights;
+ __u32 num_atomic_inflights;
+ __u32 lock_id_comp_level;
+ __u32 depth_threshold;
+ __u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_CREATE_DIR_QUEUE: Configure a directed queue.
+ * Input parameters:
+ * - port_id: Port ID. If the corresponding directed port is already created,
+ * specify its ID here. Else this argument must be 0xFFFFFFFF to indicate
+ * that the queue is being created before the port.
+ * - depth_threshold: DLB sets two bits in the received QE to indicate the
+ * depth of the queue relative to the threshold before scheduling the
+ * QE to a CQ:
+ * - 2’b11: depth > threshold
+ * - 2’b10: threshold >= depth > 0.75 * threshold
+ * - 2’b01: 0.75 * threshold >= depth > 0.5 * threshold
+ * - 2’b00: depth <= 0.5 * threshold
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Queue ID.
+ */
+struct dlb2_create_dir_queue_args {
+ /* Output parameters */
+ struct dlb2_cmd_response response;
+ /* Input parameters */
+ __s32 port_id;
+ __u32 depth_threshold;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_GET_LDB_QUEUE_DEPTH: Get a load-balanced queue's depth.
+ * Input parameters:
+ * - queue_id: The load-balanced queue ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ * - response.id: queue depth.
+ */
+struct dlb2_get_ldb_queue_depth_args {
+ /* Output parameters */
+ struct dlb2_cmd_response response;
+ /* Input parameters */
+ __u32 queue_id;
+ __u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DIR_QUEUE_DEPTH: Get a directed queue's depth.
+ * Input parameters:
+ * - queue_id: The directed queue ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ * - response.id: queue depth.
+ */
+struct dlb2_get_dir_queue_depth_args {
+ /* Output parameters */
+ struct dlb2_cmd_response response;
+ /* Input parameters */
+ __u32 queue_id;
+ __u32 padding0;
+};
+
+enum dlb2_domain_user_interface_commands {
+ DLB2_DOMAIN_CMD_CREATE_LDB_QUEUE,
+ DLB2_DOMAIN_CMD_CREATE_DIR_QUEUE,
+ DLB2_DOMAIN_CMD_GET_LDB_QUEUE_DEPTH,
+ DLB2_DOMAIN_CMD_GET_DIR_QUEUE_DEPTH,
+
+ /* NUM_DLB2_DOMAIN_CMD must be last */
+ NUM_DLB2_DOMAIN_CMD,
+};
+
/********************/
/* dlb2 ioctl codes */
/********************/
@@ -245,5 +367,21 @@ enum dlb2_user_interface_commands {
_IOR(DLB2_IOC_MAGIC, \
DLB2_CMD_GET_NUM_RESOURCES, \
struct dlb2_get_num_resources_args)
+#define DLB2_IOC_CREATE_LDB_QUEUE \
+ _IOWR(DLB2_IOC_MAGIC, \
+ DLB2_DOMAIN_CMD_CREATE_LDB_QUEUE, \
+ struct dlb2_create_ldb_queue_args)
+#define DLB2_IOC_CREATE_DIR_QUEUE \
+ _IOWR(DLB2_IOC_MAGIC, \
+ DLB2_DOMAIN_CMD_CREATE_DIR_QUEUE, \
+ struct dlb2_create_dir_queue_args)
+#define DLB2_IOC_GET_LDB_QUEUE_DEPTH \
+ _IOWR(DLB2_IOC_MAGIC, \
+ DLB2_DOMAIN_CMD_GET_LDB_QUEUE_DEPTH, \
+ struct dlb2_get_ldb_queue_depth_args)
+#define DLB2_IOC_GET_DIR_QUEUE_DEPTH \
+ _IOWR(DLB2_IOC_MAGIC, \
+ DLB2_DOMAIN_CMD_GET_DIR_QUEUE_DEPTH, \
+ struct dlb2_get_dir_queue_depth_args)
#endif /* __DLB2_USER_H */
--
2.13.6
This commit adds the hardware resource data structures and functions for
their initialization/teardown and for device power-on. In subsequent
commits, dlb2_resource.c will be expanded to hold the dlb2
resource-management and configuration logic (using the data structures
defined in dlb2_hw_types.h).
This commit also introduces dlb2_bitmap_* functions, a thin convenience
layer wrapping the Linux bitmap interfaces, used by the bitmaps in the dlb2
hardware types.
Signed-off-by: Gage Eads <[email protected]>
Reviewed-by: Magnus Karlsson <[email protected]>
---
drivers/misc/dlb2/Makefile | 7 +-
drivers/misc/dlb2/dlb2_bitmap.h | 121 +++++++++++++++++++++++
drivers/misc/dlb2/dlb2_hw_types.h | 173 +++++++++++++++++++++++++++++++++
drivers/misc/dlb2/dlb2_main.c | 51 ++++++++++
drivers/misc/dlb2/dlb2_main.h | 11 +++
drivers/misc/dlb2/dlb2_pf_ops.c | 68 +++++++++++++
drivers/misc/dlb2/dlb2_regs.h | 83 ++++++++++++++++
drivers/misc/dlb2/dlb2_resource.c | 197 ++++++++++++++++++++++++++++++++++++++
drivers/misc/dlb2/dlb2_resource.h | 46 +++++++++
9 files changed, 754 insertions(+), 3 deletions(-)
create mode 100644 drivers/misc/dlb2/dlb2_bitmap.h
create mode 100644 drivers/misc/dlb2/dlb2_regs.h
create mode 100644 drivers/misc/dlb2/dlb2_resource.c
create mode 100644 drivers/misc/dlb2/dlb2_resource.h
diff --git a/drivers/misc/dlb2/Makefile b/drivers/misc/dlb2/Makefile
index 95e67e5bd8ff..4fdf7ffc555b 100644
--- a/drivers/misc/dlb2/Makefile
+++ b/drivers/misc/dlb2/Makefile
@@ -4,6 +4,7 @@
obj-$(CONFIG_INTEL_DLB2) := dlb2.o
-dlb2-objs := \
- dlb2_main.o \
- dlb2_pf_ops.o \
+dlb2-objs := \
+ dlb2_main.o \
+ dlb2_pf_ops.o \
+ dlb2_resource.o \
diff --git a/drivers/misc/dlb2/dlb2_bitmap.h b/drivers/misc/dlb2/dlb2_bitmap.h
new file mode 100644
index 000000000000..c5bb4ba84d5c
--- /dev/null
+++ b/drivers/misc/dlb2/dlb2_bitmap.h
@@ -0,0 +1,121 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ * Copyright(c) 2017-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_OSDEP_BITMAP_H
+#define __DLB2_OSDEP_BITMAP_H
+
+#include <linux/bitmap.h>
+#include <linux/slab.h>
+
+#include "dlb2_main.h"
+
+/*************************/
+/*** Bitmap operations ***/
+/*************************/
+struct dlb2_bitmap {
+ unsigned long *map;
+ unsigned int len;
+};
+
+/**
+ * dlb2_bitmap_alloc() - alloc a bitmap data structure
+ * @bitmap: pointer to dlb2_bitmap structure pointer.
+ * @len: number of entries in the bitmap.
+ *
+ * This function allocates a bitmap and initializes it with length @len. All
+ * entries are initially zero.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or len is 0.
+ * ENOMEM - could not allocate memory for the bitmap data structure.
+ */
+static inline int dlb2_bitmap_alloc(struct dlb2_bitmap **bitmap,
+ unsigned int len)
+{
+ struct dlb2_bitmap *bm;
+
+ if (!bitmap || len == 0)
+ return -EINVAL;
+
+ bm = kzalloc(sizeof(*bm), GFP_KERNEL);
+ if (!bm)
+ return -ENOMEM;
+
+ bm->map = kcalloc(BITS_TO_LONGS(len), sizeof(*bm->map), GFP_KERNEL);
+ if (!bm->map) {
+ kfree(bm);
+ return -ENOMEM;
+ }
+
+ bm->len = len;
+
+ *bitmap = bm;
+
+ return 0;
+}
+
+/**
+ * dlb2_bitmap_free() - free a previously allocated bitmap data structure
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function frees a bitmap that was allocated with dlb2_bitmap_alloc().
+ */
+static inline void dlb2_bitmap_free(struct dlb2_bitmap *bitmap)
+{
+ if (!bitmap)
+ return;
+
+ kfree(bitmap->map);
+
+ kfree(bitmap);
+}
+
+/**
+ * dlb2_bitmap_fill() - fill a bitmap with all 1s
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function sets all bitmap values to 1.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_fill(struct dlb2_bitmap *bitmap)
+{
+ if (!bitmap || !bitmap->map)
+ return -EINVAL;
+
+ bitmap_fill(bitmap->map, bitmap->len);
+
+ return 0;
+}
+
+/**
+ * dlb2_bitmap_zero() - fill a bitmap with all 0s
+ * @bitmap: pointer to dlb2_bitmap structure.
+ *
+ * This function sets all bitmap values to 0.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb2_bitmap_zero(struct dlb2_bitmap *bitmap)
+{
+ if (!bitmap || !bitmap->map)
+ return -EINVAL;
+
+ bitmap_zero(bitmap->map, bitmap->len);
+
+ return 0;
+}
+
+#endif /* __DLB2_OSDEP_BITMAP_H */
diff --git a/drivers/misc/dlb2/dlb2_hw_types.h b/drivers/misc/dlb2/dlb2_hw_types.h
index 10b26356535f..0db398aeccd5 100644
--- a/drivers/misc/dlb2/dlb2_hw_types.h
+++ b/drivers/misc/dlb2/dlb2_hw_types.h
@@ -6,6 +6,9 @@
#define __DLB2_HW_TYPES_H
#include <linux/io.h>
+#include <linux/types.h>
+
+#include "dlb2_bitmap.h"
#define DLB2_PCI_REG_READ(addr) ioread32(addr)
#define DLB2_PCI_REG_WRITE(reg, value) iowrite32(value, reg)
@@ -45,6 +48,169 @@
#define DLB2_MAX_QID_EMPTY_CHECK_LOOPS (32 * 64 * 1024 * (800 / 30))
#define DLB2_HZ 800000000
+struct dlb2_resource_id {
+ u32 phys_id;
+ u32 virt_id;
+ u8 vdev_owned;
+ u8 vdev_id;
+};
+
+struct dlb2_freelist {
+ u32 base;
+ u32 bound;
+ u32 offset;
+};
+
+static inline u32 dlb2_freelist_count(struct dlb2_freelist *list)
+{
+ return list->bound - list->base - list->offset;
+}
+
+struct dlb2_ldb_queue {
+ struct list_head domain_list;
+ struct list_head func_list;
+ struct dlb2_resource_id id;
+ struct dlb2_resource_id domain_id;
+ u32 num_qid_inflights;
+ u32 aqed_limit;
+ u32 sn_group; /* sn == sequence number */
+ u32 sn_slot;
+ u32 num_mappings;
+ u8 sn_cfg_valid;
+ u8 num_pending_additions;
+ u8 owned;
+ u8 configured;
+};
+
+/*
+ * Directed ports and queues are paired by nature, so the driver tracks them
+ * with a single data structure.
+ */
+struct dlb2_dir_pq_pair {
+ struct list_head domain_list;
+ struct list_head func_list;
+ struct dlb2_resource_id id;
+ struct dlb2_resource_id domain_id;
+ u32 ref_cnt;
+ u8 init_tkn_cnt;
+ u8 queue_configured;
+ u8 port_configured;
+ u8 owned;
+ u8 enabled;
+};
+
+enum dlb2_qid_map_state {
+ /* The slot doesn't contain a valid queue mapping */
+ DLB2_QUEUE_UNMAPPED,
+ /* The slot contains a valid queue mapping */
+ DLB2_QUEUE_MAPPED,
+ /* The driver is mapping a queue into this slot */
+ DLB2_QUEUE_MAP_IN_PROG,
+ /* The driver is unmapping a queue from this slot */
+ DLB2_QUEUE_UNMAP_IN_PROG,
+ /*
+ * The driver is unmapping a queue from this slot, and once complete
+ * will replace it with another mapping.
+ */
+ DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP,
+};
+
+struct dlb2_ldb_port_qid_map {
+ enum dlb2_qid_map_state state;
+ u16 qid;
+ u16 pending_qid;
+ u8 priority;
+ u8 pending_priority;
+};
+
+struct dlb2_ldb_port {
+ struct list_head domain_list;
+ struct list_head func_list;
+ struct dlb2_resource_id id;
+ struct dlb2_resource_id domain_id;
+ /* The qid_map represents the hardware QID mapping state. */
+ struct dlb2_ldb_port_qid_map qid_map[DLB2_MAX_NUM_QIDS_PER_LDB_CQ];
+ u32 hist_list_entry_base;
+ u32 hist_list_entry_limit;
+ u32 ref_cnt;
+ u8 init_tkn_cnt;
+ u8 num_pending_removals;
+ u8 num_mappings;
+ u8 owned;
+ u8 enabled;
+ u8 configured;
+};
+
+struct dlb2_sn_group {
+ u32 mode;
+ u32 sequence_numbers_per_queue;
+ u32 slot_use_bitmap;
+ u32 id;
+};
+
+struct dlb2_hw_domain {
+ struct dlb2_function_resources *parent_func;
+ struct list_head func_list;
+ struct list_head used_ldb_queues;
+ struct list_head used_ldb_ports[DLB2_NUM_COS_DOMAINS];
+ struct list_head used_dir_pq_pairs;
+ struct list_head avail_ldb_queues;
+ struct list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+ struct list_head avail_dir_pq_pairs;
+ u32 total_hist_list_entries;
+ u32 avail_hist_list_entries;
+ u32 hist_list_entry_base;
+ u32 hist_list_entry_offset;
+ u32 num_ldb_credits;
+ u32 num_dir_credits;
+ u32 num_avail_aqed_entries;
+ u32 num_used_aqed_entries;
+ struct dlb2_resource_id id;
+ int num_pending_removals;
+ int num_pending_additions;
+ u8 configured;
+ u8 started;
+};
+
+struct dlb2_function_resources {
+ struct list_head avail_domains;
+ struct list_head used_domains;
+ struct list_head avail_ldb_queues;
+ struct list_head avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+ struct list_head avail_dir_pq_pairs;
+ struct dlb2_bitmap *avail_hist_list_entries;
+ u32 num_avail_domains;
+ u32 num_avail_ldb_queues;
+ u32 num_avail_ldb_ports[DLB2_NUM_COS_DOMAINS];
+ u32 num_avail_dir_pq_pairs;
+ u32 num_avail_qed_entries;
+ u32 num_avail_dqed_entries;
+ u32 num_avail_aqed_entries;
+ u8 locked; /* (VDEV only) */
+};
+
+/*
+ * After initialization, each resource in dlb2_hw_resources is located in one
+ * of the following lists:
+ * -- The PF's available resources list. These are unconfigured resources owned
+ * by the PF and not allocated to a dlb2 scheduling domain.
+ * -- A VDEV's available resources list. These are VDEV-owned unconfigured
+ * resources not allocated to a dlb2 scheduling domain.
+ * -- A domain's available resources list. These are domain-owned unconfigured
+ * resources.
+ * -- A domain's used resources list. These are are domain-owned configured
+ * resources.
+ *
+ * A resource moves to a new list when a VDEV or domain is created or destroyed,
+ * or when the resource is configured.
+ */
+struct dlb2_hw_resources {
+ struct dlb2_ldb_queue ldb_queues[DLB2_MAX_NUM_LDB_QUEUES];
+ struct dlb2_ldb_port ldb_ports[DLB2_MAX_NUM_LDB_PORTS];
+ struct dlb2_dir_pq_pair dir_pq_pairs[DLB2_MAX_NUM_DIR_PORTS];
+ struct dlb2_sn_group sn_groups[DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS];
+};
+
struct dlb2_hw {
/* BAR 0 address */
void __iomem *csr_kva;
@@ -52,6 +218,13 @@ struct dlb2_hw {
/* BAR 2 address */
void __iomem *func_kva;
unsigned long func_phys_addr;
+
+ /* Resource tracking */
+ struct dlb2_hw_resources rsrcs;
+ struct dlb2_function_resources pf;
+ struct dlb2_function_resources vdev[DLB2_MAX_NUM_VDEVS];
+ struct dlb2_hw_domain domains[DLB2_MAX_NUM_DOMAINS];
+ u8 cos_reservation[DLB2_NUM_COS_DOMAINS];
};
#endif /* __DLB2_HW_TYPES_H */
diff --git a/drivers/misc/dlb2/dlb2_main.c b/drivers/misc/dlb2/dlb2_main.c
index 5d530ea862a3..f6d78d1532b0 100644
--- a/drivers/misc/dlb2/dlb2_main.c
+++ b/drivers/misc/dlb2/dlb2_main.c
@@ -12,6 +12,7 @@
#include <linux/uaccess.h>
#include "dlb2_main.h"
+#include "dlb2_resource.h"
static const char
dlb2_driver_copyright[] = "Copyright(c) 2018-2020 Intel Corporation";
@@ -27,6 +28,23 @@ static struct list_head dlb2_dev_list = LIST_HEAD_INIT(dlb2_dev_list);
static struct class *dlb2_class;
static dev_t dlb2_dev_number_base;
+static int dlb2_reset_device(struct pci_dev *pdev)
+{
+ int ret;
+
+ ret = pci_save_state(pdev);
+ if (ret)
+ return ret;
+
+ ret = __pci_reset_function_locked(pdev);
+ if (ret)
+ return ret;
+
+ pci_restore_state(pdev);
+
+ return 0;
+}
+
/*****************************/
/****** Devfs callbacks ******/
/*****************************/
@@ -137,12 +155,41 @@ static int dlb2_probe(struct pci_dev *pdev,
if (ret)
goto dma_set_mask_fail;
+ /*
+ * PM enable must be done before any other MMIO accesses, and this
+ * setting is persistent across device reset.
+ */
+ dlb2_dev->ops->enable_pm(dlb2_dev);
+
+ ret = dlb2_dev->ops->wait_for_device_ready(dlb2_dev, pdev);
+ if (ret)
+ goto wait_for_device_ready_fail;
+
+ ret = dlb2_reset_device(pdev);
+ if (ret)
+ goto dlb2_reset_fail;
+
+ ret = dlb2_resource_init(&dlb2_dev->hw);
+ if (ret)
+ goto resource_init_fail;
+
+ ret = dlb2_dev->ops->init_driver_state(dlb2_dev);
+ if (ret)
+ goto init_driver_state_fail;
+
+ dlb2_dev->ops->init_hardware(dlb2_dev);
+
mutex_lock(&dlb2_driver_lock);
list_add(&dlb2_dev->list, &dlb2_dev_list);
mutex_unlock(&dlb2_driver_lock);
return 0;
+init_driver_state_fail:
+ dlb2_resource_free(&dlb2_dev->hw);
+resource_init_fail:
+dlb2_reset_fail:
+wait_for_device_ready_fail:
dma_set_mask_fail:
dlb2_dev->ops->device_destroy(dlb2_dev, dlb2_class);
device_add_fail:
@@ -172,6 +219,10 @@ static void dlb2_remove(struct pci_dev *pdev)
list_del(&dlb2_dev->list);
mutex_unlock(&dlb2_driver_lock);
+ dlb2_dev->ops->free_driver_state(dlb2_dev);
+
+ dlb2_resource_free(&dlb2_dev->hw);
+
dlb2_dev->ops->device_destroy(dlb2_dev, dlb2_class);
dlb2_dev->ops->cdev_del(dlb2_dev);
diff --git a/drivers/misc/dlb2/dlb2_main.h b/drivers/misc/dlb2/dlb2_main.h
index 7d04e0d8869d..115cfadc0f12 100644
--- a/drivers/misc/dlb2/dlb2_main.h
+++ b/drivers/misc/dlb2/dlb2_main.h
@@ -39,6 +39,8 @@ struct dlb2_device_ops {
int (*map_pci_bar_space)(struct dlb2_dev *dev, struct pci_dev *pdev);
void (*unmap_pci_bar_space)(struct dlb2_dev *dev,
struct pci_dev *pdev);
+ int (*init_driver_state)(struct dlb2_dev *dev);
+ void (*free_driver_state)(struct dlb2_dev *dev);
int (*device_create)(struct dlb2_dev *dlb2_dev,
struct pci_dev *pdev,
struct class *dlb2_class);
@@ -48,6 +50,10 @@ struct dlb2_device_ops {
dev_t base,
const struct file_operations *fops);
void (*cdev_del)(struct dlb2_dev *dlb2_dev);
+ void (*enable_pm)(struct dlb2_dev *dev);
+ int (*wait_for_device_ready)(struct dlb2_dev *dev,
+ struct pci_dev *pdev);
+ void (*init_hardware)(struct dlb2_dev *dev);
};
extern struct dlb2_device_ops dlb2_pf_ops;
@@ -59,6 +65,11 @@ struct dlb2_dev {
struct dlb2_device_ops *ops;
struct list_head list;
struct device *dlb2_device;
+ /*
+ * The resource mutex serializes access to driver data structures and
+ * hardware registers.
+ */
+ struct mutex resource_mutex;
enum dlb2_device_type type;
int id;
dev_t dev_number;
diff --git a/drivers/misc/dlb2/dlb2_pf_ops.c b/drivers/misc/dlb2/dlb2_pf_ops.c
index 7920e491110f..cc35c3537507 100644
--- a/drivers/misc/dlb2/dlb2_pf_ops.c
+++ b/drivers/misc/dlb2/dlb2_pf_ops.c
@@ -1,7 +1,11 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright(c) 2017-2020 Intel Corporation */
+#include <linux/delay.h>
+
#include "dlb2_main.h"
+#include "dlb2_regs.h"
+#include "dlb2_resource.h"
/********************************/
/****** PCI BAR management ******/
@@ -50,6 +54,19 @@ dlb2_pf_map_pci_bar_space(struct dlb2_dev *dlb2_dev,
/*******************************/
static int
+dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
+{
+ mutex_init(&dlb2_dev->resource_mutex);
+
+ return 0;
+}
+
+static void
+dlb2_pf_free_driver_state(struct dlb2_dev *dlb2_dev)
+{
+}
+
+static int
dlb2_pf_cdev_add(struct dlb2_dev *dlb2_dev,
dev_t base,
const struct file_operations *fops)
@@ -112,6 +129,52 @@ dlb2_pf_device_destroy(struct dlb2_dev *dlb2_dev,
device_destroy(dlb2_class, dlb2_dev->dev_number);
}
+static void
+dlb2_pf_enable_pm(struct dlb2_dev *dlb2_dev)
+{
+ dlb2_clr_pmcsr_disable(&dlb2_dev->hw);
+}
+
+#define DLB2_READY_RETRY_LIMIT 1000
+static int
+dlb2_pf_wait_for_device_ready(struct dlb2_dev *dlb2_dev,
+ struct pci_dev *pdev)
+{
+ u32 retries = DLB2_READY_RETRY_LIMIT;
+
+ /* Allow at least 1s for the device to become active after power-on */
+ do {
+ union dlb2_cfg_mstr_cfg_diagnostic_idle_status idle;
+ union dlb2_cfg_mstr_cfg_pm_status pm_st;
+ u32 addr;
+
+ addr = DLB2_CFG_MSTR_CFG_PM_STATUS;
+
+ pm_st.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
+
+ addr = DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS;
+
+ idle.val = DLB2_CSR_RD(&dlb2_dev->hw, addr);
+
+ if (pm_st.field.pmsm == 1 && idle.field.dlb_func_idle == 1)
+ break;
+
+ usleep_range(1000, 2000);
+ } while (--retries);
+
+ if (!retries) {
+ dev_err(&pdev->dev, "Device idle test failed\n");
+ return -EIO;
+ }
+
+ return 0;
+}
+
+static void
+dlb2_pf_init_hardware(struct dlb2_dev *dlb2_dev)
+{
+}
+
/********************************/
/****** DLB2 PF Device Ops ******/
/********************************/
@@ -119,8 +182,13 @@ dlb2_pf_device_destroy(struct dlb2_dev *dlb2_dev,
struct dlb2_device_ops dlb2_pf_ops = {
.map_pci_bar_space = dlb2_pf_map_pci_bar_space,
.unmap_pci_bar_space = dlb2_pf_unmap_pci_bar_space,
+ .init_driver_state = dlb2_pf_init_driver_state,
+ .free_driver_state = dlb2_pf_free_driver_state,
.device_create = dlb2_pf_device_create,
.device_destroy = dlb2_pf_device_destroy,
.cdev_add = dlb2_pf_cdev_add,
.cdev_del = dlb2_pf_cdev_del,
+ .enable_pm = dlb2_pf_enable_pm,
+ .wait_for_device_ready = dlb2_pf_wait_for_device_ready,
+ .init_hardware = dlb2_pf_init_hardware,
};
diff --git a/drivers/misc/dlb2/dlb2_regs.h b/drivers/misc/dlb2/dlb2_regs.h
new file mode 100644
index 000000000000..b1a3ef2f4537
--- /dev/null
+++ b/drivers/misc/dlb2/dlb2_regs.h
@@ -0,0 +1,83 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_REGS_H
+#define __DLB2_REGS_H
+
+#include "linux/types.h"
+
+#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS 0xb4000004
+#define DLB2_CFG_MSTR_CFG_DIAGNOSTIC_IDLE_STATUS_RST 0x9d0fffff
+union dlb2_cfg_mstr_cfg_diagnostic_idle_status {
+ struct {
+ u32 chp_pipeidle : 1;
+ u32 rop_pipeidle : 1;
+ u32 lsp_pipeidle : 1;
+ u32 nalb_pipeidle : 1;
+ u32 ap_pipeidle : 1;
+ u32 dp_pipeidle : 1;
+ u32 qed_pipeidle : 1;
+ u32 dqed_pipeidle : 1;
+ u32 aqed_pipeidle : 1;
+ u32 sys_pipeidle : 1;
+ u32 chp_unit_idle : 1;
+ u32 rop_unit_idle : 1;
+ u32 lsp_unit_idle : 1;
+ u32 nalb_unit_idle : 1;
+ u32 ap_unit_idle : 1;
+ u32 dp_unit_idle : 1;
+ u32 qed_unit_idle : 1;
+ u32 dqed_unit_idle : 1;
+ u32 aqed_unit_idle : 1;
+ u32 sys_unit_idle : 1;
+ u32 rsvd1 : 4;
+ u32 mstr_cfg_ring_idle : 1;
+ u32 mstr_cfg_mstr_idle : 1;
+ u32 mstr_flr_clkreq_b : 1;
+ u32 mstr_proc_idle : 1;
+ u32 mstr_proc_idle_masked : 1;
+ u32 rsvd0 : 2;
+ u32 dlb_func_idle : 1;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_PM_STATUS 0xb4000014
+#define DLB2_CFG_MSTR_CFG_PM_STATUS_RST 0x100403e
+union dlb2_cfg_mstr_cfg_pm_status {
+ struct {
+ u32 prochot : 1;
+ u32 pgcb_dlb_idle : 1;
+ u32 pgcb_dlb_pg_rdy_ack_b : 1;
+ u32 pmsm_pgcb_req_b : 1;
+ u32 pgbc_pmc_pg_req_b : 1;
+ u32 pmc_pgcb_pg_ack_b : 1;
+ u32 pmc_pgcb_fet_en_b : 1;
+ u32 pgcb_fet_en_b : 1;
+ u32 rsvz0 : 1;
+ u32 rsvz1 : 1;
+ u32 fuse_force_on : 1;
+ u32 fuse_proc_disable : 1;
+ u32 rsvz2 : 1;
+ u32 rsvz3 : 1;
+ u32 pm_fsm_d0tod3_ok : 1;
+ u32 pm_fsm_d3tod0_ok : 1;
+ u32 dlb_in_d3 : 1;
+ u32 rsvz4 : 7;
+ u32 pmsm : 8;
+ } field;
+ u32 val;
+};
+
+#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE 0xb4000018
+#define DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE_RST 0x1
+union dlb2_cfg_mstr_cfg_pm_pmcsr_disable {
+ struct {
+ u32 disable : 1;
+ u32 rsvz0 : 31;
+ } field;
+ u32 val;
+};
+
+#endif /* __DLB2_REGS_H */
diff --git a/drivers/misc/dlb2/dlb2_resource.c b/drivers/misc/dlb2/dlb2_resource.c
new file mode 100644
index 000000000000..70036a2f5d60
--- /dev/null
+++ b/drivers/misc/dlb2/dlb2_resource.c
@@ -0,0 +1,197 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(c) 2016-2020 Intel Corporation */
+
+#include "dlb2_bitmap.h"
+#include "dlb2_hw_types.h"
+#include "dlb2_regs.h"
+#include "dlb2_resource.h"
+
+static void dlb2_init_fn_rsrc_lists(struct dlb2_function_resources *rsrc)
+{
+ int i;
+
+ INIT_LIST_HEAD(&rsrc->avail_domains);
+ INIT_LIST_HEAD(&rsrc->used_domains);
+ INIT_LIST_HEAD(&rsrc->avail_ldb_queues);
+ INIT_LIST_HEAD(&rsrc->avail_dir_pq_pairs);
+
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+ INIT_LIST_HEAD(&rsrc->avail_ldb_ports[i]);
+}
+
+static void dlb2_init_domain_rsrc_lists(struct dlb2_hw_domain *domain)
+{
+ int i;
+
+ INIT_LIST_HEAD(&domain->used_ldb_queues);
+ INIT_LIST_HEAD(&domain->used_dir_pq_pairs);
+ INIT_LIST_HEAD(&domain->avail_ldb_queues);
+ INIT_LIST_HEAD(&domain->avail_dir_pq_pairs);
+
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+ INIT_LIST_HEAD(&domain->used_ldb_ports[i]);
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+ INIT_LIST_HEAD(&domain->avail_ldb_ports[i]);
+}
+
+void dlb2_resource_free(struct dlb2_hw *hw)
+{
+ int i;
+
+ if (hw->pf.avail_hist_list_entries)
+ dlb2_bitmap_free(hw->pf.avail_hist_list_entries);
+
+ for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+ if (hw->vdev[i].avail_hist_list_entries)
+ dlb2_bitmap_free(hw->vdev[i].avail_hist_list_entries);
+ }
+}
+
+int dlb2_resource_init(struct dlb2_hw *hw)
+{
+ struct list_head *list;
+ unsigned int i;
+ int ret;
+
+ /*
+ * For optimal load-balancing, ports that map to one or more QIDs in
+ * common should not be in numerical sequence. This is application
+ * dependent, but the driver interleaves port IDs as much as possible
+ * to reduce the likelihood of this. This initial allocation maximizes
+ * the average distance between an ID and its immediate neighbors (i.e.
+ * the distance from 1 to 0 and to 2, the distance from 2 to 1 and to
+ * 3, etc.).
+ */
+ u8 init_ldb_port_allocation[DLB2_MAX_NUM_LDB_PORTS] = {
+ 0, 7, 14, 5, 12, 3, 10, 1, 8, 15, 6, 13, 4, 11, 2, 9,
+ 16, 23, 30, 21, 28, 19, 26, 17, 24, 31, 22, 29, 20, 27, 18, 25,
+ 32, 39, 46, 37, 44, 35, 42, 33, 40, 47, 38, 45, 36, 43, 34, 41,
+ 48, 55, 62, 53, 60, 51, 58, 49, 56, 63, 54, 61, 52, 59, 50, 57,
+ };
+
+ /* Zero-out resource tracking data structures */
+ memset(&hw->rsrcs, 0, sizeof(hw->rsrcs));
+ memset(&hw->pf, 0, sizeof(hw->pf));
+
+ dlb2_init_fn_rsrc_lists(&hw->pf);
+
+ for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+ memset(&hw->vdev[i], 0, sizeof(hw->vdev[i]));
+ dlb2_init_fn_rsrc_lists(&hw->vdev[i]);
+ }
+
+ for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+ memset(&hw->domains[i], 0, sizeof(hw->domains[i]));
+ dlb2_init_domain_rsrc_lists(&hw->domains[i]);
+ hw->domains[i].parent_func = &hw->pf;
+ }
+
+ /* Give all resources to the PF driver */
+ hw->pf.num_avail_domains = DLB2_MAX_NUM_DOMAINS;
+ for (i = 0; i < hw->pf.num_avail_domains; i++) {
+ list = &hw->domains[i].func_list;
+
+ list_add(list, &hw->pf.avail_domains);
+ }
+
+ hw->pf.num_avail_ldb_queues = DLB2_MAX_NUM_LDB_QUEUES;
+ for (i = 0; i < hw->pf.num_avail_ldb_queues; i++) {
+ list = &hw->rsrcs.ldb_queues[i].func_list;
+
+ list_add(list, &hw->pf.avail_ldb_queues);
+ }
+
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+ hw->pf.num_avail_ldb_ports[i] =
+ DLB2_MAX_NUM_LDB_PORTS / DLB2_NUM_COS_DOMAINS;
+
+ for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+ int cos_id = i >> DLB2_NUM_COS_DOMAINS;
+ struct dlb2_ldb_port *port;
+
+ port = &hw->rsrcs.ldb_ports[init_ldb_port_allocation[i]];
+
+ list_add(&port->func_list, &hw->pf.avail_ldb_ports[cos_id]);
+ }
+
+ hw->pf.num_avail_dir_pq_pairs = DLB2_MAX_NUM_DIR_PORTS;
+ for (i = 0; i < hw->pf.num_avail_dir_pq_pairs; i++) {
+ list = &hw->rsrcs.dir_pq_pairs[i].func_list;
+
+ list_add(list, &hw->pf.avail_dir_pq_pairs);
+ }
+
+ hw->pf.num_avail_qed_entries = DLB2_MAX_NUM_LDB_CREDITS;
+ hw->pf.num_avail_dqed_entries = DLB2_MAX_NUM_DIR_CREDITS;
+ hw->pf.num_avail_aqed_entries = DLB2_MAX_NUM_AQED_ENTRIES;
+
+ ret = dlb2_bitmap_alloc(&hw->pf.avail_hist_list_entries,
+ DLB2_MAX_NUM_HIST_LIST_ENTRIES);
+ if (ret)
+ goto unwind;
+
+ ret = dlb2_bitmap_fill(hw->pf.avail_hist_list_entries);
+ if (ret)
+ goto unwind;
+
+ for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+ ret = dlb2_bitmap_alloc(&hw->vdev[i].avail_hist_list_entries,
+ DLB2_MAX_NUM_HIST_LIST_ENTRIES);
+ if (ret)
+ goto unwind;
+
+ ret = dlb2_bitmap_zero(hw->vdev[i].avail_hist_list_entries);
+ if (ret)
+ goto unwind;
+ }
+
+ /* Initialize the hardware resource IDs */
+ for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+ hw->domains[i].id.phys_id = i;
+ hw->domains[i].id.vdev_owned = false;
+ }
+
+ for (i = 0; i < DLB2_MAX_NUM_LDB_QUEUES; i++) {
+ hw->rsrcs.ldb_queues[i].id.phys_id = i;
+ hw->rsrcs.ldb_queues[i].id.vdev_owned = false;
+ }
+
+ for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+ hw->rsrcs.ldb_ports[i].id.phys_id = i;
+ hw->rsrcs.ldb_ports[i].id.vdev_owned = false;
+ }
+
+ for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++) {
+ hw->rsrcs.dir_pq_pairs[i].id.phys_id = i;
+ hw->rsrcs.dir_pq_pairs[i].id.vdev_owned = false;
+ }
+
+ for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+ hw->rsrcs.sn_groups[i].id = i;
+ /* Default mode (0) is 64 sequence numbers per queue */
+ hw->rsrcs.sn_groups[i].mode = 0;
+ hw->rsrcs.sn_groups[i].sequence_numbers_per_queue = 64;
+ hw->rsrcs.sn_groups[i].slot_use_bitmap = 0;
+ }
+
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+ hw->cos_reservation[i] = 100 / DLB2_NUM_COS_DOMAINS;
+
+ return 0;
+
+unwind:
+ dlb2_resource_free(hw);
+
+ return ret;
+}
+
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
+{
+ union dlb2_cfg_mstr_cfg_pm_pmcsr_disable r0;
+
+ r0.val = DLB2_CSR_RD(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE);
+
+ r0.field.disable = 0;
+
+ DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
+}
diff --git a/drivers/misc/dlb2/dlb2_resource.h b/drivers/misc/dlb2/dlb2_resource.h
new file mode 100644
index 000000000000..af20fcf0a0cb
--- /dev/null
+++ b/drivers/misc/dlb2/dlb2_resource.h
@@ -0,0 +1,46 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_RESOURCE_H
+#define __DLB2_RESOURCE_H
+
+#include <linux/types.h>
+
+#include "dlb2_hw_types.h"
+
+/**
+ * dlb2_resource_init() - initialize the device
+ * @hw: pointer to struct dlb2_hw.
+ *
+ * This function initializes the device's software state (pointed to by the hw
+ * argument) and programs global scheduling QoS registers. This function should
+ * be called during driver initialization.
+ *
+ * The dlb2_hw struct must be unique per DLB 2.0 device and persist until the
+ * device is reset.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ */
+int dlb2_resource_init(struct dlb2_hw *hw);
+
+/**
+ * dlb2_resource_free() - free device state memory
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function frees software state pointed to by dlb2_hw. This function
+ * should be called when resetting the device or unloading the driver.
+ */
+void dlb2_resource_free(struct dlb2_hw *hw);
+
+/**
+ * dlb2_clr_pmcsr_disable() - power on bulk of DLB 2.0 logic
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * Clearing the PMCSR must be done at initialization to make the device fully
+ * operational.
+ */
+void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw);
+
+#endif /* __DLB2_RESOURCE_H */
--
2.13.6
This initial commit contains basic driver functionality (load, unload,
probe, and remove callbacks) file_operations stubs, and device
documentation as well.
Signed-off-by: Gage Eads <[email protected]>
Reviewed-by: Magnus Karlsson <[email protected]>
---
Documentation/misc-devices/dlb2.rst | 310 +++++++++++++++++++++++++++++++++++
Documentation/misc-devices/index.rst | 1 +
MAINTAINERS | 7 +
drivers/misc/Kconfig | 1 +
drivers/misc/Makefile | 1 +
drivers/misc/dlb2/Kconfig | 17 ++
drivers/misc/dlb2/Makefile | 8 +
drivers/misc/dlb2/dlb2_hw_types.h | 29 ++++
drivers/misc/dlb2/dlb2_main.c | 208 +++++++++++++++++++++++
drivers/misc/dlb2/dlb2_main.h | 37 +++++
include/linux/pci_ids.h | 2 +
11 files changed, 621 insertions(+)
create mode 100644 Documentation/misc-devices/dlb2.rst
create mode 100644 drivers/misc/dlb2/Kconfig
create mode 100644 drivers/misc/dlb2/Makefile
create mode 100644 drivers/misc/dlb2/dlb2_hw_types.h
create mode 100644 drivers/misc/dlb2/dlb2_main.c
create mode 100644 drivers/misc/dlb2/dlb2_main.h
diff --git a/Documentation/misc-devices/dlb2.rst b/Documentation/misc-devices/dlb2.rst
new file mode 100644
index 000000000000..a278662c66e0
--- /dev/null
+++ b/Documentation/misc-devices/dlb2.rst
@@ -0,0 +1,310 @@
+.. SPDX-License-Identifier: GPL-2.0-only
+
+===========================================
+Intel(R) Dynamic Load Balancer 2.0 Overview
+===========================================
+
+:Author: Gage Eads
+
+Contents
+========
+
+- Introduction
+- Scheduling
+- Queue Entry
+- Port
+- Queue
+- Credits
+- Scheduling Domain
+- Interrupts
+- Power Management
+- Virtualization
+- User Interface
+- Reset
+
+Introduction
+============
+
+The Intel(r) Dynamic Load Balancer 2.0 (Intel(r) DLB 2.0) is a PCIe device that
+provides load-balanced, prioritized scheduling of core-to-core communication.
+
+The Intel DLB 2.0 device consists of queues and arbiters that connect producer
+cores and consumer cores. The device implements load-balanced queueing features
+including:
+- Lock-free multi-producer/multi-consumer operation.
+- Multiple priority levels for varying traffic types.
+- 'Direct' traffic (i.e. multi-producer/single-consumer)
+- Simple unordered load-balanced distribution.
+- Atomic lock free load balancing across multiple consumers.
+- Queue element reordering feature allowing ordered load-balanced distribution.
+
+Intel DLB 2.0 can be used in an event-driven programming model, such as DPDK's
+Event Device Library[2]. Such frameworks are commonly used in packet processing
+pipelines that benefit from the framework's multi-core scalability, dynamic
+load-balancing, and variety of packet distribution and synchronization schemes.
+
+Scheduling Types
+================
+
+Intel DLB 2.0 supports four types of scheduling of 'events' (using DPDK
+terminology), where an event can represent any type of data (e.g. a network
+packet). The first, ``directed``, is multi-producer/single-consumer style
+core-to-core communication. The remaining three are
+multi-producer/multi-consumer, and support load-balancing across the consumers.
+
+- ``Unordered``: events are load-balanced across consumers without any ordering
+ guarantees.
+
+- ``Ordered``: events are load-balanced across consumers, and when the consumer
+ re-injects each event into the device it is re-ordered into the
+ original order. This scheduling type allows software to
+ parallelize ordered event processing without the synchronization
+ cost of re-ordering packets.
+
+- ``Atomic``: events are load-balanced across consumers, with the guarantee that
+ events from a particular 'flow' are only scheduled to a single
+ consumer at a time (but can migrate over time). This allows, for
+ example, packet processing applications to parallelize while
+ avoiding locks on per-flow data and maintaining ordering within a
+ flow.
+
+Intel DLB 2.0 provides hierarchical priority scheduling, with eight priority
+levels within each. Each consumer selects up to eight queues to receive events
+from, and assigns a priority to each of these 'connected' queues. To schedule
+an event to a consumer, the device selects the highest priority non-empty queue
+of the (up to) eight connected queues. Within that queue, the device selects
+the highest priority event available (selecting a lower priority event for
+starvation avoidance 1% of the time, by default).
+
+The device also supports four load-balanced scheduler classes of service. Each
+class of service receives a (user-configurable) guaranteed percentage of the
+scheduler bandwidth, and any unreserved bandwidth is divided evenly among the
+four classes.
+
+Queue Entry
+===========
+
+Each event is contained in a queue entry (QE), the fundamental unit of
+communication through the device, which consists of 8B of data and 8B of
+metadata, as depicted below.
+
+QE structure format
+::
+ data :64
+ opaque :16
+ qid :8
+ sched :2
+ priority :3
+ msg_type :3
+ lock_id :16
+ rsvd :8
+ cmd :8
+
+The ``data`` field can be any type that fits within 8B (pointer, integer,
+etc.); DLB 2.0 merely copies this field from producer to consumer. The
+``opaque`` and ``msg_type`` fields behave the same way.
+
+``qid`` is set by the producer to specify to which DLB 2.0 queue it wishes to
+enqueue this QE. The ID spaces for load-balanced and directed queues are both
+zero-based; the ``sched`` field is used to distinguish whether the queue is
+load-balanced or directed.
+
+``sched`` controls the scheduling type: atomic, unordered, ordered, or
+directed. The first three scheduling types are only valid for load-balanced
+queues, and the directed scheduling type is only valid for directed queues.
+
+``priority`` is the priority with which this QE should be scheduled.
+
+``lock_id``, used only by atomic scheduling, identifies the atomic flow to
+which the QE belongs. When sending a directed event, ``lock_id`` is simply
+copied like the ``data``, ``opaque``, and ``msg_type`` fields.
+
+``cmd`` specifies the operation, such as:
+- Enqueue a new QE
+- Forward a QE that was dequeued
+- Complete/terminate a QE that was dequeued
+- Return one or more consumer queue tokens.
+- Arm the port's consumer queue interrupt.
+
+Port
+====
+
+A core's interface to the DLB 2.0 is called a "port," and consists of an MMIO
+region through which the core enqueues a queue entry, and an in-memory queue
+(the "consumer queue") to which the device schedules QEs. A core enqueues a QE
+to a device queue, then the device schedules the event to a port. Software
+specifies the connection of queues and ports; i.e. for each queue, to which
+ports the device is allowed to schedule its events. The device uses a credit
+scheme to prevent overflow of the on-device queue storage.
+
+Applications interface directly with the device by mapping the port's memory
+and MMIO regions into the application's address space for enqueue and dequeue
+operations, but call into the kernel driver for configuration operations. An
+application can also be polling- or interrupt-driven; DLB 2.0 supports both
+modes of operation.
+
+Queue
+=====
+
+The device contains 32 load-balanced (i.e. capable of atomic, ordered, and
+unordered scheduling) queues and 64 directed queues. Each queue comprises 8
+internal queues, one per priority level. The internal queue that an event is
+enqueued to is selected by the event's priority field.
+
+A load-balanced queue is capable of scheduling its events to any combination of
+load-balanced ports, whereas each directed queue has one-to-one mapping with a
+directed port. There is no restriction on port or queue types when a port
+enqueues an event to a queue; that is, a load-balanced port can enqueue to a
+directed queue and vice versa.
+
+Credits
+=======
+
+The Intel DLB 2.0 uses a credit scheme to prevent overflow of the on-device
+queue storage, with separate credits for load-balanced and directed queues. A
+port spends one credit when it enqueues a QE, and one credit is replenished
+when a QE is scheduled to a consumer queue. Each scheduling domain has one pool
+of load-balanced credits and one pool of directed credits; software is
+responsible for managing the allocation and replenishment of these credits among
+the scheduling domain's ports.
+
+Scheduling Domain
+=================
+
+Device resources -- including ports, queues, and credits -- are contained
+within a scheduling domain. Scheduling domains are isolated from one another; a
+port can only enqueue to and dequeue from queues within its scheduling domain.
+
+The scheduling domain creation ioctl returns a domain file descriptor, through
+which the domain's resources are configured. For a multi-process scenario, the
+owner of this descriptor must share it with the other processes (e.g. inherited
+through fork() or shared over a unix domain socket).
+
+Consumer Queue Interrupts
+=========================
+
+Each port has its own interrupt which fires, if armed, when the consumer queue
+depth becomes non-zero. Software arms an interrupt by enqueueing a special
+'interrupt arm' command to the device through the port's MMIO window.
+
+Power Management
+================
+
+The kernel driver keeps the device in D3Hot when not in use. The driver
+transitions the device to D0 when the first device file is opened or a virtual
+function is created, and keeps it there until there are no open device files,
+memory mappings, or virtual functions.
+
+Virtualization
+==============
+
+The DLB 2.0 supports both SR-IOV and Scalable IOV, and can flexibly divide its
+resources among the physical function (PF) and its virtual devices. Virtual
+devices do not configure the device directly; they use a hardware mailbox to
+proxy configuration requests to the PF driver. Mailbox communication is
+initiated by the virtual device with a registration message that establishes
+the mailbox interface version.
+
+SR-IOV
+------
+
+Each SR-IOV virtual function (VF) has 32 interrupts, 1 for PF->VF mailbox
+messages and the remainder for CQ interrupts. If a VF user (e.g. a guest OS)
+needs more CQ interrupts, they have to use more than one VF.
+
+To support this case, the driver introduces the notion of primary and auxiliary
+VFs. A VF is either considered primary or auxiliary:
+- Primary: the VF is used as a regular DLB 2.0 device. The primary VF has 0+
+ auxiliary VFs supporting it.
+- Auxiliary: the VF doesn't have any resources of its own, and serves only to
+ provide the primary VF with MSI vectors for its CQ interrupts.
+
+Each VF has an aux_vf_ids file in its sysfs directory, which is a R/W file that
+controls the primary VF’s auxiliaries. When a VF is made auxiliary to another,
+its resources are relinquished to the PF device.
+
+When the VF driver registers its device with the PF driver, the PF driver tells
+the VF driver whether its device is auxiliary or primary, and if so then the ID
+of its primary VF. If it is auxiliary, the VF device will “claim” up to 31 of
+the primary VF’s CQs, such that they use the auxiliary VF’s MSI vectors.
+
+When a primary VF has one or more auxiliary VFs, the entire VF group must be
+assigned to the same virtual machine. The PF driver will not allow the primary
+VF to configure its resources until all its auxiliary VFs have been registered
+by the guest OS’s driver.
+
+Scalable IOV
+------------
+
+Scalable IOV is a virtualization solution that, compared to SR-IOV, enables
+highly-scalable, high-performance, and fine-grained sharing of I/O devices
+across isolated domains.
+
+In Scalable IOV, the smallest granularity of sharing a device is the Assignable Device
+Interface (ADI). Similar to SR-IOV’s Virtual Function, Scalable IOV defines the
+Virtual Device (VDEV) as the abstraction at which a Scalable IOV device is exposed to
+guest software, and a VDEV contains one or more ADIs.
+
+Kernel software is responsible for composing and managing VDEV instances in
+Scalable IOV. The device-specific software components are the Host (PF) Driver,
+the Guest (VDEV) Driver, and the Virtual Device Composition Module (VDCM). The
+VDCM is responsible for managing the software-based virtualization of (slow)
+control path operations, like the mailbox between Host and Guest drivers.
+
+For DLB 2.0, the ADI is the scheduling domain, which consists of load-balanced
+and directed queues, ports, and other resources. Each port, whether
+load-balanced or directed, consists of:
+- A producer port: a 4-KB separated MMIO window
+- A consumer queue: a memory-based ring to which the device writes queue entries
+- One CQ interrupt message
+
+DLB 2.0 supports up to 16 VDEVs per PF.
+
+For Scalable IOV guest-host communication, DLB 2.0 uses a software-based
+mailbox. This mailbox interface matches the SR-IOV hardware mailbox (i.e. PF2VF
+and VF2PF MMIO regions) except that it is backed by shared memory (allocated
+and managed by the VDCM). Similarly, the VF2PF interrupt trigger register
+instead causes a VM exit into the VDCM driver, and the PF2VF interrupt is
+replaced by a virtual interrupt injected into the guest through the VDCM.
+
+User Interface
+==============
+
+The dlb2 driver uses ioctls as its primary interface. It provides two types of
+files: the dlb2 device file and the scheduling domain file.
+
+The two types support different ioctl interfaces; the dlb2 device file is used
+for device-wide operations (including scheduling domain creation), and the
+scheduling domain device file supports operations on the scheduling domain's
+resources such as port and queue configuration.
+
+The driver also exports an mmap interface through port files, which are
+acquired through scheduling domain ioctls. This mmap interface is used to map
+a port's memory and MMIO window into the process's address space.
+
+Reset
+=====
+
+The dlb2 driver supports reset at two levels: scheduling domain and device-wide
+(i.e. FLR).
+
+Scheduling domain reset occurs when an application stops using its domain.
+Specifically, when no more file references or memory mappings exist. At this
+time, the driver resets all the domain's resources (flushes its queues and
+ports) and puts them in their respective available-resource lists for later
+use.
+
+An FLR can occur while the device is in use by user-space software, so the
+driver uses its reset_prepare callback to ensure that no applications continue
+to use the device while the FLR executes. First, the driver blocks user-space
+from executing ioctls or opening a device file, and evicts any threads blocked
+on a CQ interrupt. The driver then notifies applications and virtual functions
+that an FLR is pending, and waits for them to clean up with a timeout (default
+of 5 seconds). If the timeout expires and the device is still in use by an
+application, the driver zaps its MMIO mappings. Virtual functions, whether in
+use or not, are reset as part of a PF FLR.
+
+While PF FLR is a hardware procedure, VF FLR is a software procedure. When a
+VF FLR is triggered, this causes an interrupt to be delivered to the PF driver,
+which performs the actual reset. This consists of performing the scheduling
+domain reset operation for each of the VF's scheduling domains.
diff --git a/Documentation/misc-devices/index.rst b/Documentation/misc-devices/index.rst
index 46072ce3d7ef..653bd32316b8 100644
--- a/Documentation/misc-devices/index.rst
+++ b/Documentation/misc-devices/index.rst
@@ -17,6 +17,7 @@ fit into other categories.
ad525x_dpot
apds990x
bh1770glc
+ dlb2
eeprom
c2port
ibmvmc
diff --git a/MAINTAINERS b/MAINTAINERS
index f77df02e4121..f2b37a2c616f 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -8713,6 +8713,13 @@ L: [email protected]
S: Supported
F: arch/x86/include/asm/intel-family.h
+INTEL DYNAMIC LOAD BALANCER 2.0 DRIVER
+M: Gage Eads <[email protected]>
+S: Maintained
+F: Documentation/ABI/testing/sysfs-driver-dlb2
+F: drivers/misc/dlb*
+F: include/uapi/linux/dlb2_user.h
+
INTEL DRM DRIVERS (excluding Poulsbo, Moorestown and derivative chipsets)
M: Jani Nikula <[email protected]>
M: Joonas Lahtinen <[email protected]>
diff --git a/drivers/misc/Kconfig b/drivers/misc/Kconfig
index ce136d685d14..efe070f63f60 100644
--- a/drivers/misc/Kconfig
+++ b/drivers/misc/Kconfig
@@ -472,4 +472,5 @@ source "drivers/misc/ocxl/Kconfig"
source "drivers/misc/cardreader/Kconfig"
source "drivers/misc/habanalabs/Kconfig"
source "drivers/misc/uacce/Kconfig"
+source "drivers/misc/dlb2/Kconfig"
endmenu
diff --git a/drivers/misc/Makefile b/drivers/misc/Makefile
index c7bd01ac6291..b6afc8edea2b 100644
--- a/drivers/misc/Makefile
+++ b/drivers/misc/Makefile
@@ -57,3 +57,4 @@ obj-$(CONFIG_PVPANIC) += pvpanic.o
obj-$(CONFIG_HABANA_AI) += habanalabs/
obj-$(CONFIG_UACCE) += uacce/
obj-$(CONFIG_XILINX_SDFEC) += xilinx_sdfec.o
+obj-$(CONFIG_INTEL_DLB2) += dlb2/
diff --git a/drivers/misc/dlb2/Kconfig b/drivers/misc/dlb2/Kconfig
new file mode 100644
index 000000000000..677114247f7e
--- /dev/null
+++ b/drivers/misc/dlb2/Kconfig
@@ -0,0 +1,17 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+config INTEL_DLB2
+ tristate "Intel Dynamic Load Balancer 2.0 Driver"
+ depends on 64BIT && PCI && X86
+ help
+ This driver supports the Intel Dynamic Load Balancer 2.0 (DLB 2.0), a
+ PCIe device (PCI ID 8086:2710) that provides load-balanced,
+ prioritized scheduling of core-to-core communication.
+
+ The user-space interface is described in
+ include/uapi/linux/dlb2_user.h
+
+ To compile this driver as a module, choose M here. The module
+ will be called dlb2.
+
+ If unsure, select N.
diff --git a/drivers/misc/dlb2/Makefile b/drivers/misc/dlb2/Makefile
new file mode 100644
index 000000000000..90ae953d2a8f
--- /dev/null
+++ b/drivers/misc/dlb2/Makefile
@@ -0,0 +1,8 @@
+#
+# Makefile for the Intel(R) Dynamic Load Balancer 2.0 (dlb2.ko) driver
+#
+
+obj-$(CONFIG_INTEL_DLB2) := dlb2.o
+
+dlb2-objs := \
+ dlb2_main.o \
diff --git a/drivers/misc/dlb2/dlb2_hw_types.h b/drivers/misc/dlb2/dlb2_hw_types.h
new file mode 100644
index 000000000000..a0f908c603eb
--- /dev/null
+++ b/drivers/misc/dlb2/dlb2_hw_types.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ * Copyright(c) 2016-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_HW_TYPES_H
+#define __DLB2_HW_TYPES_H
+
+#define DLB2_MAX_NUM_VDEVS 16
+#define DLB2_MAX_NUM_DOMAINS 32
+#define DLB2_MAX_NUM_LDB_QUEUES 32 /* LDB == load-balanced */
+#define DLB2_MAX_NUM_DIR_QUEUES 64 /* DIR == directed */
+#define DLB2_MAX_NUM_LDB_PORTS 64
+#define DLB2_MAX_NUM_DIR_PORTS DLB2_MAX_NUM_DIR_QUEUES
+#define DLB2_MAX_NUM_LDB_CREDITS 8192
+#define DLB2_MAX_NUM_DIR_CREDITS 2048
+#define DLB2_MAX_NUM_HIST_LIST_ENTRIES 2048
+#define DLB2_MAX_NUM_AQED_ENTRIES 2048
+#define DLB2_MAX_NUM_QIDS_PER_LDB_CQ 8
+#define DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS 2
+#define DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES 5
+#define DLB2_QID_PRIORITIES 8
+#define DLB2_NUM_ARB_WEIGHTS 8
+#define DLB2_MAX_WEIGHT 255
+#define DLB2_NUM_COS_DOMAINS 4
+#define DLB2_MAX_CQ_COMP_CHECK_LOOPS 409600
+#define DLB2_MAX_QID_EMPTY_CHECK_LOOPS (32 * 64 * 1024 * (800 / 30))
+#define DLB2_HZ 800000000
+
+#endif /* __DLB2_HW_TYPES_H */
diff --git a/drivers/misc/dlb2/dlb2_main.c b/drivers/misc/dlb2/dlb2_main.c
new file mode 100644
index 000000000000..ffd6df788e2e
--- /dev/null
+++ b/drivers/misc/dlb2/dlb2_main.c
@@ -0,0 +1,208 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(c) 2018-2020 Intel Corporation */
+
+#include <linux/aer.h>
+#include <linux/cdev.h>
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/init.h>
+#include <linux/list.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/uaccess.h>
+
+#include "dlb2_main.h"
+
+static const char
+dlb2_driver_copyright[] = "Copyright(c) 2018-2020 Intel Corporation";
+
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Copyright(c) 2018-2020 Intel Corporation");
+MODULE_DESCRIPTION("Intel(R) Dynamic Load Balancer 2.0 Driver");
+
+/* The driver lock protects data structures that used by multiple devices. */
+static DEFINE_MUTEX(dlb2_driver_lock);
+static struct list_head dlb2_dev_list = LIST_HEAD_INIT(dlb2_dev_list);
+
+static struct class *dlb2_class;
+static dev_t dlb2_dev_number_base;
+
+/*****************************/
+/****** Devfs callbacks ******/
+/*****************************/
+
+static int dlb2_open(struct inode *i, struct file *f)
+{
+ return 0;
+}
+
+static int dlb2_close(struct inode *i, struct file *f)
+{
+ return 0;
+}
+
+static const struct file_operations dlb2_fops = {
+ .owner = THIS_MODULE,
+ .open = dlb2_open,
+ .release = dlb2_close,
+};
+
+/**********************************/
+/****** PCI driver callbacks ******/
+/**********************************/
+
+static DEFINE_IDA(dlb2_ids);
+
+static int dlb2_alloc_id(void)
+{
+ return ida_alloc_max(&dlb2_ids, DLB2_MAX_NUM_DEVICES - 1, GFP_KERNEL);
+}
+
+static void dlb2_free_id(int id)
+{
+ ida_free(&dlb2_ids, id);
+}
+
+static int dlb2_probe(struct pci_dev *pdev,
+ const struct pci_device_id *pdev_id)
+{
+ struct dlb2_dev *dlb2_dev;
+ int ret;
+
+ dlb2_dev = devm_kzalloc(&pdev->dev, sizeof(*dlb2_dev), GFP_KERNEL);
+ if (!dlb2_dev)
+ return -ENOMEM;
+
+ pci_set_drvdata(pdev, dlb2_dev);
+
+ dlb2_dev->pdev = pdev;
+
+ dlb2_dev->id = dlb2_alloc_id();
+ if (dlb2_dev->id < 0) {
+ dev_err(&pdev->dev, "probe: device ID allocation failed\n");
+
+ ret = dlb2_dev->id;
+ goto alloc_id_fail;
+ }
+
+ ret = pci_enable_device(pdev);
+ if (ret != 0) {
+ dev_err(&pdev->dev, "pci_enable_device() returned %d\n", ret);
+
+ goto pci_enable_device_fail;
+ }
+
+ ret = pci_request_regions(pdev, dlb2_driver_name);
+ if (ret != 0) {
+ dev_err(&pdev->dev,
+ "pci_request_regions(): returned %d\n", ret);
+
+ goto pci_request_regions_fail;
+ }
+
+ pci_set_master(pdev);
+
+ if (pci_enable_pcie_error_reporting(pdev))
+ dev_info(&pdev->dev, "[%s()] Failed to enable AER\n", __func__);
+
+ mutex_lock(&dlb2_driver_lock);
+ list_add(&dlb2_dev->list, &dlb2_dev_list);
+ mutex_unlock(&dlb2_driver_lock);
+
+ return 0;
+
+pci_request_regions_fail:
+ pci_disable_device(pdev);
+pci_enable_device_fail:
+ dlb2_free_id(dlb2_dev->id);
+alloc_id_fail:
+ devm_kfree(&pdev->dev, dlb2_dev);
+ return ret;
+}
+
+static void dlb2_remove(struct pci_dev *pdev)
+{
+ struct dlb2_dev *dlb2_dev;
+
+ /* Undo all the dlb2_probe() operations */
+ dlb2_dev = pci_get_drvdata(pdev);
+
+ mutex_lock(&dlb2_driver_lock);
+ list_del(&dlb2_dev->list);
+ mutex_unlock(&dlb2_driver_lock);
+
+ pci_disable_pcie_error_reporting(pdev);
+
+ pci_release_regions(pdev);
+
+ pci_disable_device(pdev);
+
+ dlb2_free_id(dlb2_dev->id);
+
+ devm_kfree(&pdev->dev, dlb2_dev);
+}
+
+static struct pci_device_id dlb2_id_table[] = {
+ { PCI_DEVICE_DATA(INTEL, DLB2_PF, NULL) },
+ { 0 }
+};
+MODULE_DEVICE_TABLE(pci, dlb2_id_table);
+
+static struct pci_driver dlb2_pci_driver = {
+ .name = (char *)dlb2_driver_name,
+ .id_table = dlb2_id_table,
+ .probe = dlb2_probe,
+ .remove = dlb2_remove,
+};
+
+static int __init dlb2_init_module(void)
+{
+ int err;
+
+ dlb2_class = class_create(THIS_MODULE, dlb2_driver_name);
+
+ if (IS_ERR(dlb2_class)) {
+ pr_err("%s: class_create() returned %ld\n",
+ dlb2_driver_name, PTR_ERR(dlb2_class));
+
+ return PTR_ERR(dlb2_class);
+ }
+
+ /* Allocate one minor number per domain */
+ err = alloc_chrdev_region(&dlb2_dev_number_base,
+ 0,
+ DLB2_MAX_NUM_DEVICES,
+ dlb2_driver_name);
+
+ if (err < 0) {
+ pr_err("%s: alloc_chrdev_region() returned %d\n",
+ dlb2_driver_name, err);
+
+ return err;
+ }
+
+ err = pci_register_driver(&dlb2_pci_driver);
+ if (err < 0) {
+ pr_err("%s: pci_register_driver() returned %d\n",
+ dlb2_driver_name, err);
+ return err;
+ }
+
+ return 0;
+}
+
+static void __exit dlb2_exit_module(void)
+{
+ pci_unregister_driver(&dlb2_pci_driver);
+
+ unregister_chrdev_region(dlb2_dev_number_base,
+ DLB2_MAX_NUM_DEVICES);
+
+ if (dlb2_class) {
+ class_destroy(dlb2_class);
+ dlb2_class = NULL;
+ }
+}
+
+module_init(dlb2_init_module);
+module_exit(dlb2_exit_module);
diff --git a/drivers/misc/dlb2/dlb2_main.h b/drivers/misc/dlb2/dlb2_main.h
new file mode 100644
index 000000000000..cc05546fba13
--- /dev/null
+++ b/drivers/misc/dlb2/dlb2_main.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ * Copyright(c) 2017-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_MAIN_H
+#define __DLB2_MAIN_H
+
+#include <linux/cdev.h>
+#include <linux/device.h>
+#include <linux/ktime.h>
+#include <linux/list.h>
+#include <linux/mutex.h>
+#include <linux/pci.h>
+#include <linux/types.h>
+
+#include "dlb2_hw_types.h"
+
+static const char dlb2_driver_name[] = KBUILD_MODNAME;
+
+/*
+ * The dlb2 driver uses a different minor number for each device file, of which
+ * there are:
+ * - 33 per device (PF or VF/VDEV): 1 for the device, 32 for scheduling domains
+ * - Up to 17 devices per PF: 1 PF and up to 16 VFs/VDEVs
+ * - Up to 16 PFs per system
+ */
+#define DLB2_MAX_NUM_PFS 16
+#define DLB2_NUM_FUNCS_PER_DEVICE (1 + DLB2_MAX_NUM_VDEVS)
+#define DLB2_MAX_NUM_DEVICES (DLB2_MAX_NUM_PFS * DLB2_NUM_FUNCS_PER_DEVICE)
+
+struct dlb2_dev {
+ struct pci_dev *pdev;
+ struct list_head list;
+ int id;
+};
+
+#endif /* __DLB2_MAIN_H */
diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
index 5c709a1450b1..eb865b4eb900 100644
--- a/include/linux/pci_ids.h
+++ b/include/linux/pci_ids.h
@@ -2809,6 +2809,8 @@
#define PCI_DEVICE_ID_INTEL_ESB2_14 0x2698
#define PCI_DEVICE_ID_INTEL_ESB2_17 0x269b
#define PCI_DEVICE_ID_INTEL_ESB2_18 0x269e
+#define PCI_DEVICE_ID_INTEL_DLB2_PF 0x2710
+#define PCI_DEVICE_ID_INTEL_DLB2_VF 0x2711
#define PCI_DEVICE_ID_INTEL_ICH7_0 0x27b8
#define PCI_DEVICE_ID_INTEL_ICH7_1 0x27b9
#define PCI_DEVICE_ID_INTEL_ICH7_30 0x27b0
--
2.13.6
Domain alerts are a mechanism for the driver to asynchronously notify
user-space applications of device reset or hardware alarms (both to be
added in later commits). This mechanism also allows the application to
enqueue an alert to its domain, as a form of (limited) IPC in a
multi-process scenario.
An application can read its domain alerts through the domain device file's
read callback. Applications are expected to spawn a thread that performs a
blocking read, and rarely (if ever) wakes and returns to user-space.
Signed-off-by: Gage Eads <[email protected]>
Reviewed-by: Björn Töpel <[email protected]>
---
drivers/misc/dlb2/dlb2_ioctl.c | 17 ++++++
drivers/misc/dlb2/dlb2_main.c | 130 +++++++++++++++++++++++++++++++++++++++
drivers/misc/dlb2/dlb2_main.h | 16 +++++
include/uapi/linux/dlb2_user.h | 134 +++++++++++++++++++++++++++++++++++++++++
4 files changed, 297 insertions(+)
diff --git a/drivers/misc/dlb2/dlb2_ioctl.c b/drivers/misc/dlb2/dlb2_ioctl.c
index 010e67941cf9..2350d8ff823e 100644
--- a/drivers/misc/dlb2/dlb2_ioctl.c
+++ b/drivers/misc/dlb2/dlb2_ioctl.c
@@ -493,6 +493,21 @@ static int dlb2_domain_ioctl_get_dir_port_cq_fd(struct dlb2_dev *dev,
"dlb2_dir_cq:", &dlb2_cq_fops, false);
}
+static int dlb2_domain_ioctl_enqueue_domain_alert(struct dlb2_dev *dev,
+ struct dlb2_domain *domain,
+ unsigned long user_arg)
+{
+ struct dlb2_enqueue_domain_alert_args arg;
+
+ if (copy_from_user(&arg, (void __user *)user_arg, sizeof(arg)))
+ return -EFAULT;
+
+ return dlb2_write_domain_alert(dev,
+ domain,
+ DLB2_DOMAIN_ALERT_USER,
+ arg.aux_alert_data);
+}
+
long dlb2_domain_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
{
struct dlb2_domain *dom = f->private_data;
@@ -537,6 +552,8 @@ long dlb2_domain_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
return dlb2_domain_ioctl_disable_dir_port(dev, dom, arg);
case DLB2_IOC_BLOCK_ON_CQ_INTERRUPT:
return dlb2_domain_ioctl_block_on_cq_interrupt(dev, dom, arg);
+ case DLB2_IOC_ENQUEUE_DOMAIN_ALERT:
+ return dlb2_domain_ioctl_enqueue_domain_alert(dev, dom, arg);
default:
return -ENOTTY;
}
diff --git a/drivers/misc/dlb2/dlb2_main.c b/drivers/misc/dlb2/dlb2_main.c
index b542c2c081a5..b457bda7be44 100644
--- a/drivers/misc/dlb2/dlb2_main.c
+++ b/drivers/misc/dlb2/dlb2_main.c
@@ -101,6 +101,9 @@ int dlb2_init_domain(struct dlb2_dev *dlb2_dev, u32 domain_id)
kref_init(&domain->refcnt);
domain->dlb2_dev = dlb2_dev;
+ spin_lock_init(&domain->alert_lock);
+ init_waitqueue_head(&domain->wq_head);
+
dlb2_dev->sched_domains[domain_id] = domain;
dlb2_dev->ops->inc_pm_refcnt(dlb2_dev->pdev, true);
@@ -227,9 +230,136 @@ static int dlb2_domain_close(struct inode *i, struct file *f)
return ret;
}
+int dlb2_write_domain_alert(struct dlb2_dev *dev,
+ struct dlb2_domain *domain,
+ u64 alert_id,
+ u64 aux_alert_data)
+{
+ struct dlb2_domain_alert alert;
+ int idx;
+
+ if (!domain)
+ return -EINVAL;
+
+ /* Grab the alert lock to access the read and write indexes */
+ spin_lock(&domain->alert_lock);
+
+ /* If there's no space for this notification, return */
+ if ((domain->alert_wr_idx - domain->alert_rd_idx) ==
+ (DLB2_DOMAIN_ALERT_RING_SIZE - 1)) {
+ spin_unlock(&domain->alert_lock);
+ return 0;
+ }
+
+ alert.alert_id = alert_id;
+ alert.aux_alert_data = aux_alert_data;
+
+ idx = domain->alert_wr_idx % DLB2_DOMAIN_ALERT_RING_SIZE;
+
+ domain->alerts[idx] = alert;
+
+ domain->alert_wr_idx++;
+
+ spin_unlock(&domain->alert_lock);
+
+ /* Wake any blocked readers */
+ wake_up_interruptible(&domain->wq_head);
+
+ return 0;
+}
+
+static bool dlb2_alerts_avail(struct dlb2_domain *domain)
+{
+ bool ret;
+
+ spin_lock(&domain->alert_lock);
+
+ ret = domain->alert_rd_idx != domain->alert_wr_idx;
+
+ spin_unlock(&domain->alert_lock);
+
+ return ret;
+}
+
+static int dlb2_read_domain_alert(struct dlb2_dev *dev,
+ struct dlb2_domain *domain,
+ struct dlb2_domain_alert *alert,
+ bool nonblock)
+{
+ int idx;
+
+ /* Grab the alert lock to access the read and write indexes */
+ spin_lock(&domain->alert_lock);
+
+ while (domain->alert_rd_idx == domain->alert_wr_idx) {
+ /*
+ * Release the alert lock before putting the thread on the wait
+ * queue.
+ */
+ spin_unlock(&domain->alert_lock);
+
+ if (nonblock)
+ return -EWOULDBLOCK;
+
+ dev_dbg(dev->dlb2_device,
+ "Thread %d is blocking waiting for an alert in domain %d\n",
+ current->pid, domain->id);
+
+ if (wait_event_interruptible(domain->wq_head,
+ dlb2_alerts_avail(domain)))
+ return -ERESTARTSYS;
+
+ spin_lock(&domain->alert_lock);
+ }
+
+ /* The alert indexes are not equal, so there is an alert available. */
+ idx = domain->alert_rd_idx % DLB2_DOMAIN_ALERT_RING_SIZE;
+
+ memcpy(alert, &domain->alerts[idx], sizeof(*alert));
+
+ domain->alert_rd_idx++;
+
+ spin_unlock(&domain->alert_lock);
+
+ return 0;
+}
+
+static ssize_t dlb2_domain_read(struct file *f,
+ char __user *buf,
+ size_t len,
+ loff_t *offset)
+{
+ struct dlb2_domain *domain = f->private_data;
+ struct dlb2_dev *dev = domain->dlb2_dev;
+ struct dlb2_domain_alert alert;
+ int ret;
+
+ if (len != sizeof(alert))
+ return -EINVAL;
+
+ /* See dlb2_user.h for details on domain alert notifications */
+
+ ret = dlb2_read_domain_alert(dev,
+ domain,
+ &alert,
+ f->f_flags & O_NONBLOCK);
+ if (ret)
+ return ret;
+
+ if (copy_to_user(buf, &alert, sizeof(alert)))
+ return -EFAULT;
+
+ dev_dbg(dev->dlb2_device,
+ "Thread %d received alert 0x%llx, with aux data 0x%llx\n",
+ current->pid, ((u64 *)&alert)[0], ((u64 *)&alert)[1]);
+
+ return sizeof(alert);
+}
+
const struct file_operations dlb2_domain_fops = {
.owner = THIS_MODULE,
.release = dlb2_domain_close,
+ .read = dlb2_domain_read,
.unlocked_ioctl = dlb2_domain_ioctl,
.compat_ioctl = compat_ptr_ioctl,
};
diff --git a/drivers/misc/dlb2/dlb2_main.h b/drivers/misc/dlb2/dlb2_main.h
index db462209fa6a..c1ae4267ff19 100644
--- a/drivers/misc/dlb2/dlb2_main.h
+++ b/drivers/misc/dlb2/dlb2_main.h
@@ -11,6 +11,7 @@
#include <linux/list.h>
#include <linux/mutex.h>
#include <linux/pci.h>
+#include <linux/spinlock.h>
#include <linux/types.h>
#include <uapi/linux/dlb2_user.h>
@@ -161,9 +162,20 @@ struct dlb2_port {
u8 valid;
};
+#define DLB2_DOMAIN_ALERT_RING_SIZE 256
+
struct dlb2_domain {
struct dlb2_dev *dlb2_dev;
+ struct dlb2_domain_alert alerts[DLB2_DOMAIN_ALERT_RING_SIZE];
+ wait_queue_head_t wq_head;
+ /*
+ * The alert lock protects access to the alert ring and its read and
+ * write indexes.
+ */
+ spinlock_t alert_lock;
struct kref refcnt;
+ u8 alert_rd_idx;
+ u8 alert_wr_idx;
u8 id;
};
@@ -225,6 +237,10 @@ struct dlb2_dev {
int dlb2_init_domain(struct dlb2_dev *dlb2_dev, u32 domain_id);
void dlb2_free_domain(struct kref *kref);
+int dlb2_write_domain_alert(struct dlb2_dev *dev,
+ struct dlb2_domain *domain,
+ u64 alert_id,
+ u64 aux_alert_data);
#define DLB2_HW_ERR(dlb2, ...) do { \
struct dlb2_dev *dev; \
diff --git a/include/uapi/linux/dlb2_user.h b/include/uapi/linux/dlb2_user.h
index 9edeff826e15..48783a8e91c2 100644
--- a/include/uapi/linux/dlb2_user.h
+++ b/include/uapi/linux/dlb2_user.h
@@ -249,6 +249,117 @@ enum dlb2_user_interface_commands {
NUM_DLB2_CMD,
};
+/*******************************/
+/* 'domain' device file alerts */
+/*******************************/
+
+/*
+ * Scheduling domain device files can be read to receive domain-specific
+ * notifications, for alerts such as hardware errors or device reset.
+ *
+ * Each alert is encoded in a 16B message. The first 8B contains the alert ID,
+ * and the second 8B is optional and contains additional information.
+ * Applications should cast read data to a struct dlb2_domain_alert, and
+ * interpret the struct's alert_id according to dlb2_domain_alert_id. The read
+ * length must be 16B, or the function will return -EINVAL.
+ *
+ * Reads are destructive, and in the case of multiple file descriptors for the
+ * same domain device file, an alert will be read by only one of the file
+ * descriptors.
+ *
+ * The driver stores alerts in a fixed-size alert ring until they are read. If
+ * the alert ring fills completely, subsequent alerts will be dropped. It is
+ * recommended that DLB2 applications dedicate a thread to perform blocking
+ * reads on the device file.
+ */
+enum dlb2_domain_alert_id {
+ /*
+ * Software issued an illegal enqueue for a port in this domain. An
+ * illegal enqueue could be:
+ * - Illegal (excess) completion
+ * - Illegal fragment
+ * - Insufficient credits
+ * aux_alert_data[7:0] contains the port ID, and aux_alert_data[15:8]
+ * contains a flag indicating whether the port is load-balanced (1) or
+ * directed (0).
+ */
+ DLB2_DOMAIN_ALERT_PP_ILLEGAL_ENQ,
+ /*
+ * Software issued excess CQ token pops for a port in this domain.
+ * aux_alert_data[7:0] contains the port ID, and aux_alert_data[15:8]
+ * contains a flag indicating whether the port is load-balanced (1) or
+ * directed (0).
+ */
+ DLB2_DOMAIN_ALERT_PP_EXCESS_TOKEN_POPS,
+ /*
+ * A enqueue contained either an invalid command encoding or a REL,
+ * REL_T, RLS, FWD, FWD_T, FRAG, or FRAG_T from a directed port.
+ *
+ * aux_alert_data[7:0] contains the port ID, and aux_alert_data[15:8]
+ * contains a flag indicating whether the port is load-balanced (1) or
+ * directed (0).
+ */
+ DLB2_DOMAIN_ALERT_ILLEGAL_HCW,
+ /*
+ * The QID must be valid and less than 128.
+ *
+ * aux_alert_data[7:0] contains the port ID, and aux_alert_data[15:8]
+ * contains a flag indicating whether the port is load-balanced (1) or
+ * directed (0).
+ */
+ DLB2_DOMAIN_ALERT_ILLEGAL_QID,
+ /*
+ * An enqueue went to a disabled QID.
+ *
+ * aux_alert_data[7:0] contains the port ID, and aux_alert_data[15:8]
+ * contains a flag indicating whether the port is load-balanced (1) or
+ * directed (0).
+ */
+ DLB2_DOMAIN_ALERT_DISABLED_QID,
+ /*
+ * The device containing this domain was reset. All applications using
+ * the device need to exit for the driver to complete the reset
+ * procedure.
+ *
+ * aux_alert_data doesn't contain any information for this alert.
+ */
+ DLB2_DOMAIN_ALERT_DEVICE_RESET,
+ /*
+ * User-space has enqueued an alert.
+ *
+ * aux_alert_data contains user-provided data.
+ */
+ DLB2_DOMAIN_ALERT_USER,
+ /*
+ * The watchdog timer fired for the specified port. This occurs if its
+ * CQ was not serviced for a large amount of time, likely indicating a
+ * hung thread.
+ * aux_alert_data[7:0] contains the port ID, and aux_alert_data[15:8]
+ * contains a flag indicating whether the port is load-balanced (1) or
+ * directed (0).
+ */
+ DLB2_DOMAIN_ALERT_CQ_WATCHDOG_TIMEOUT,
+
+ /* Number of DLB2 domain alerts */
+ NUM_DLB2_DOMAIN_ALERTS
+};
+
+static const char dlb2_domain_alert_strings[][128] = {
+ "DLB2_DOMAIN_ALERT_PP_ILLEGAL_ENQ",
+ "DLB2_DOMAIN_ALERT_PP_EXCESS_TOKEN_POPS",
+ "DLB2_DOMAIN_ALERT_ILLEGAL_HCW",
+ "DLB2_DOMAIN_ALERT_ILLEGAL_QID",
+ "DLB2_DOMAIN_ALERT_DISABLED_QID",
+ "DLB2_DOMAIN_ALERT_DEVICE_RESET",
+ "DLB2_DOMAIN_ALERT_USER",
+ "DLB2_DOMAIN_ALERT_CQ_WATCHDOG_TIMEOUT",
+};
+
+struct dlb2_domain_alert {
+ __u64 alert_id;
+ __u64 aux_alert_data;
+};
+
/*********************************/
/* 'domain' device file commands */
/*********************************/
@@ -641,6 +752,24 @@ struct dlb2_block_on_cq_interrupt_args {
__u64 cq_va;
};
+/*
+ * DLB2_DOMAIN_CMD_ENQUEUE_DOMAIN_ALERT: Enqueue a domain alert that will be
+ * read by one reader thread.
+ *
+ * Input parameters:
+ * - aux_alert_data: user-defined auxiliary data.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_enqueue_domain_alert_args {
+ /* Output parameters */
+ struct dlb2_cmd_response response;
+ /* Input parameters */
+ __u64 aux_alert_data;
+};
+
enum dlb2_domain_user_interface_commands {
DLB2_DOMAIN_CMD_CREATE_LDB_QUEUE,
DLB2_DOMAIN_CMD_CREATE_DIR_QUEUE,
@@ -661,6 +790,7 @@ enum dlb2_domain_user_interface_commands {
DLB2_DOMAIN_CMD_DISABLE_LDB_PORT,
DLB2_DOMAIN_CMD_DISABLE_DIR_PORT,
DLB2_DOMAIN_CMD_BLOCK_ON_CQ_INTERRUPT,
+ DLB2_DOMAIN_CMD_ENQUEUE_DOMAIN_ALERT,
/* NUM_DLB2_DOMAIN_CMD must be last */
NUM_DLB2_DOMAIN_CMD,
@@ -771,5 +901,9 @@ enum dlb2_domain_user_interface_commands {
_IOWR(DLB2_IOC_MAGIC, \
DLB2_DOMAIN_CMD_BLOCK_ON_CQ_INTERRUPT, \
struct dlb2_block_on_cq_interrupt_args)
+#define DLB2_IOC_ENQUEUE_DOMAIN_ALERT \
+ _IOWR(DLB2_IOC_MAGIC, \
+ DLB2_DOMAIN_CMD_ENQUEUE_DOMAIN_ALERT, \
+ struct dlb2_enqueue_domain_alert_args)
#endif /* __DLB2_USER_H */
--
2.13.6
A device FLR can be triggered while applications are actively using the
device, which poses two problems:
- If applications continue to enqueue to the hardware they will cause
hardware errors, because the FLR will have reset the scheduling domains,
ports, and queues.
- When the applications end, they must not trigger the driver's domain
reset logic, which would fail because the device's registers will have
been reset by the FLR.
The driver needs to stop applications from using the device before the FLR
begins, and detect when these applications exit (so they don't trigger the
domain reset that occurs when their last file reference (or memory mapping)
closes). The driver must also disallow applications from configuring the
device while the reset is in progress.
To avoid these problems, the driver handles unexpected resets as follows:
1. Set the reset_active flag. This flag blocks new device files from being
opened and is used as a wakeup condition in the driver's wait queues.
2. If this is a PF FLR and there are active VFs, send them a pre-reset
notification, so they can stop any VF applications. (Added in a later
commit.)
3. Disable all device files (set the per-file valid flag to false, which
prevents the file from being used after FLR completes) and wake any
threads blocked on a wait queue.
4. If the device is not in use -- i.e. no open device files or memory
mappings, and no VFs in use (PF FLR only) -- the FLR can begin.
5. Else, the driver waits (up to a user-specified timeout, default 5s) for
software to stop using the driver and the device. If the timeout
elapses, the driver zaps any remaining MMIO mappings so the
extant applications can no longer use the device.
6. Do not unlock the resource mutex at the end of the reset prepare
function, to prevent device access during the reset.
The reset timeout is exposed as a module parameter.
After the FLR:
1. Clear the per-domain pointers (the memory is freed elsewhere).
2. Release any remaining allocated port or CQ memory, now that it's
guaranteed the device is unconfigured and won't write to memory.
3. Reset software and hardware state.
4. Notify VFs that the FLR is complete. (Added in a later commit.)
5. Set reset_active to false.
6. Unlock the resource mutex.
Signed-off-by: Gage Eads <[email protected]>
Reviewed-by: Björn Töpel <[email protected]>
---
Documentation/ABI/testing/sysfs-driver-dlb2 | 22 ++
drivers/misc/dlb2/dlb2_intr.c | 13 +-
drivers/misc/dlb2/dlb2_intr.h | 1 +
drivers/misc/dlb2/dlb2_ioctl.c | 54 ++++-
drivers/misc/dlb2/dlb2_main.c | 315 +++++++++++++++++++++++++++-
drivers/misc/dlb2/dlb2_main.h | 4 +
drivers/misc/dlb2/dlb2_resource.c | 17 ++
drivers/misc/dlb2/dlb2_resource.h | 9 +
8 files changed, 424 insertions(+), 11 deletions(-)
create mode 100644 Documentation/ABI/testing/sysfs-driver-dlb2
diff --git a/Documentation/ABI/testing/sysfs-driver-dlb2 b/Documentation/ABI/testing/sysfs-driver-dlb2
new file mode 100644
index 000000000000..38d0d3d92670
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-driver-dlb2
@@ -0,0 +1,22 @@
+What: /sys/bus/pci/drivers/dlb2/module/parameters/reset_timeout_s
+Date: June 22, 2020
+KernelVersion: 5.9
+Contact: [email protected]
+Description: Interface for setting the driver's reset timeout.
+ When a device reset (FLR) is issued, the driver waits for
+ user-space to stop using the device before allowing the FLR to
+ proceed, with a timeout. The device is considered in use if
+ there are any open domain device file descriptors or memory
+ mapped producer ports. (For PF device resets, this includes all
+ VF-owned domains and producer ports.)
+
+ The amount of time the driver waits for userspace to stop using
+ the device is controlled by the module parameter
+ reset_timeout_s, which is in units of seconds and defaults to
+ 5. If reset_timeout_s seconds elapse and any user is still
+ using the device, the driver zaps those processes' memory
+ mappings and marks their device file descriptors as invalid.
+ This is necessary because user processes that do not relinquish
+ their device mappings can interfere with processes that use the
+ device after the reset completes. To ensure that user processes
+ have enough time to clean up, reset_timeout_s can be increased.
diff --git a/drivers/misc/dlb2/dlb2_intr.c b/drivers/misc/dlb2/dlb2_intr.c
index 0e20197e96fb..ace7df36294e 100644
--- a/drivers/misc/dlb2/dlb2_intr.c
+++ b/drivers/misc/dlb2/dlb2_intr.c
@@ -30,7 +30,10 @@ static inline bool wake_condition(struct dlb2_cq_intr *intr,
struct dlb2_dev *dev,
struct dlb2_domain *domain)
{
- return (READ_ONCE(intr->wake) || READ_ONCE(intr->disabled));
+ return (READ_ONCE(intr->wake) ||
+ READ_ONCE(dev->reset_active) ||
+ !READ_ONCE(domain->valid) ||
+ READ_ONCE(intr->disabled));
}
struct dlb2_dequeue_qe {
@@ -114,8 +117,12 @@ int dlb2_block_on_cq_interrupt(struct dlb2_dev *dev,
ret = wait_event_interruptible(intr->wq_head,
wake_condition(intr, dev, dom));
- if (ret == 0 && READ_ONCE(intr->disabled))
- ret = -EACCES;
+ if (ret == 0) {
+ if (READ_ONCE(dev->reset_active) || !READ_ONCE(dom->valid))
+ ret = -EINTR;
+ else if (READ_ONCE(intr->disabled))
+ ret = -EACCES;
+ }
WRITE_ONCE(intr->wake, false);
diff --git a/drivers/misc/dlb2/dlb2_intr.h b/drivers/misc/dlb2/dlb2_intr.h
index 613179795d8f..250c391d729e 100644
--- a/drivers/misc/dlb2/dlb2_intr.h
+++ b/drivers/misc/dlb2/dlb2_intr.h
@@ -20,6 +20,7 @@ int dlb2_block_on_cq_interrupt(struct dlb2_dev *dev,
enum dlb2_wake_reason {
WAKE_CQ_INTR,
WAKE_PORT_DISABLED,
+ WAKE_DEV_RESET
};
void dlb2_wake_thread(struct dlb2_dev *dev,
diff --git a/drivers/misc/dlb2/dlb2_ioctl.c b/drivers/misc/dlb2/dlb2_ioctl.c
index 3e4fb19dd726..3589acceedf8 100644
--- a/drivers/misc/dlb2/dlb2_ioctl.c
+++ b/drivers/misc/dlb2/dlb2_ioctl.c
@@ -31,6 +31,11 @@ static int dlb2_domain_ioctl_##lower_name(struct dlb2_dev *dev, \
\
mutex_lock(&dev->resource_mutex); \
\
+ if (!domain->valid) { \
+ mutex_unlock(&dev->resource_mutex); \
+ return -EINVAL; \
+ } \
+ \
ret = dev->ops->lower_name(&dev->hw, \
domain->id, \
&arg, \
@@ -74,6 +79,11 @@ static int dlb2_domain_ioctl_enable_ldb_port(struct dlb2_dev *dev,
mutex_lock(&dev->resource_mutex);
+ if (!domain->valid) {
+ mutex_unlock(&dev->resource_mutex);
+ return -EINVAL;
+ }
+
ret = dev->ops->enable_ldb_port(&dev->hw, domain->id, &arg, &response);
/* Allow threads to block on this port's CQ interrupt */
@@ -103,6 +113,11 @@ static int dlb2_domain_ioctl_enable_dir_port(struct dlb2_dev *dev,
mutex_lock(&dev->resource_mutex);
+ if (!domain->valid) {
+ mutex_unlock(&dev->resource_mutex);
+ return -EINVAL;
+ }
+
ret = dev->ops->enable_dir_port(&dev->hw, domain->id, &arg, &response);
/* Allow threads to block on this port's CQ interrupt */
@@ -132,6 +147,11 @@ static int dlb2_domain_ioctl_disable_ldb_port(struct dlb2_dev *dev,
mutex_lock(&dev->resource_mutex);
+ if (!domain->valid) {
+ mutex_unlock(&dev->resource_mutex);
+ return -EINVAL;
+ }
+
ret = dev->ops->disable_ldb_port(&dev->hw, domain->id, &arg, &response);
/*
@@ -166,6 +186,11 @@ static int dlb2_domain_ioctl_disable_dir_port(struct dlb2_dev *dev,
mutex_lock(&dev->resource_mutex);
+ if (!domain->valid) {
+ mutex_unlock(&dev->resource_mutex);
+ return -EINVAL;
+ }
+
ret = dev->ops->disable_dir_port(&dev->hw, domain->id, &arg, &response);
/*
@@ -206,6 +231,11 @@ static int dlb2_domain_ioctl_create_ldb_port(struct dlb2_dev *dev,
mutex_lock(&dev->resource_mutex);
+ if (!domain->valid) {
+ mutex_unlock(&dev->resource_mutex);
+ return -EINVAL;
+ }
+
cq_base = dma_alloc_coherent(&dev->pdev->dev,
DLB2_CQ_SIZE,
&cq_dma_base,
@@ -271,6 +301,11 @@ static int dlb2_domain_ioctl_create_dir_port(struct dlb2_dev *dev,
mutex_lock(&dev->resource_mutex);
+ if (!domain->valid) {
+ mutex_unlock(&dev->resource_mutex);
+ return -EINVAL;
+ }
+
cq_base = dma_alloc_coherent(&dev->pdev->dev,
DLB2_CQ_SIZE,
&cq_dma_base,
@@ -332,6 +367,13 @@ static int dlb2_domain_ioctl_block_on_cq_interrupt(struct dlb2_dev *dev,
if (copy_from_user(&arg, (void __user *)user_arg, sizeof(arg)))
return -EFAULT;
+ /*
+ * Note: dlb2_block_on_cq_interrupt() checks domain->valid again when
+ * it puts the thread on the waitqueue
+ */
+ if (!domain->valid)
+ return -EINVAL;
+
ret = dlb2_block_on_cq_interrupt(dev,
domain,
arg.port_id,
@@ -401,6 +443,11 @@ static int dlb2_domain_get_port_fd(struct dlb2_dev *dev,
mutex_lock(&dev->resource_mutex);
+ if (!domain->valid) {
+ mutex_unlock(&dev->resource_mutex);
+ return -EINVAL;
+ }
+
if ((is_ldb &&
dev->ops->ldb_port_owned_by_domain(&dev->hw,
domain->id,
@@ -590,6 +637,11 @@ static int dlb2_ioctl_create_sched_domain(struct dlb2_dev *dev,
mutex_lock(&dev->resource_mutex);
+ if (dev->reset_active) {
+ ret = -EINVAL;
+ goto unlock;
+ }
+
if (dev->domain_reset_failed) {
response.status = DLB2_ST_DOMAIN_RESET_FAILED;
ret = -EINVAL;
@@ -612,8 +664,6 @@ static int dlb2_ioctl_create_sched_domain(struct dlb2_dev *dev,
domain, O_RDWR);
if (fd < 0) {
- dev_err(dev->dlb2_device,
- "[%s()] Failed to get anon fd.\n", __func__);
kref_put(&domain->refcnt, dlb2_free_domain);
ret = fd;
goto unlock;
diff --git a/drivers/misc/dlb2/dlb2_main.c b/drivers/misc/dlb2/dlb2_main.c
index b457bda7be44..bc065788061a 100644
--- a/drivers/misc/dlb2/dlb2_main.c
+++ b/drivers/misc/dlb2/dlb2_main.c
@@ -12,6 +12,7 @@
#include <linux/uaccess.h>
#include "dlb2_file.h"
+#include "dlb2_intr.h"
#include "dlb2_ioctl.h"
#include "dlb2_main.h"
#include "dlb2_resource.h"
@@ -26,6 +27,11 @@ MODULE_LICENSE("GPL v2");
MODULE_AUTHOR("Copyright(c) 2018-2020 Intel Corporation");
MODULE_DESCRIPTION("Intel(R) Dynamic Load Balancer 2.0 Driver");
+static unsigned int dlb2_reset_timeout_s = DLB2_DEFAULT_RESET_TIMEOUT_S;
+module_param_named(reset_timeout_s, dlb2_reset_timeout_s, uint, 0644);
+MODULE_PARM_DESC(reset_timeout_s,
+ "Wait time (in seconds) after reset is requested given for app shutdown until driver zaps VMAs");
+
/* The driver lock protects data structures that used by multiple devices. */
static DEFINE_MUTEX(dlb2_driver_lock);
static struct list_head dlb2_dev_list = LIST_HEAD_INIT(dlb2_dev_list);
@@ -60,6 +66,10 @@ static int dlb2_open(struct inode *i, struct file *f)
dev = container_of(f->f_inode->i_cdev, struct dlb2_dev, cdev);
+ /* See dlb2_reset_prepare() for more details */
+ if (dev->reset_active)
+ return -EINVAL;
+
f->private_data = dev;
dev->ops->inc_pm_refcnt(dev->pdev, true);
@@ -98,6 +108,7 @@ int dlb2_init_domain(struct dlb2_dev *dlb2_dev, u32 domain_id)
domain->id = domain_id;
+ domain->valid = true;
kref_init(&domain->refcnt);
domain->dlb2_dev = dlb2_dev;
@@ -179,6 +190,22 @@ static int __dlb2_free_domain(struct dlb2_dev *dev, struct dlb2_domain *domain)
{
int i, ret = 0;
+ /*
+ * Check if the domain was reset and its memory released during FLR
+ * handling.
+ */
+ if (!domain->valid) {
+ /*
+ * Before clearing the sched_domains[] pointer, confirm the
+ * slot isn't in use by a newer (valid) domain.
+ */
+ if (dev->sched_domains[domain->id] == domain)
+ dev->sched_domains[domain->id] = NULL;
+
+ devm_kfree(dev->dlb2_device, domain);
+ return 0;
+ }
+
ret = dev->ops->reset_domain(&dev->hw, domain->id);
/* Unpin and free all memory pages associated with the domain */
@@ -238,7 +265,7 @@ int dlb2_write_domain_alert(struct dlb2_dev *dev,
struct dlb2_domain_alert alert;
int idx;
- if (!domain)
+ if (!domain || !domain->valid)
return -EINVAL;
/* Grab the alert lock to access the read and write indexes */
@@ -306,9 +333,16 @@ static int dlb2_read_domain_alert(struct dlb2_dev *dev,
current->pid, domain->id);
if (wait_event_interruptible(domain->wq_head,
- dlb2_alerts_avail(domain)))
+ dlb2_alerts_avail(domain) ||
+ !READ_ONCE(domain->valid)))
return -ERESTARTSYS;
+ /* See dlb2_reset_prepare() for more details */
+ if (!READ_ONCE(domain->valid)) {
+ alert->alert_id = DLB2_DOMAIN_ALERT_DEVICE_RESET;
+ return 0;
+ }
+
spin_lock(&domain->alert_lock);
}
@@ -337,6 +371,11 @@ static ssize_t dlb2_domain_read(struct file *f,
if (len != sizeof(alert))
return -EINVAL;
+ if (!domain->valid) {
+ alert.alert_id = DLB2_DOMAIN_ALERT_DEVICE_RESET;
+ goto copy;
+ }
+
/* See dlb2_user.h for details on domain alert notifications */
ret = dlb2_read_domain_alert(dev,
@@ -346,6 +385,7 @@ static ssize_t dlb2_domain_read(struct file *f,
if (ret)
return ret;
+copy:
if (copy_to_user(buf, &alert, sizeof(alert)))
return -EFAULT;
@@ -375,6 +415,11 @@ static int dlb2_pp_mmap(struct file *f, struct vm_area_struct *vma)
mutex_lock(&dev->resource_mutex);
+ if (!domain->valid) {
+ ret = -EINVAL;
+ goto end;
+ }
+
if ((vma->vm_end - vma->vm_start) != DLB2_PP_SIZE) {
ret = -EINVAL;
goto end;
@@ -411,6 +456,11 @@ static int dlb2_cq_mmap(struct file *f, struct vm_area_struct *vma)
mutex_lock(&dev->resource_mutex);
+ if (!domain->valid) {
+ ret = -EINVAL;
+ goto end;
+ }
+
if ((vma->vm_end - vma->vm_start) != DLB2_CQ_SIZE) {
ret = -EINVAL;
goto end;
@@ -661,10 +711,10 @@ static void dlb2_remove(struct pci_dev *pdev)
devm_kfree(&pdev->dev, dlb2_dev);
}
-#ifdef CONFIG_PM
-static void dlb2_reset_hardware_state(struct dlb2_dev *dev)
+static void dlb2_reset_hardware_state(struct dlb2_dev *dev, bool issue_flr)
{
- dlb2_reset_device(dev->pdev);
+ if (issue_flr)
+ dlb2_reset_device(dev->pdev);
/* Reinitialize interrupt configuration */
dev->ops->reinit_interrupts(dev);
@@ -673,6 +723,7 @@ static void dlb2_reset_hardware_state(struct dlb2_dev *dev)
dev->ops->init_hardware(dev);
}
+#ifdef CONFIG_PM
static int dlb2_runtime_suspend(struct device *dev)
{
/* Return and let the PCI subsystem put the device in D3hot. */
@@ -696,7 +747,7 @@ static int dlb2_runtime_resume(struct device *dev)
return ret;
/* Now reinitialize the device state. */
- dlb2_reset_hardware_state(dlb2_dev);
+ dlb2_reset_hardware_state(dlb2_dev, true);
return 0;
}
@@ -708,6 +759,257 @@ static struct pci_device_id dlb2_id_table[] = {
};
MODULE_DEVICE_TABLE(pci, dlb2_id_table);
+static unsigned int dlb2_total_device_file_refcnt(struct dlb2_dev *dev)
+{
+ unsigned int cnt = 0;
+ int i;
+
+ for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++)
+ if (dev->sched_domains[i])
+ cnt += kref_read(&dev->sched_domains[i]->refcnt);
+
+ return cnt;
+}
+
+static bool dlb2_in_use(struct dlb2_dev *dev)
+{
+ return dlb2_total_device_file_refcnt(dev) != 0;
+}
+
+static void dlb2_wait_for_idle(struct dlb2_dev *dlb2_dev)
+{
+ int i;
+
+ for (i = 0; i < dlb2_reset_timeout_s * 10; i++) {
+ bool idle;
+
+ mutex_lock(&dlb2_dev->resource_mutex);
+
+ /*
+ * Check for any application threads in the driver, extant
+ * mmaps, or open device files.
+ */
+ idle = !dlb2_in_use(dlb2_dev);
+
+ mutex_unlock(&dlb2_dev->resource_mutex);
+
+ if (idle)
+ return;
+
+ cond_resched();
+ msleep(100);
+ }
+
+ dev_err(dlb2_dev->dlb2_device,
+ "PF driver timed out waiting for applications to idle\n");
+}
+
+static void dlb2_unmap_all_mappings(struct dlb2_dev *dlb2_dev)
+{
+ if (dlb2_dev->inode)
+ unmap_mapping_range(dlb2_dev->inode->i_mapping, 0, 0, 1);
+}
+
+static void dlb2_disable_domain_files(struct dlb2_dev *dlb2_dev)
+{
+ int i;
+
+ /*
+ * Set all domain->valid flags to false to prevent existing device
+ * files from being used to enter the device driver.
+ */
+ for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+ if (dlb2_dev->sched_domains[i])
+ dlb2_dev->sched_domains[i]->valid = false;
+ }
+}
+
+static void dlb2_wake_threads(struct dlb2_dev *dlb2_dev)
+{
+ int i;
+
+ /*
+ * Wake any blocked device file readers. These threads will return the
+ * DLB2_DOMAIN_ALERT_DEVICE_RESET alert, and well-behaved applications
+ * will close their fds and unmap DLB memory as a result.
+ */
+ for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+ if (!dlb2_dev->sched_domains[i])
+ continue;
+
+ wake_up_interruptible(&dlb2_dev->sched_domains[i]->wq_head);
+ }
+
+ /* Wake threads blocked on a CQ interrupt */
+ for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++)
+ dlb2_wake_thread(dlb2_dev,
+ &dlb2_dev->intr.ldb_cq_intr[i],
+ WAKE_DEV_RESET);
+
+ for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++)
+ dlb2_wake_thread(dlb2_dev,
+ &dlb2_dev->intr.dir_cq_intr[i],
+ WAKE_DEV_RESET);
+}
+
+static void dlb2_stop_users(struct dlb2_dev *dlb2_dev)
+{
+ /*
+ * Disable existing domain files to prevent applications from enter the
+ * device driver through file operations. (New files can't be opened
+ * while the resource mutex is held.)
+ */
+ dlb2_disable_domain_files(dlb2_dev);
+
+ /* Wake any threads blocked in the kernel */
+ dlb2_wake_threads(dlb2_dev);
+}
+
+static void dlb2_reset_prepare(struct pci_dev *pdev)
+{
+ /*
+ * Unexpected FLR. Applications may be actively using the device at
+ * the same time, which poses two problems:
+ * - If applications continue to enqueue to the hardware they will
+ * cause hardware errors, because the FLR will have reset the
+ * scheduling domains, ports, and queues.
+ * - When the applications end, they must not trigger the driver's
+ * domain reset code. The domain reset procedure would fail because
+ * the device's registers will have been reset by the FLR.
+ *
+ * To avoid these problems, the driver handles unexpected resets as
+ * follows:
+ * 1. Set the reset_active flag. This flag blocks new device files
+ * from being opened and is used as a wakeup condition in the
+ * driver's wait queues.
+ * 2. If this is a PF FLR and there are active VFs, send them a
+ * pre-reset notification, so they can stop any VF applications.
+ * 3. Disable all device files (set the per-file valid flag to false,
+ * which prevents the file from being used after FLR completes) and
+ * wake any threads on a wait queue.
+ * 4. If the DLB is not in use -- i.e. no open device files or memory
+ * mappings, and no VFs in use (PF FLR only) -- the FLR can begin.
+ * 5. Else, the driver waits (up to a user-specified timeout, default
+ * 5s) for software to stop using the driver and the device. If the
+ * timeout elapses, the driver zaps any remaining MMIO mappings.
+ *
+ * After the FLR:
+ * 1. Clear the per-domain pointers (the memory is freed in either
+ * dlb2_close or dlb2_stop_users).
+ * 2. Release any remaining allocated port or CQ memory, now that it's
+ * guaranteed the device is unconfigured and won't write to memory.
+ * 3. Reset software and hardware state
+ * 4. Notify VFs that the FLR is complete.
+ * 5. Set reset_active to false.
+ */
+
+ struct dlb2_dev *dlb2_dev;
+
+ dlb2_dev = pci_get_drvdata(pdev);
+
+ mutex_lock(&dlb2_dev->resource_mutex);
+
+ /* Block any new device files from being opened */
+ dlb2_dev->reset_active = true;
+
+ /*
+ * If the device has 1+ VFs, even if they're not in use, it will not be
+ * suspended. To avoid having to handle two cases (reset while device
+ * suspended and reset while device active), increment the device's PM
+ * refcnt here, to guarantee that the device is in D0 for the duration
+ * of the reset.
+ */
+ dlb2_dev->ops->inc_pm_refcnt(dlb2_dev->pdev, true);
+
+ /*
+ * Stop existing applications from continuing to use the device by
+ * blocking kernel driver interfaces and waking any threads on wait
+ * queues, but don't zap VMA entries yet. If this is a PF FLR, notify
+ * any VFs of the impending FLR so they can stop their users as well.
+ */
+ dlb2_stop_users(dlb2_dev);
+
+ /* If no software is using the device, there's nothing to clean up. */
+ if (!dlb2_in_use(dlb2_dev))
+ return;
+
+ /*
+ * Release the resource mutex so threads can complete their work and
+ * exit the driver
+ */
+ mutex_unlock(&dlb2_dev->resource_mutex);
+
+ /*
+ * Wait until the device is idle or dlb2_reset_timeout_s seconds
+ * elapse. If the timeout occurs, zap any remaining VMA entries to
+ * guarantee applications can't reach the device.
+ */
+ dlb2_wait_for_idle(dlb2_dev);
+
+ mutex_lock(&dlb2_dev->resource_mutex);
+
+ if (!dlb2_in_use(dlb2_dev))
+ return;
+
+ dlb2_unmap_all_mappings(dlb2_dev);
+
+ /*
+ * Don't release resource_mutex until after the FLR occurs. This
+ * prevents applications from accessing the device during reset.
+ */
+}
+
+static void dlb2_reset_done(struct pci_dev *pdev)
+{
+ struct dlb2_dev *dlb2_dev;
+ int i;
+
+ dlb2_dev = pci_get_drvdata(pdev);
+
+ /*
+ * Clear all domain pointers, to be filled in by post-FLR applications
+ * using the device driver.
+ *
+ * Note that domain memory isn't leaked -- it is either freed during
+ * dlb2_stop_users() or in the file close callback.
+ */
+ for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++)
+ dlb2_dev->sched_domains[i] = NULL;
+
+ /*
+ * Free allocated CQ memory. These are no longer accessible to
+ * user-space: either the applications closed, or their mappings were
+ * zapped in dlb2_reset_prepare().
+ */
+ dlb2_release_device_memory(dlb2_dev);
+
+ /* Reset interrupt state */
+ for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++)
+ dlb2_dev->intr.ldb_cq_intr[i].configured = false;
+ for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++)
+ dlb2_dev->intr.dir_cq_intr[i].configured = false;
+
+ /* Reset resource allocation state */
+ dlb2_resource_reset(&dlb2_dev->hw);
+
+ /* Reset the hardware state, but don't issue an additional FLR */
+ dlb2_reset_hardware_state(dlb2_dev, false);
+
+ dlb2_dev->domain_reset_failed = false;
+
+ dlb2_dev->reset_active = false;
+
+ /* Undo the PM refcnt increment in dlb2_reset_prepare(). */
+ dlb2_dev->ops->dec_pm_refcnt(dlb2_dev->pdev);
+
+ mutex_unlock(&dlb2_dev->resource_mutex);
+}
+
+static const struct pci_error_handlers dlb2_err_handler = {
+ .reset_prepare = dlb2_reset_prepare,
+ .reset_done = dlb2_reset_done,
+};
+
#ifdef CONFIG_PM
static const struct dev_pm_ops dlb2_pm_ops = {
SET_RUNTIME_PM_OPS(dlb2_runtime_suspend, dlb2_runtime_resume, NULL)
@@ -722,6 +1024,7 @@ static struct pci_driver dlb2_pci_driver = {
#ifdef CONFIG_PM
.driver.pm = &dlb2_pm_ops,
#endif
+ .err_handler = &dlb2_err_handler,
};
static int __init dlb2_init_module(void)
diff --git a/drivers/misc/dlb2/dlb2_main.h b/drivers/misc/dlb2/dlb2_main.h
index e3d7c9362257..45050403a4be 100644
--- a/drivers/misc/dlb2/dlb2_main.h
+++ b/drivers/misc/dlb2/dlb2_main.h
@@ -31,6 +31,8 @@ static const char dlb2_driver_name[] = KBUILD_MODNAME;
#define DLB2_NUM_FUNCS_PER_DEVICE (1 + DLB2_MAX_NUM_VDEVS)
#define DLB2_MAX_NUM_DEVICES (DLB2_MAX_NUM_PFS * DLB2_NUM_FUNCS_PER_DEVICE)
+#define DLB2_DEFAULT_RESET_TIMEOUT_S 5
+
enum dlb2_device_type {
DLB2_PF,
DLB2_VF,
@@ -182,6 +184,7 @@ struct dlb2_domain {
u8 alert_rd_idx;
u8 alert_wr_idx;
u8 id;
+ u8 valid;
};
struct dlb2_cq_intr {
@@ -237,6 +240,7 @@ struct dlb2_dev {
int id;
dev_t dev_number;
u8 domain_reset_failed;
+ u8 reset_active;
u8 worker_launched;
};
diff --git a/drivers/misc/dlb2/dlb2_resource.c b/drivers/misc/dlb2/dlb2_resource.c
index f2f650bc979e..4d751a0ce1db 100644
--- a/drivers/misc/dlb2/dlb2_resource.c
+++ b/drivers/misc/dlb2/dlb2_resource.c
@@ -24,6 +24,9 @@
#define DLB2_DOM_LIST_FOR_SAFE(head, ptr, ptr_tmp) \
list_for_each_entry_safe(ptr, ptr_tmp, &(head), domain_list)
+#define DLB2_FUNC_LIST_FOR_SAFE(head, ptr, ptr_tmp) \
+ list_for_each_entry_safe(ptr, ptr_tmp, &(head), func_list)
+
/*
* The PF driver cannot assume that a register write will affect subsequent HCW
* writes. To ensure a write completes, the driver must read back a CSR. This
@@ -5327,6 +5330,20 @@ static int dlb2_domain_reset_software_state(struct dlb2_hw *hw,
return 0;
}
+void dlb2_resource_reset(struct dlb2_hw *hw)
+{
+ struct dlb2_hw_domain *domain, *next __attribute__((unused));
+ int i;
+
+ for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+ DLB2_FUNC_LIST_FOR_SAFE(hw->vdev[i].used_domains, domain, next)
+ dlb2_domain_reset_software_state(hw, domain);
+ }
+
+ DLB2_FUNC_LIST_FOR_SAFE(hw->pf.used_domains, domain, next)
+ dlb2_domain_reset_software_state(hw, domain);
+}
+
static u32 dlb2_dir_queue_depth(struct dlb2_hw *hw,
struct dlb2_dir_pq_pair *queue)
{
diff --git a/drivers/misc/dlb2/dlb2_resource.h b/drivers/misc/dlb2/dlb2_resource.h
index e325c584cb3f..258ec1c11032 100644
--- a/drivers/misc/dlb2/dlb2_resource.h
+++ b/drivers/misc/dlb2/dlb2_resource.h
@@ -37,6 +37,15 @@ int dlb2_resource_init(struct dlb2_hw *hw);
void dlb2_resource_free(struct dlb2_hw *hw);
/**
+ * dlb2_resource_reset() - reset in-use resources to their initial state
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function resets in-use resources, and makes them available for use.
+ * All resources go back to their owning function, whether a PF or a VF.
+ */
+void dlb2_resource_reset(struct dlb2_hw *hw);
+
+/**
* dlb2_hw_create_sched_domain() - create a scheduling domain
* @hw: dlb2_hw handle for a particular device.
* @args: scheduling domain creation arguments.
--
2.13.6
The DLB 2.0 supports four load-balanced port classes of service (CoS). Each
CoS receives a guaranteed percentage of the load-balanced scheduler's
bandwidth, and any unreserved bandwidth is divided among the four CoS.
These two ioctls allow applications to query CoS allocations and adjust
them as needed. These allocations are controlled by the PF driver; virtual
devices, when support for them is added, will not be able to adjust them.
Signed-off-by: Gage Eads <[email protected]>
Reviewed-by: Björn Töpel <[email protected]>
---
drivers/misc/dlb2/dlb2_ioctl.c | 54 +++++++++++++++++++++++++++++++++++
drivers/misc/dlb2/dlb2_main.h | 2 ++
drivers/misc/dlb2/dlb2_pf_ops.c | 14 ++++++++++
drivers/misc/dlb2/dlb2_resource.c | 59 +++++++++++++++++++++++++++++++++++++++
drivers/misc/dlb2/dlb2_resource.h | 28 +++++++++++++++++++
include/uapi/linux/dlb2_user.h | 52 ++++++++++++++++++++++++++++++++++
6 files changed, 209 insertions(+)
diff --git a/drivers/misc/dlb2/dlb2_ioctl.c b/drivers/misc/dlb2/dlb2_ioctl.c
index 3e5ca65f9feb..3e4fb19dd726 100644
--- a/drivers/misc/dlb2/dlb2_ioctl.c
+++ b/drivers/misc/dlb2/dlb2_ioctl.c
@@ -761,6 +761,56 @@ static int dlb2_ioctl_get_sn_occupancy(struct dlb2_dev *dev,
return ret;
}
+static int dlb2_ioctl_set_cos_bw(struct dlb2_dev *dev, unsigned long user_arg)
+{
+ struct dlb2_cmd_response response = {0};
+ struct dlb2_set_cos_bw_args arg;
+ int ret;
+
+ if (copy_from_user(&arg, (void __user *)user_arg, sizeof(arg)))
+ return -EFAULT;
+
+ mutex_lock(&dev->resource_mutex);
+
+ ret = dev->ops->set_cos_bw(&dev->hw, arg.cos_id, arg.bandwidth);
+
+ mutex_unlock(&dev->resource_mutex);
+
+ BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
+
+ if (copy_to_user((void __user *)user_arg, &response, sizeof(response)))
+ return -EFAULT;
+
+ return ret;
+}
+
+static int dlb2_ioctl_get_cos_bw(struct dlb2_dev *dev, unsigned long user_arg)
+{
+ struct dlb2_cmd_response response = {0};
+ struct dlb2_get_cos_bw_args arg;
+ int ret;
+
+ if (copy_from_user(&arg, (void __user *)user_arg, sizeof(arg)))
+ return -EFAULT;
+
+ mutex_lock(&dev->resource_mutex);
+
+ ret = dev->ops->get_cos_bw(&dev->hw, arg.cos_id);
+
+ response.id = ret;
+
+ ret = (ret > 0) ? 0 : ret;
+
+ mutex_unlock(&dev->resource_mutex);
+
+ BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
+
+ if (copy_to_user((void __user *)user_arg, &response, sizeof(response)))
+ return -EFAULT;
+
+ return ret;
+}
+
long dlb2_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
{
struct dlb2_dev *dev;
@@ -782,6 +832,10 @@ long dlb2_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
return dlb2_ioctl_get_sn_allocation(dev, arg);
case DLB2_IOC_GET_SN_OCCUPANCY:
return dlb2_ioctl_get_sn_occupancy(dev, arg);
+ case DLB2_IOC_SET_COS_BW:
+ return dlb2_ioctl_set_cos_bw(dev, arg);
+ case DLB2_IOC_GET_COS_BW:
+ return dlb2_ioctl_get_cos_bw(dev, arg);
default:
return -ENOTTY;
}
diff --git a/drivers/misc/dlb2/dlb2_main.h b/drivers/misc/dlb2/dlb2_main.h
index a7be9399255f..e3d7c9362257 100644
--- a/drivers/misc/dlb2/dlb2_main.h
+++ b/drivers/misc/dlb2/dlb2_main.h
@@ -146,6 +146,8 @@ struct dlb2_device_ops {
u32 domain_id,
struct dlb2_get_dir_queue_depth_args *args,
struct dlb2_cmd_response *resp);
+ int (*set_cos_bw)(struct dlb2_hw *hw, u32 cos_id, u8 bandwidth);
+ int (*get_cos_bw)(struct dlb2_hw *hw, u32 cos_id);
void (*init_hardware)(struct dlb2_dev *dev);
int (*query_cq_poll_mode)(struct dlb2_dev *dev,
struct dlb2_cmd_response *user_resp);
diff --git a/drivers/misc/dlb2/dlb2_pf_ops.c b/drivers/misc/dlb2/dlb2_pf_ops.c
index 337cc8a72a5c..771fd793870b 100644
--- a/drivers/misc/dlb2/dlb2_pf_ops.c
+++ b/drivers/misc/dlb2/dlb2_pf_ops.c
@@ -617,6 +617,18 @@ dlb2_pf_set_sn_allocation(struct dlb2_hw *hw, u32 group_id, u32 num)
}
static int
+dlb2_pf_set_cos_bw(struct dlb2_hw *hw, u32 cos_id, u8 bandwidth)
+{
+ return dlb2_hw_set_cos_bandwidth(hw, cos_id, bandwidth);
+}
+
+static int
+dlb2_pf_get_cos_bw(struct dlb2_hw *hw, u32 cos_id)
+{
+ return dlb2_hw_get_cos_bandwidth(hw, cos_id);
+}
+
+static int
dlb2_pf_get_sn_occupancy(struct dlb2_hw *hw, u32 group_id)
{
return dlb2_get_group_sequence_number_occupancy(hw, group_id);
@@ -667,6 +679,8 @@ struct dlb2_device_ops dlb2_pf_ops = {
.get_sn_occupancy = dlb2_pf_get_sn_occupancy,
.get_ldb_queue_depth = dlb2_pf_get_ldb_queue_depth,
.get_dir_queue_depth = dlb2_pf_get_dir_queue_depth,
+ .set_cos_bw = dlb2_pf_set_cos_bw,
+ .get_cos_bw = dlb2_pf_get_cos_bw,
.init_hardware = dlb2_pf_init_hardware,
.query_cq_poll_mode = dlb2_pf_query_cq_poll_mode,
};
diff --git a/drivers/misc/dlb2/dlb2_resource.c b/drivers/misc/dlb2/dlb2_resource.c
index 646f52a06842..f2f650bc979e 100644
--- a/drivers/misc/dlb2/dlb2_resource.c
+++ b/drivers/misc/dlb2/dlb2_resource.c
@@ -6769,6 +6769,65 @@ void dlb2_clr_pmcsr_disable(struct dlb2_hw *hw)
DLB2_CSR_WR(hw, DLB2_CFG_MSTR_CFG_PM_PMCSR_DISABLE, r0.val);
}
+int dlb2_hw_get_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id)
+{
+ if (cos_id >= DLB2_NUM_COS_DOMAINS)
+ return -EINVAL;
+
+ return hw->cos_reservation[cos_id];
+}
+
+static void dlb2_log_set_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id, u8 bw)
+{
+ DLB2_HW_DBG(hw, "DLB2 set port CoS bandwidth:\n");
+ DLB2_HW_DBG(hw, "\tCoS ID: %u\n", cos_id);
+ DLB2_HW_DBG(hw, "\tBandwidth: %u\n", bw);
+}
+
+int dlb2_hw_set_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id, u8 bandwidth)
+{
+ union dlb2_lsp_cfg_shdw_range_cos r0 = { {0} };
+ union dlb2_lsp_cfg_shdw_ctrl r1 = { {0} };
+ unsigned int i;
+ u8 total;
+
+ if (cos_id >= DLB2_NUM_COS_DOMAINS)
+ return -EINVAL;
+
+ if (bandwidth > 100)
+ return -EINVAL;
+
+ total = 0;
+
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++)
+ total += (i == cos_id) ? bandwidth : hw->cos_reservation[i];
+
+ if (total > 100)
+ return -EINVAL;
+
+ r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CFG_SHDW_RANGE_COS(cos_id));
+
+ /*
+ * Normalize the bandwidth to a value in the range 0-255. Integer
+ * division may leave unreserved scheduling slots; these will be
+ * divided among the 4 classes of service.
+ */
+ r0.field.bw_range = (bandwidth * 256) / 100;
+
+ DLB2_CSR_WR(hw, DLB2_LSP_CFG_SHDW_RANGE_COS(cos_id), r0.val);
+
+ r1.field.transfer = 1;
+
+ /* Atomically transfer the newly configured service weight */
+ DLB2_CSR_WR(hw, DLB2_LSP_CFG_SHDW_CTRL, r1.val);
+
+ dlb2_log_set_cos_bandwidth(hw, cos_id, bandwidth);
+
+ hw->cos_reservation[cos_id] = bandwidth;
+
+ return 0;
+}
+
void dlb2_hw_enable_sparse_ldb_cq_mode(struct dlb2_hw *hw)
{
union dlb2_chp_cfg_chp_csr_ctrl r0;
diff --git a/drivers/misc/dlb2/dlb2_resource.h b/drivers/misc/dlb2/dlb2_resource.h
index c1e1d677b2ae..e325c584cb3f 100644
--- a/drivers/misc/dlb2/dlb2_resource.h
+++ b/drivers/misc/dlb2/dlb2_resource.h
@@ -823,6 +823,34 @@ int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
unsigned int vf_id);
/**
+ * dlb2_hw_get_cos_bandwidth() - returns the percent of bandwidth allocated
+ * to a port class-of-service.
+ * @hw: dlb2_hw handle for a particular device.
+ * @cos_id: class-of-service ID.
+ *
+ * Return:
+ * Returns -EINVAL if cos_id is invalid, else the class' bandwidth allocation.
+ */
+int dlb2_hw_get_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id);
+
+/**
+ * dlb2_hw_set_cos_bandwidth() - set a bandwidth allocation percentage for a
+ * port class-of-service.
+ * @hw: dlb2_hw handle for a particular device.
+ * @cos_id: class-of-service ID.
+ * @bandwidth: class-of-service bandwidth.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - Invalid cos ID, bandwidth is greater than 100, or bandwidth would
+ * cause the total bandwidth across all classes of service to exceed
+ * 100%.
+ */
+int dlb2_hw_set_cos_bandwidth(struct dlb2_hw *hw, u32 cos_id, u8 bandwidth);
+
+/**
* dlb2_hw_enable_sparse_ldb_cq_mode() - enable sparse mode for load-balanced
* ports.
* @hw: dlb2_hw handle for a particular device.
diff --git a/include/uapi/linux/dlb2_user.h b/include/uapi/linux/dlb2_user.h
index a117ca71dbbc..946e01fae63d 100644
--- a/include/uapi/linux/dlb2_user.h
+++ b/include/uapi/linux/dlb2_user.h
@@ -308,6 +308,48 @@ struct dlb2_get_sn_occupancy_args {
__u32 padding0;
};
+/*
+ * DLB2_CMD_SET_COS_BW: Set a bandwidth allocation percentage for a
+ * load-balanced port class-of-service (PF only).
+ *
+ * Input parameters:
+ * - cos_id: class-of-service ID, between 0 and 3 (inclusive).
+ * - bandwidth: class-of-service bandwidth percentage. Total bandwidth
+ * percentages across all 4 classes cannot exceed 100%.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_set_cos_bw_args {
+ /* Output parameters */
+ struct dlb2_cmd_response response;
+ /* Input parameters */
+ __u32 cos_id;
+ __u32 bandwidth;
+};
+
+/*
+ * DLB2_CMD_GET_COS_BW: Get the bandwidth allocation percentage for a
+ * load-balanced port class-of-service.
+ *
+ * Input parameters:
+ * - cos_id: class-of-service ID, between 0 and 3 (inclusive).
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified class's bandwidth percentage.
+ */
+struct dlb2_get_cos_bw_args {
+ /* Output parameters */
+ struct dlb2_cmd_response response;
+ /* Input parameters */
+ __u32 cos_id;
+ __u32 padding0;
+};
+
enum dlb2_user_interface_commands {
DLB2_CMD_GET_DEVICE_VERSION,
DLB2_CMD_CREATE_SCHED_DOMAIN,
@@ -316,6 +358,8 @@ enum dlb2_user_interface_commands {
DLB2_CMD_SET_SN_ALLOCATION,
DLB2_CMD_GET_SN_ALLOCATION,
DLB2_CMD_GET_SN_OCCUPANCY,
+ DLB2_CMD_SET_COS_BW,
+ DLB2_CMD_GET_COS_BW,
/* NUM_DLB2_CMD must be last */
NUM_DLB2_CMD,
@@ -909,6 +953,14 @@ enum dlb2_domain_user_interface_commands {
_IOWR(DLB2_IOC_MAGIC, \
DLB2_CMD_GET_SN_OCCUPANCY, \
struct dlb2_get_sn_occupancy_args)
+#define DLB2_IOC_SET_COS_BW \
+ _IOWR(DLB2_IOC_MAGIC, \
+ DLB2_CMD_SET_COS_BW, \
+ struct dlb2_set_cos_bw_args)
+#define DLB2_IOC_GET_COS_BW \
+ _IOWR(DLB2_IOC_MAGIC, \
+ DLB2_CMD_GET_COS_BW, \
+ struct dlb2_get_cos_bw_args)
#define DLB2_IOC_CREATE_LDB_QUEUE \
_IOWR(DLB2_IOC_MAGIC, \
DLB2_DOMAIN_CMD_CREATE_LDB_QUEUE, \
--
2.13.6
Once a port is created, the application can mmap the corresponding DMA
memory and MMIO into user-space. This allows user-space applications to
do (performance-sensitive) enqueue and dequeue independent of the kernel
driver.
The mmap callback is only available through special port files: a producer
port (PP) file and a consumer queue (CQ) file. User-space gets an fd for
these files by calling a new ioctl, DLB2_DOMAIN_CMD_GET_{LDB,
DIR}_PORT_{PP, CQ}_FD, and passing in a port ID. If the ioctl succeeds, the
returned fd can be used to mmap that port's PP/CQ.
Device reset requires first unmapping all user-space mappings, to prevent
applications from interfering with the reset operation. To this end, the
driver uses a single inode -- allocated when the first PP/CQ file is
created, and freed when the last such file is closed -- and attaches all
port files to this common inode, as done elsewhere in Linux (e.g. cxl,
dax).
Allocating this inode requires creating a pseudo-filesystem. The driver
initializes this FS when the inode is allocated, and frees the FS after the
inode is freed.
The driver doesn't use anon_inode_getfd() for these port mmap files because
the anon inode layer uses a single inode that is shared with other kernel
components -- calling unmap_mapping_range() on that shared inode would
likely break the kernel.
Signed-off-by: Gage Eads <[email protected]>
Reviewed-by: Magnus Karlsson <[email protected]>
---
drivers/misc/dlb2/Makefile | 1 +
drivers/misc/dlb2/dlb2_file.c | 133 ++++++++++++++++++++++++++++++++
drivers/misc/dlb2/dlb2_file.h | 19 +++++
drivers/misc/dlb2/dlb2_ioctl.c | 154 ++++++++++++++++++++++++++++++++++++++
drivers/misc/dlb2/dlb2_main.c | 100 +++++++++++++++++++++++++
drivers/misc/dlb2/dlb2_main.h | 13 ++++
drivers/misc/dlb2/dlb2_pf_ops.c | 22 ++++++
drivers/misc/dlb2/dlb2_resource.c | 99 ++++++++++++++++++++++++
drivers/misc/dlb2/dlb2_resource.h | 50 +++++++++++++
include/uapi/linux/dlb2_user.h | 59 +++++++++++++++
10 files changed, 650 insertions(+)
create mode 100644 drivers/misc/dlb2/dlb2_file.c
create mode 100644 drivers/misc/dlb2/dlb2_file.h
diff --git a/drivers/misc/dlb2/Makefile b/drivers/misc/dlb2/Makefile
index 18b5498b20e6..12361461dcff 100644
--- a/drivers/misc/dlb2/Makefile
+++ b/drivers/misc/dlb2/Makefile
@@ -6,6 +6,7 @@ obj-$(CONFIG_INTEL_DLB2) := dlb2.o
dlb2-objs := \
dlb2_main.o \
+ dlb2_file.o \
dlb2_ioctl.o \
dlb2_pf_ops.o \
dlb2_resource.o \
diff --git a/drivers/misc/dlb2/dlb2_file.c b/drivers/misc/dlb2/dlb2_file.c
new file mode 100644
index 000000000000..8e73231336d7
--- /dev/null
+++ b/drivers/misc/dlb2/dlb2_file.c
@@ -0,0 +1,133 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(c) 2020 Intel Corporation */
+
+#include <linux/anon_inodes.h>
+#include <linux/file.h>
+#include <linux/module.h>
+#include <linux/mount.h>
+#include <linux/pseudo_fs.h>
+
+#include "dlb2_file.h"
+#include "dlb2_main.h"
+
+/*
+ * dlb2 tracks its memory mappings so it can revoke them when an FLR is
+ * requested and user-space cannot be allowed to access the device. To achieve
+ * that, the driver creates a single inode through which all driver-created
+ * files can share a struct address_space, and unmaps the inode's address space
+ * during the reset preparation phase. Since the anon inode layer shares its
+ * inode with multiple kernel components, we cannot use that here.
+ *
+ * Doing so requires a custom pseudo-filesystem to allocate the inode. The FS
+ * and the inode are allocated on demand when a file is created, and both are
+ * freed when the last such file is closed.
+ *
+ * This is inspired by other drivers (cxl, dax, mem) and the anon inode layer.
+ */
+static int dlb2_fs_cnt;
+static struct vfsmount *dlb2_vfs_mount;
+
+#define DLB2FS_MAGIC 0x444C4232
+static int dlb2_init_fs_context(struct fs_context *fc)
+{
+ return init_pseudo(fc, DLB2FS_MAGIC) ? 0 : -ENOMEM;
+}
+
+static struct file_system_type dlb2_fs_type = {
+ .name = "dlb2",
+ .owner = THIS_MODULE,
+ .init_fs_context = dlb2_init_fs_context,
+ .kill_sb = kill_anon_super,
+};
+
+static struct inode *dlb2_alloc_inode(struct dlb2_dev *dev)
+{
+ struct inode *inode;
+ int ret;
+
+ /* Increment the pseudo-FS's refcnt and (if not already) mount it. */
+ ret = simple_pin_fs(&dlb2_fs_type, &dlb2_vfs_mount, &dlb2_fs_cnt);
+ if (ret < 0) {
+ dev_err(dev->dlb2_device,
+ "[%s()] Cannot mount pseudo filesystem: %d\n",
+ __func__, ret);
+ return ERR_PTR(ret);
+ }
+
+ if (dlb2_fs_cnt > 1) {
+ /*
+ * Return the previously allocated inode. In this case, there
+ * is guaranteed >= 1 reference and so ihold() is safe to call.
+ */
+ ihold(dev->inode);
+ return dev->inode;
+ }
+
+ inode = alloc_anon_inode(dlb2_vfs_mount->mnt_sb);
+ if (IS_ERR(inode)) {
+ dev_err(dev->dlb2_device,
+ "[%s()] Cannot allocate inode: %d\n",
+ __func__, ret);
+ simple_release_fs(&dlb2_vfs_mount, &dlb2_fs_cnt);
+ }
+
+ dev->inode = inode;
+
+ return inode;
+}
+
+/*
+ * Decrement the inode reference count and release the FS. Intended for
+ * unwinding dlb2_alloc_inode(). Must hold the resource mutex while calling.
+ */
+static void dlb2_free_inode(struct inode *inode)
+{
+ iput(inode);
+ simple_release_fs(&dlb2_vfs_mount, &dlb2_fs_cnt);
+}
+
+/*
+ * Release the FS. Intended for use in a file_operations release callback,
+ * which decrements the inode reference count separately. Must hold the
+ * resource mutex while calling.
+ */
+void dlb2_release_fs(struct dlb2_dev *dev)
+{
+ simple_release_fs(&dlb2_vfs_mount, &dlb2_fs_cnt);
+
+ /* When the fs refcnt reaches zero, the inode has been freed */
+ if (dlb2_fs_cnt == 0)
+ dev->inode = NULL;
+}
+
+/*
+ * Allocate a file with the requested flags, file operations, and name that
+ * uses the device's shared inode. Must hold the resource mutex while calling.
+ *
+ * Caller must separately allocate an fd and install the file in that fd.
+ */
+struct file *dlb2_getfile(struct dlb2_dev *dev,
+ int flags,
+ const struct file_operations *fops,
+ const char *name)
+{
+ struct inode *inode;
+ struct file *f;
+
+ if (!try_module_get(THIS_MODULE))
+ return ERR_PTR(-ENOENT);
+
+ inode = dlb2_alloc_inode(dev);
+ if (IS_ERR(inode)) {
+ module_put(THIS_MODULE);
+ return ERR_CAST(inode);
+ }
+
+ f = alloc_file_pseudo(inode, dlb2_vfs_mount, name, flags, fops);
+ if (IS_ERR(f)) {
+ dlb2_free_inode(inode);
+ module_put(THIS_MODULE);
+ }
+
+ return f;
+}
diff --git a/drivers/misc/dlb2/dlb2_file.h b/drivers/misc/dlb2/dlb2_file.h
new file mode 100644
index 000000000000..20a3b04eb00e
--- /dev/null
+++ b/drivers/misc/dlb2/dlb2_file.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ * Copyright(c) 2020 Intel Corporation
+ */
+
+#ifndef __DLB2_FILE_H
+#define __DLB2_FILE_H
+
+#include <linux/file.h>
+
+#include "dlb2_main.h"
+
+void dlb2_release_fs(struct dlb2_dev *dev);
+
+struct file *dlb2_getfile(struct dlb2_dev *dev,
+ int flags,
+ const struct file_operations *fops,
+ const char *name);
+
+#endif /* __DLB2_FILE_H */
diff --git a/drivers/misc/dlb2/dlb2_ioctl.c b/drivers/misc/dlb2/dlb2_ioctl.c
index 7a3fea60c302..68db45842e34 100644
--- a/drivers/misc/dlb2/dlb2_ioctl.c
+++ b/drivers/misc/dlb2/dlb2_ioctl.c
@@ -6,6 +6,7 @@
#include <uapi/linux/dlb2_user.h>
+#include "dlb2_file.h"
#include "dlb2_ioctl.h"
#include "dlb2_main.h"
@@ -171,6 +172,151 @@ static int dlb2_domain_ioctl_create_dir_port(struct dlb2_dev *dev,
return ret;
}
+static int dlb2_create_port_fd(struct dlb2_dev *dev,
+ struct dlb2_domain *domain,
+ const char *prefix,
+ u32 id,
+ const struct file_operations *fops,
+ int *fd,
+ struct file **f)
+{
+ char *name;
+ int ret;
+
+ ret = get_unused_fd_flags(O_RDWR);
+ if (ret < 0)
+ return ret;
+
+ *fd = ret;
+
+ name = kasprintf(GFP_KERNEL, "%s:%d", prefix, id);
+ if (!name) {
+ put_unused_fd(*fd);
+ return -ENOMEM;
+ }
+
+ *f = dlb2_getfile(dev, O_RDWR, fops, name);
+
+ kfree(name);
+
+ if (IS_ERR(*f)) {
+ put_unused_fd(*fd);
+ return PTR_ERR(*f);
+ }
+
+ return 0;
+}
+
+static int dlb2_domain_get_port_fd(struct dlb2_dev *dev,
+ struct dlb2_domain *domain,
+ unsigned long user_arg,
+ const char *name,
+ const struct file_operations *fops,
+ bool is_ldb)
+{
+ struct dlb2_cmd_response response = {0};
+ struct dlb2_get_port_fd_args arg;
+ struct file *file = NULL;
+ struct dlb2_port *port;
+ int ret, fd;
+
+ if (copy_from_user(&arg, (void __user *)user_arg, sizeof(arg)))
+ return -EFAULT;
+
+ mutex_lock(&dev->resource_mutex);
+
+ if ((is_ldb &&
+ dev->ops->ldb_port_owned_by_domain(&dev->hw,
+ domain->id,
+ arg.port_id) != 1)) {
+ response.status = DLB2_ST_INVALID_PORT_ID;
+ ret = -EINVAL;
+ goto end;
+ }
+
+ if (!is_ldb &&
+ dev->ops->dir_port_owned_by_domain(&dev->hw,
+ domain->id,
+ arg.port_id) != 1) {
+ response.status = DLB2_ST_INVALID_PORT_ID;
+ ret = -EINVAL;
+ goto end;
+ }
+
+ port = (is_ldb) ? &dev->ldb_port[arg.port_id] :
+ &dev->dir_port[arg.port_id];
+
+ if (!port->valid) {
+ response.status = DLB2_ST_INVALID_PORT_ID;
+ ret = -EINVAL;
+ goto end;
+ }
+
+ ret = dlb2_create_port_fd(dev, domain, name, arg.port_id,
+ fops, &fd, &file);
+ if (ret < 0)
+ goto end;
+
+ file->private_data = port;
+
+ response.id = fd;
+ ret = 0;
+
+end:
+ BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
+
+ if (copy_to_user((void __user *)user_arg, &response, sizeof(response)))
+ ret = -EFAULT;
+
+ /*
+ * Save fd_install() until after the last point of failure. The domain
+ * and pm refcnt are decremented in the close callback.
+ */
+ if (ret == 0) {
+ kref_get(&domain->refcnt);
+
+ dev->ops->inc_pm_refcnt(dev->pdev, true);
+
+ fd_install(fd, file);
+ }
+
+ mutex_unlock(&dev->resource_mutex);
+
+ return ret;
+}
+
+static int dlb2_domain_ioctl_get_ldb_port_pp_fd(struct dlb2_dev *dev,
+ struct dlb2_domain *domain,
+ unsigned long user_arg)
+{
+ return dlb2_domain_get_port_fd(dev, domain, user_arg,
+ "dlb2_ldb_pp:", &dlb2_pp_fops, true);
+}
+
+static int dlb2_domain_ioctl_get_ldb_port_cq_fd(struct dlb2_dev *dev,
+ struct dlb2_domain *domain,
+ unsigned long user_arg)
+{
+ return dlb2_domain_get_port_fd(dev, domain, user_arg,
+ "dlb2_ldb_cq:", &dlb2_cq_fops, true);
+}
+
+static int dlb2_domain_ioctl_get_dir_port_pp_fd(struct dlb2_dev *dev,
+ struct dlb2_domain *domain,
+ unsigned long user_arg)
+{
+ return dlb2_domain_get_port_fd(dev, domain, user_arg,
+ "dlb2_dir_pp:", &dlb2_pp_fops, false);
+}
+
+static int dlb2_domain_ioctl_get_dir_port_cq_fd(struct dlb2_dev *dev,
+ struct dlb2_domain *domain,
+ unsigned long user_arg)
+{
+ return dlb2_domain_get_port_fd(dev, domain, user_arg,
+ "dlb2_dir_cq:", &dlb2_cq_fops, false);
+}
+
long dlb2_domain_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
{
struct dlb2_domain *dom = f->private_data;
@@ -189,6 +335,14 @@ long dlb2_domain_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
return dlb2_domain_ioctl_create_ldb_port(dev, dom, arg);
case DLB2_IOC_CREATE_DIR_PORT:
return dlb2_domain_ioctl_create_dir_port(dev, dom, arg);
+ case DLB2_IOC_GET_LDB_PORT_PP_FD:
+ return dlb2_domain_ioctl_get_ldb_port_pp_fd(dev, dom, arg);
+ case DLB2_IOC_GET_LDB_PORT_CQ_FD:
+ return dlb2_domain_ioctl_get_ldb_port_cq_fd(dev, dom, arg);
+ case DLB2_IOC_GET_DIR_PORT_PP_FD:
+ return dlb2_domain_ioctl_get_dir_port_pp_fd(dev, dom, arg);
+ case DLB2_IOC_GET_DIR_PORT_CQ_FD:
+ return dlb2_domain_ioctl_get_dir_port_cq_fd(dev, dom, arg);
default:
return -ENOTTY;
}
diff --git a/drivers/misc/dlb2/dlb2_main.c b/drivers/misc/dlb2/dlb2_main.c
index a7512832fc1c..3fe59e99b402 100644
--- a/drivers/misc/dlb2/dlb2_main.c
+++ b/drivers/misc/dlb2/dlb2_main.c
@@ -11,6 +11,7 @@
#include <linux/pci.h>
#include <linux/uaccess.h>
+#include "dlb2_file.h"
#include "dlb2_ioctl.h"
#include "dlb2_main.h"
#include "dlb2_resource.h"
@@ -226,6 +227,105 @@ const struct file_operations dlb2_domain_fops = {
.compat_ioctl = compat_ptr_ioctl,
};
+static int dlb2_pp_mmap(struct file *f, struct vm_area_struct *vma)
+{
+ struct dlb2_port *port = f->private_data;
+ struct dlb2_domain *domain = port->domain;
+ struct dlb2_dev *dev = domain->dlb2_dev;
+ unsigned long pgoff;
+ pgprot_t pgprot;
+ int ret;
+
+ mutex_lock(&dev->resource_mutex);
+
+ if ((vma->vm_end - vma->vm_start) != DLB2_PP_SIZE) {
+ ret = -EINVAL;
+ goto end;
+ }
+
+ pgprot = pgprot_noncached(vma->vm_page_prot);
+
+ pgoff = dev->hw.func_phys_addr;
+
+ if (port->is_ldb)
+ pgoff += DLB2_LDB_PP_OFFS(port->id);
+ else
+ pgoff += DLB2_DIR_PP_OFFS(port->id);
+
+ ret = io_remap_pfn_range(vma,
+ vma->vm_start,
+ pgoff >> PAGE_SHIFT,
+ vma->vm_end - vma->vm_start,
+ pgprot);
+
+end:
+ mutex_unlock(&dev->resource_mutex);
+
+ return ret;
+}
+
+static int dlb2_cq_mmap(struct file *f, struct vm_area_struct *vma)
+{
+ struct dlb2_port *port = f->private_data;
+ struct dlb2_domain *domain = port->domain;
+ struct dlb2_dev *dev = domain->dlb2_dev;
+ struct page *page;
+ int ret;
+
+ mutex_lock(&dev->resource_mutex);
+
+ if ((vma->vm_end - vma->vm_start) != DLB2_CQ_SIZE) {
+ ret = -EINVAL;
+ goto end;
+ }
+
+ page = virt_to_page(port->cq_base);
+
+ ret = remap_pfn_range(vma,
+ vma->vm_start,
+ page_to_pfn(page),
+ vma->vm_end - vma->vm_start,
+ vma->vm_page_prot);
+
+end:
+ mutex_unlock(&dev->resource_mutex);
+
+ return ret;
+}
+
+static int dlb2_port_close(struct inode *i, struct file *f)
+{
+ struct dlb2_port *port = f->private_data;
+ struct dlb2_domain *domain = port->domain;
+ struct dlb2_dev *dev = domain->dlb2_dev;
+ int ret = 0;
+
+ mutex_lock(&dev->resource_mutex);
+
+ kref_put(&domain->refcnt, dlb2_free_domain);
+
+ dev->ops->dec_pm_refcnt(dev->pdev);
+
+ /* Decrement the refcnt of the pseudo-FS used to allocate the inode */
+ dlb2_release_fs(dev);
+
+ mutex_unlock(&dev->resource_mutex);
+
+ return ret;
+}
+
+const struct file_operations dlb2_pp_fops = {
+ .owner = THIS_MODULE,
+ .release = dlb2_port_close,
+ .mmap = dlb2_pp_mmap,
+};
+
+const struct file_operations dlb2_cq_fops = {
+ .owner = THIS_MODULE,
+ .release = dlb2_port_close,
+ .mmap = dlb2_cq_mmap,
+};
+
/**********************************/
/****** PCI driver callbacks ******/
/**********************************/
diff --git a/drivers/misc/dlb2/dlb2_main.h b/drivers/misc/dlb2/dlb2_main.h
index ceb5512a104c..06b9130643cb 100644
--- a/drivers/misc/dlb2/dlb2_main.h
+++ b/drivers/misc/dlb2/dlb2_main.h
@@ -81,6 +81,12 @@ struct dlb2_device_ops {
int (*get_num_resources)(struct dlb2_hw *hw,
struct dlb2_get_num_resources_args *args);
int (*reset_domain)(struct dlb2_hw *hw, u32 domain_id);
+ int (*ldb_port_owned_by_domain)(struct dlb2_hw *hw,
+ u32 domain_id,
+ u32 port_id);
+ int (*dir_port_owned_by_domain)(struct dlb2_hw *hw,
+ u32 domain_id,
+ u32 port_id);
int (*get_ldb_queue_depth)(struct dlb2_hw *hw,
u32 domain_id,
struct dlb2_get_ldb_queue_depth_args *args,
@@ -96,6 +102,8 @@ struct dlb2_device_ops {
extern struct dlb2_device_ops dlb2_pf_ops;
extern const struct file_operations dlb2_domain_fops;
+extern const struct file_operations dlb2_pp_fops;
+extern const struct file_operations dlb2_cq_fops;
struct dlb2_port {
void *cq_base;
@@ -123,6 +131,11 @@ struct dlb2_dev {
struct dlb2_port ldb_port[DLB2_MAX_NUM_LDB_PORTS];
struct dlb2_port dir_port[DLB2_MAX_NUM_DIR_PORTS];
/*
+ * Anonymous inode used to share an address_space for all domain
+ * device file mappings.
+ */
+ struct inode *inode;
+ /*
* The resource mutex serializes access to driver data structures and
* hardware registers.
*/
diff --git a/drivers/misc/dlb2/dlb2_pf_ops.c b/drivers/misc/dlb2/dlb2_pf_ops.c
index 8556dcdff8c4..1e2e89d6945e 100644
--- a/drivers/misc/dlb2/dlb2_pf_ops.c
+++ b/drivers/misc/dlb2/dlb2_pf_ops.c
@@ -307,6 +307,26 @@ dlb2_pf_query_cq_poll_mode(struct dlb2_dev *dlb2_dev,
return 0;
}
+/**************************************/
+/****** Resource query callbacks ******/
+/**************************************/
+
+static int
+dlb2_pf_ldb_port_owned_by_domain(struct dlb2_hw *hw,
+ u32 domain_id,
+ u32 port_id)
+{
+ return dlb2_ldb_port_owned_by_domain(hw, domain_id, port_id, false, 0);
+}
+
+static int
+dlb2_pf_dir_port_owned_by_domain(struct dlb2_hw *hw,
+ u32 domain_id,
+ u32 port_id)
+{
+ return dlb2_dir_port_owned_by_domain(hw, domain_id, port_id, false, 0);
+}
+
/********************************/
/****** DLB2 PF Device Ops ******/
/********************************/
@@ -331,6 +351,8 @@ struct dlb2_device_ops dlb2_pf_ops = {
.create_dir_port = dlb2_pf_create_dir_port,
.get_num_resources = dlb2_pf_get_num_resources,
.reset_domain = dlb2_pf_reset_domain,
+ .ldb_port_owned_by_domain = dlb2_pf_ldb_port_owned_by_domain,
+ .dir_port_owned_by_domain = dlb2_pf_dir_port_owned_by_domain,
.get_ldb_queue_depth = dlb2_pf_get_ldb_queue_depth,
.get_dir_queue_depth = dlb2_pf_get_dir_queue_depth,
.init_hardware = dlb2_pf_init_hardware,
diff --git a/drivers/misc/dlb2/dlb2_resource.c b/drivers/misc/dlb2/dlb2_resource.c
index 03824fd48f60..454a1aacfaf2 100644
--- a/drivers/misc/dlb2/dlb2_resource.c
+++ b/drivers/misc/dlb2/dlb2_resource.c
@@ -237,6 +237,32 @@ static struct dlb2_hw_domain *dlb2_get_domain_from_id(struct dlb2_hw *hw,
return NULL;
}
+static struct dlb2_ldb_port *
+dlb2_get_domain_ldb_port(u32 id,
+ bool vdev_req,
+ struct dlb2_hw_domain *domain)
+{
+ struct dlb2_ldb_port *port;
+ int i;
+
+ if (id >= DLB2_MAX_NUM_LDB_PORTS)
+ return NULL;
+
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port)
+ if ((!vdev_req && port->id.phys_id == id) ||
+ (vdev_req && port->id.virt_id == id))
+ return port;
+
+ DLB2_DOM_LIST_FOR(domain->avail_ldb_ports[i], port)
+ if ((!vdev_req && port->id.phys_id == id) ||
+ (vdev_req && port->id.virt_id == id))
+ return port;
+ }
+
+ return NULL;
+}
+
static struct dlb2_dir_pq_pair *
dlb2_get_domain_used_dir_pq(u32 id,
bool vdev_req,
@@ -255,6 +281,29 @@ dlb2_get_domain_used_dir_pq(u32 id,
return NULL;
}
+static struct dlb2_dir_pq_pair *
+dlb2_get_domain_dir_pq(u32 id,
+ bool vdev_req,
+ struct dlb2_hw_domain *domain)
+{
+ struct dlb2_dir_pq_pair *port;
+
+ if (id >= DLB2_MAX_NUM_DIR_PORTS)
+ return NULL;
+
+ DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port)
+ if ((!vdev_req && port->id.phys_id == id) ||
+ (vdev_req && port->id.virt_id == id))
+ return port;
+
+ DLB2_DOM_LIST_FOR(domain->avail_dir_pq_pairs, port)
+ if ((!vdev_req && port->id.phys_id == id) ||
+ (vdev_req && port->id.virt_id == id))
+ return port;
+
+ return NULL;
+}
+
static struct dlb2_ldb_queue *
dlb2_get_ldb_queue_from_id(struct dlb2_hw *hw,
u32 id,
@@ -4949,6 +4998,56 @@ int dlb2_reset_domain(struct dlb2_hw *hw,
return 0;
}
+int dlb2_ldb_port_owned_by_domain(struct dlb2_hw *hw,
+ u32 domain_id,
+ u32 port_id,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_hw_domain *domain;
+ struct dlb2_ldb_port *port;
+
+ if (vdev_req && vdev_id >= DLB2_MAX_NUM_VDEVS)
+ return -EINVAL;
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+ if (!domain || !domain->configured)
+ return -EINVAL;
+
+ port = dlb2_get_domain_ldb_port(port_id, vdev_req, domain);
+
+ if (!port)
+ return -EINVAL;
+
+ return port->domain_id.phys_id == domain->id.phys_id;
+}
+
+int dlb2_dir_port_owned_by_domain(struct dlb2_hw *hw,
+ u32 domain_id,
+ u32 port_id,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_dir_pq_pair *port;
+ struct dlb2_hw_domain *domain;
+
+ if (vdev_req && vdev_id >= DLB2_MAX_NUM_VDEVS)
+ return -EINVAL;
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+ if (!domain || !domain->configured)
+ return -EINVAL;
+
+ port = dlb2_get_domain_dir_pq(port_id, vdev_req, domain);
+
+ if (!port)
+ return -EINVAL;
+
+ return port->domain_id.phys_id == domain->id.phys_id;
+}
+
int dlb2_hw_get_num_resources(struct dlb2_hw *hw,
struct dlb2_get_num_resources_args *arg,
bool vdev_req,
diff --git a/drivers/misc/dlb2/dlb2_resource.h b/drivers/misc/dlb2/dlb2_resource.h
index d355bc1a8f08..ffdc35ebbc1e 100644
--- a/drivers/misc/dlb2/dlb2_resource.h
+++ b/drivers/misc/dlb2/dlb2_resource.h
@@ -239,6 +239,56 @@ int dlb2_reset_domain(struct dlb2_hw *hw,
unsigned int vdev_id);
/**
+ * dlb2_ldb_port_owned_by_domain() - query whether a port is owned by a domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @port_id: port ID.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns whether a load-balanced port is owned by a specified
+ * domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 if false, 1 if true, <0 otherwise.
+ *
+ * EINVAL - Invalid domain or port ID, or the domain is not configured.
+ */
+int dlb2_ldb_port_owned_by_domain(struct dlb2_hw *hw,
+ u32 domain_id,
+ u32 port_id,
+ bool vdev_request,
+ unsigned int vdev_id);
+
+/**
+ * dlb2_dir_port_owned_by_domain() - query whether a port is owned by a domain
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @port_id: port ID.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function returns whether a directed port is owned by a specified
+ * domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 if false, 1 if true, <0 otherwise.
+ *
+ * EINVAL - Invalid domain or port ID, or the domain is not configured.
+ */
+int dlb2_dir_port_owned_by_domain(struct dlb2_hw *hw,
+ u32 domain_id,
+ u32 port_id,
+ bool vdev_request,
+ unsigned int vdev_id);
+
+/**
* dlb2_hw_get_num_resources() - query the PCI function's available resources
* @hw: dlb2_hw handle for a particular device.
* @arg: pointer to resource counts.
diff --git a/include/uapi/linux/dlb2_user.h b/include/uapi/linux/dlb2_user.h
index df7bb72096cd..898ff4df3462 100644
--- a/include/uapi/linux/dlb2_user.h
+++ b/include/uapi/linux/dlb2_user.h
@@ -422,6 +422,40 @@ struct dlb2_create_dir_port_args {
__s32 queue_id;
};
+/*
+ * DLB2_CMD_GET_LDB_PORT_PP_FD: Get file descriptor to mmap a load-balanced
+ * port's producer port (PP).
+ * DLB2_CMD_GET_LDB_PORT_CQ_FD: Get file descriptor to mmap a load-balanced
+ * port's consumer queue (CQ).
+ *
+ * The load-balanced port must have been previously created with the ioctl
+ * DLB2_CMD_CREATE_LDB_PORT. The fd is used to mmap the PP/CQ region.
+ *
+ * DLB2_CMD_GET_DIR_PORT_PP_FD: Get file descriptor to mmap a directed port's
+ * producer port (PP).
+ * DLB2_CMD_GET_DIR_PORT_CQ_FD: Get file descriptor to mmap a directed port's
+ * consumer queue (CQ).
+ *
+ * The directed port must have been previously created with the ioctl
+ * DLB2_CMD_CREATE_DIR_PORT. The fd is used to mmap PP/CQ region.
+ *
+ * Input parameters:
+ * - port_id: port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ * - response.id: fd.
+ */
+struct dlb2_get_port_fd_args {
+ /* Output parameters */
+ struct dlb2_cmd_response response;
+ /* Input parameters */
+ __u32 port_id;
+ __u32 padding0;
+};
+
enum dlb2_domain_user_interface_commands {
DLB2_DOMAIN_CMD_CREATE_LDB_QUEUE,
DLB2_DOMAIN_CMD_CREATE_DIR_QUEUE,
@@ -429,12 +463,21 @@ enum dlb2_domain_user_interface_commands {
DLB2_DOMAIN_CMD_GET_DIR_QUEUE_DEPTH,
DLB2_DOMAIN_CMD_CREATE_LDB_PORT,
DLB2_DOMAIN_CMD_CREATE_DIR_PORT,
+ DLB2_DOMAIN_CMD_GET_LDB_PORT_PP_FD,
+ DLB2_DOMAIN_CMD_GET_LDB_PORT_CQ_FD,
+ DLB2_DOMAIN_CMD_GET_DIR_PORT_PP_FD,
+ DLB2_DOMAIN_CMD_GET_DIR_PORT_CQ_FD,
/* NUM_DLB2_DOMAIN_CMD must be last */
NUM_DLB2_DOMAIN_CMD,
};
+/*
+ * Mapping sizes for memory mapping the consumer queue (CQ) memory space, and
+ * producer port (PP) MMIO space.
+ */
#define DLB2_CQ_SIZE 65536
+#define DLB2_PP_SIZE 4096
/********************/
/* dlb2 ioctl codes */
@@ -482,5 +525,21 @@ enum dlb2_domain_user_interface_commands {
_IOWR(DLB2_IOC_MAGIC, \
DLB2_DOMAIN_CMD_CREATE_DIR_PORT, \
struct dlb2_create_dir_port_args)
+#define DLB2_IOC_GET_LDB_PORT_PP_FD \
+ _IOWR(DLB2_IOC_MAGIC, \
+ DLB2_DOMAIN_CMD_GET_LDB_PORT_PP_FD, \
+ struct dlb2_get_port_fd_args)
+#define DLB2_IOC_GET_LDB_PORT_CQ_FD \
+ _IOWR(DLB2_IOC_MAGIC, \
+ DLB2_DOMAIN_CMD_GET_LDB_PORT_CQ_FD, \
+ struct dlb2_get_port_fd_args)
+#define DLB2_IOC_GET_DIR_PORT_PP_FD \
+ _IOWR(DLB2_IOC_MAGIC, \
+ DLB2_DOMAIN_CMD_GET_DIR_PORT_PP_FD, \
+ struct dlb2_get_port_fd_args)
+#define DLB2_IOC_GET_DIR_PORT_CQ_FD \
+ _IOWR(DLB2_IOC_MAGIC, \
+ DLB2_DOMAIN_CMD_GET_DIR_PORT_CQ_FD, \
+ struct dlb2_get_port_fd_args)
#endif /* __DLB2_USER_H */
--
2.13.6
The ioctls can be used to dynamically enable or disable scheduling to a
port. (By default, ports start with their scheduling enabled.) Doing so
allows software to, for example, quickly add/remove cores to/from a worker
pool.
Signed-off-by: Gage Eads <[email protected]>
Reviewed-by: Björn Töpel <[email protected]>
---
drivers/misc/dlb2/dlb2_ioctl.c | 112 +++++++++++
drivers/misc/dlb2/dlb2_main.h | 16 ++
drivers/misc/dlb2/dlb2_pf_ops.c | 40 ++++
drivers/misc/dlb2/dlb2_resource.c | 382 ++++++++++++++++++++++++++++++++++++++
drivers/misc/dlb2/dlb2_resource.h | 120 ++++++++++++
include/uapi/linux/dlb2_user.h | 92 +++++++++
6 files changed, 762 insertions(+)
diff --git a/drivers/misc/dlb2/dlb2_ioctl.c b/drivers/misc/dlb2/dlb2_ioctl.c
index 0ab4a95fb34a..490c380670cc 100644
--- a/drivers/misc/dlb2/dlb2_ioctl.c
+++ b/drivers/misc/dlb2/dlb2_ioctl.c
@@ -57,6 +57,110 @@ DLB2_DOMAIN_IOCTL_CALLBACK_TEMPLATE(unmap_qid)
DLB2_DOMAIN_IOCTL_CALLBACK_TEMPLATE(pending_port_unmaps)
/*
+ * Port enable/disable ioctls don't use the callback template macro because
+ * they have additional CQ interrupt management logic.
+ */
+static int dlb2_domain_ioctl_enable_ldb_port(struct dlb2_dev *dev,
+ struct dlb2_domain *domain,
+ unsigned long user_arg)
+{
+ struct dlb2_cmd_response response = {0};
+ struct dlb2_enable_ldb_port_args arg;
+ int ret;
+
+ if (copy_from_user(&arg, (void __user *)user_arg, sizeof(arg)))
+ return -EFAULT;
+
+ mutex_lock(&dev->resource_mutex);
+
+ ret = dev->ops->enable_ldb_port(&dev->hw, domain->id, &arg, &response);
+
+ mutex_unlock(&dev->resource_mutex);
+
+ BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
+
+ if (copy_to_user((void __user *)user_arg, &response, sizeof(response)))
+ return -EFAULT;
+
+ return ret;
+}
+
+static int dlb2_domain_ioctl_enable_dir_port(struct dlb2_dev *dev,
+ struct dlb2_domain *domain,
+ unsigned long user_arg)
+{
+ struct dlb2_cmd_response response = {0};
+ struct dlb2_enable_dir_port_args arg;
+ int ret;
+
+ if (copy_from_user(&arg, (void __user *)user_arg, sizeof(arg)))
+ return -EFAULT;
+
+ mutex_lock(&dev->resource_mutex);
+
+ ret = dev->ops->enable_dir_port(&dev->hw, domain->id, &arg, &response);
+
+ mutex_unlock(&dev->resource_mutex);
+
+ BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
+
+ if (copy_to_user((void __user *)user_arg, &response, sizeof(response)))
+ return -EFAULT;
+
+ return ret;
+}
+
+static int dlb2_domain_ioctl_disable_ldb_port(struct dlb2_dev *dev,
+ struct dlb2_domain *domain,
+ unsigned long user_arg)
+{
+ struct dlb2_cmd_response response = {0};
+ struct dlb2_disable_ldb_port_args arg;
+ int ret;
+
+ if (copy_from_user(&arg, (void __user *)user_arg, sizeof(arg)))
+ return -EFAULT;
+
+ mutex_lock(&dev->resource_mutex);
+
+ ret = dev->ops->disable_ldb_port(&dev->hw, domain->id, &arg, &response);
+
+ mutex_unlock(&dev->resource_mutex);
+
+ BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
+
+ if (copy_to_user((void __user *)user_arg, &response, sizeof(response)))
+ return -EFAULT;
+
+ return ret;
+}
+
+static int dlb2_domain_ioctl_disable_dir_port(struct dlb2_dev *dev,
+ struct dlb2_domain *domain,
+ unsigned long user_arg)
+{
+ struct dlb2_cmd_response response = {0};
+ struct dlb2_disable_dir_port_args arg;
+ int ret;
+
+ if (copy_from_user(&arg, (void __user *)user_arg, sizeof(arg)))
+ return -EFAULT;
+
+ mutex_lock(&dev->resource_mutex);
+
+ ret = dev->ops->disable_dir_port(&dev->hw, domain->id, &arg, &response);
+
+ mutex_unlock(&dev->resource_mutex);
+
+ BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
+
+ if (copy_to_user((void __user *)user_arg, &response, sizeof(response)))
+ return -EFAULT;
+
+ return ret;
+}
+
+/*
* Port creation ioctls don't use the callback template macro because they have
* a number of OS-dependent memory operations.
*/
@@ -355,6 +459,14 @@ long dlb2_domain_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
return dlb2_domain_ioctl_unmap_qid(dev, dom, arg);
case DLB2_IOC_PENDING_PORT_UNMAPS:
return dlb2_domain_ioctl_pending_port_unmaps(dev, dom, arg);
+ case DLB2_IOC_ENABLE_LDB_PORT:
+ return dlb2_domain_ioctl_enable_ldb_port(dev, dom, arg);
+ case DLB2_IOC_ENABLE_DIR_PORT:
+ return dlb2_domain_ioctl_enable_dir_port(dev, dom, arg);
+ case DLB2_IOC_DISABLE_LDB_PORT:
+ return dlb2_domain_ioctl_disable_ldb_port(dev, dom, arg);
+ case DLB2_IOC_DISABLE_DIR_PORT:
+ return dlb2_domain_ioctl_disable_dir_port(dev, dom, arg);
default:
return -ENOTTY;
}
diff --git a/drivers/misc/dlb2/dlb2_main.h b/drivers/misc/dlb2/dlb2_main.h
index 626a3311eb18..92e182603445 100644
--- a/drivers/misc/dlb2/dlb2_main.h
+++ b/drivers/misc/dlb2/dlb2_main.h
@@ -94,6 +94,22 @@ struct dlb2_device_ops {
u32 domain_id,
struct dlb2_pending_port_unmaps_args *args,
struct dlb2_cmd_response *resp);
+ int (*enable_ldb_port)(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_enable_ldb_port_args *args,
+ struct dlb2_cmd_response *resp);
+ int (*disable_ldb_port)(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_disable_ldb_port_args *args,
+ struct dlb2_cmd_response *resp);
+ int (*enable_dir_port)(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_enable_dir_port_args *args,
+ struct dlb2_cmd_response *resp);
+ int (*disable_dir_port)(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_disable_dir_port_args *args,
+ struct dlb2_cmd_response *resp);
int (*get_num_resources)(struct dlb2_hw *hw,
struct dlb2_get_num_resources_args *args);
int (*reset_domain)(struct dlb2_hw *hw, u32 domain_id);
diff --git a/drivers/misc/dlb2/dlb2_pf_ops.c b/drivers/misc/dlb2/dlb2_pf_ops.c
index b531448d1a25..8b1aac196073 100644
--- a/drivers/misc/dlb2/dlb2_pf_ops.c
+++ b/drivers/misc/dlb2/dlb2_pf_ops.c
@@ -299,6 +299,42 @@ dlb2_pf_pending_port_unmaps(struct dlb2_hw *hw,
}
static int
+dlb2_pf_enable_ldb_port(struct dlb2_hw *hw,
+ u32 id,
+ struct dlb2_enable_ldb_port_args *args,
+ struct dlb2_cmd_response *resp)
+{
+ return dlb2_hw_enable_ldb_port(hw, id, args, resp, false, 0);
+}
+
+static int
+dlb2_pf_disable_ldb_port(struct dlb2_hw *hw,
+ u32 id,
+ struct dlb2_disable_ldb_port_args *args,
+ struct dlb2_cmd_response *resp)
+{
+ return dlb2_hw_disable_ldb_port(hw, id, args, resp, false, 0);
+}
+
+static int
+dlb2_pf_enable_dir_port(struct dlb2_hw *hw,
+ u32 id,
+ struct dlb2_enable_dir_port_args *args,
+ struct dlb2_cmd_response *resp)
+{
+ return dlb2_hw_enable_dir_port(hw, id, args, resp, false, 0);
+}
+
+static int
+dlb2_pf_disable_dir_port(struct dlb2_hw *hw,
+ u32 id,
+ struct dlb2_disable_dir_port_args *args,
+ struct dlb2_cmd_response *resp)
+{
+ return dlb2_hw_disable_dir_port(hw, id, args, resp, false, 0);
+}
+
+static int
dlb2_pf_get_num_resources(struct dlb2_hw *hw,
struct dlb2_get_num_resources_args *args)
{
@@ -389,6 +425,10 @@ struct dlb2_device_ops dlb2_pf_ops = {
.map_qid = dlb2_pf_map_qid,
.unmap_qid = dlb2_pf_unmap_qid,
.pending_port_unmaps = dlb2_pf_pending_port_unmaps,
+ .enable_ldb_port = dlb2_pf_enable_ldb_port,
+ .enable_dir_port = dlb2_pf_enable_dir_port,
+ .disable_ldb_port = dlb2_pf_disable_ldb_port,
+ .disable_dir_port = dlb2_pf_disable_dir_port,
.get_num_resources = dlb2_pf_get_num_resources,
.reset_domain = dlb2_pf_reset_domain,
.ldb_port_owned_by_domain = dlb2_pf_ldb_port_owned_by_domain,
diff --git a/drivers/misc/dlb2/dlb2_resource.c b/drivers/misc/dlb2/dlb2_resource.c
index 6afac8c99c06..665841ca7919 100644
--- a/drivers/misc/dlb2/dlb2_resource.c
+++ b/drivers/misc/dlb2/dlb2_resource.c
@@ -1707,6 +1707,150 @@ static int dlb2_verify_unmap_qid_args(struct dlb2_hw *hw,
return -EINVAL;
}
+static int
+dlb2_verify_enable_ldb_port_args(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_enable_ldb_port_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_hw_domain *domain;
+ struct dlb2_ldb_port *port;
+ int id;
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+ if (!domain) {
+ resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+ return -EINVAL;
+ }
+
+ if (!domain->configured) {
+ resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+ return -EINVAL;
+ }
+
+ id = args->port_id;
+
+ port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+ if (!port || !port->configured) {
+ resp->status = DLB2_ST_INVALID_PORT_ID;
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int
+dlb2_verify_enable_dir_port_args(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_enable_dir_port_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_hw_domain *domain;
+ struct dlb2_dir_pq_pair *port;
+ int id;
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+ if (!domain) {
+ resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+ return -EINVAL;
+ }
+
+ if (!domain->configured) {
+ resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+ return -EINVAL;
+ }
+
+ id = args->port_id;
+
+ port = dlb2_get_domain_used_dir_pq(id, vdev_req, domain);
+
+ if (!port || !port->port_configured) {
+ resp->status = DLB2_ST_INVALID_PORT_ID;
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int
+dlb2_verify_disable_ldb_port_args(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_disable_ldb_port_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_hw_domain *domain;
+ struct dlb2_ldb_port *port;
+ int id;
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+ if (!domain) {
+ resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+ return -EINVAL;
+ }
+
+ if (!domain->configured) {
+ resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+ return -EINVAL;
+ }
+
+ id = args->port_id;
+
+ port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+ if (!port || !port->configured) {
+ resp->status = DLB2_ST_INVALID_PORT_ID;
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int
+dlb2_verify_disable_dir_port_args(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_disable_dir_port_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_hw_domain *domain;
+ struct dlb2_dir_pq_pair *port;
+ int id;
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+ if (!domain) {
+ resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+ return -EINVAL;
+ }
+
+ if (!domain->configured) {
+ resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+ return -EINVAL;
+ }
+
+ id = args->port_id;
+
+ port = dlb2_get_domain_used_dir_pq(id, vdev_req, domain);
+
+ if (!port || !port->port_configured) {
+ resp->status = DLB2_ST_INVALID_PORT_ID;
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
static void dlb2_configure_domain_credits(struct dlb2_hw *hw,
struct dlb2_hw_domain *domain)
{
@@ -4303,6 +4447,244 @@ int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
return 0;
}
+static void dlb2_log_enable_port(struct dlb2_hw *hw,
+ u32 domain_id,
+ u32 port_id,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ DLB2_HW_DBG(hw, "DLB2 enable port arguments:\n");
+ if (vdev_req)
+ DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+ DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+ domain_id);
+ DLB2_HW_DBG(hw, "\tPort ID: %d\n",
+ port_id);
+}
+
+int dlb2_hw_enable_ldb_port(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_enable_ldb_port_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_hw_domain *domain;
+ struct dlb2_ldb_port *port;
+ int ret, id;
+
+ dlb2_log_enable_port(hw, domain_id, args->port_id, vdev_req, vdev_id);
+
+ /*
+ * Verify that hardware resources are available before attempting to
+ * satisfy the request. This simplifies the error unwinding code.
+ */
+ ret = dlb2_verify_enable_ldb_port_args(hw,
+ domain_id,
+ args,
+ resp,
+ vdev_req,
+ vdev_id);
+ if (ret)
+ return ret;
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+ id = args->port_id;
+
+ port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+ /* Hardware requires disabling the CQ before unmapping QIDs. */
+ if (!port->enabled) {
+ dlb2_ldb_port_cq_enable(hw, port);
+ port->enabled = true;
+ }
+
+ resp->status = 0;
+
+ return 0;
+}
+
+static void dlb2_log_disable_port(struct dlb2_hw *hw,
+ u32 domain_id,
+ u32 port_id,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ DLB2_HW_DBG(hw, "DLB2 disable port arguments:\n");
+ if (vdev_req)
+ DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+ DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+ domain_id);
+ DLB2_HW_DBG(hw, "\tPort ID: %d\n",
+ port_id);
+}
+
+int dlb2_hw_disable_ldb_port(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_disable_ldb_port_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_hw_domain *domain;
+ struct dlb2_ldb_port *port;
+ int ret, id;
+
+ dlb2_log_disable_port(hw, domain_id, args->port_id, vdev_req, vdev_id);
+
+ /*
+ * Verify that hardware resources are available before attempting to
+ * satisfy the request. This simplifies the error unwinding code.
+ */
+ ret = dlb2_verify_disable_ldb_port_args(hw,
+ domain_id,
+ args,
+ resp,
+ vdev_req,
+ vdev_id);
+ if (ret)
+ return ret;
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+ if (!domain) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: domain not found\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ id = args->port_id;
+
+ port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+ if (!port) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: port not found\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ /* Hardware requires disabling the CQ before unmapping QIDs. */
+ if (port->enabled) {
+ dlb2_ldb_port_cq_disable(hw, port);
+ port->enabled = false;
+ }
+
+ resp->status = 0;
+
+ return 0;
+}
+
+int dlb2_hw_enable_dir_port(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_enable_dir_port_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_dir_pq_pair *port;
+ struct dlb2_hw_domain *domain;
+ int ret, id;
+
+ dlb2_log_enable_port(hw, domain_id, args->port_id, vdev_req, vdev_id);
+
+ /*
+ * Verify that hardware resources are available before attempting to
+ * satisfy the request. This simplifies the error unwinding code.
+ */
+ ret = dlb2_verify_enable_dir_port_args(hw,
+ domain_id,
+ args,
+ resp,
+ vdev_req,
+ vdev_id);
+ if (ret)
+ return ret;
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+ if (!domain) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: domain not found\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ id = args->port_id;
+
+ port = dlb2_get_domain_used_dir_pq(id, vdev_req, domain);
+ if (!port) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: port not found\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ /* Hardware requires disabling the CQ before unmapping QIDs. */
+ if (!port->enabled) {
+ dlb2_dir_port_cq_enable(hw, port);
+ port->enabled = true;
+ }
+
+ resp->status = 0;
+
+ return 0;
+}
+
+int dlb2_hw_disable_dir_port(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_disable_dir_port_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_dir_pq_pair *port;
+ struct dlb2_hw_domain *domain;
+ int ret, id;
+
+ dlb2_log_disable_port(hw, domain_id, args->port_id, vdev_req, vdev_id);
+
+ /*
+ * Verify that hardware resources are available before attempting to
+ * satisfy the request. This simplifies the error unwinding code.
+ */
+ ret = dlb2_verify_disable_dir_port_args(hw,
+ domain_id,
+ args,
+ resp,
+ vdev_req,
+ vdev_id);
+ if (ret)
+ return ret;
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+ if (!domain) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: domain not found\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ id = args->port_id;
+
+ port = dlb2_get_domain_used_dir_pq(id, vdev_req, domain);
+ if (!port) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: port not found\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ /* Hardware requires disabling the CQ before unmapping QIDs. */
+ if (port->enabled) {
+ dlb2_dir_port_cq_disable(hw, port);
+ port->enabled = false;
+ }
+
+ resp->status = 0;
+
+ return 0;
+}
+
static u32 dlb2_ldb_cq_inflight_count(struct dlb2_hw *hw,
struct dlb2_ldb_port *port)
{
diff --git a/drivers/misc/dlb2/dlb2_resource.h b/drivers/misc/dlb2/dlb2_resource.h
index 4843da49935f..a78a8664eabc 100644
--- a/drivers/misc/dlb2/dlb2_resource.h
+++ b/drivers/misc/dlb2/dlb2_resource.h
@@ -325,6 +325,126 @@ int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
unsigned int vdev_id);
/**
+ * dlb2_hw_enable_ldb_port() - enable a load-balanced port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port enable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs to a load-balanced port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_enable_ldb_port(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_enable_ldb_port_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_request,
+ unsigned int vdev_id);
+
+/**
+ * dlb2_hw_disable_ldb_port() - disable a load-balanced port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port disable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs to a load-balanced
+ * port. Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_disable_ldb_port(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_disable_ldb_port_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_request,
+ unsigned int vdev_id);
+
+/**
+ * dlb2_hw_enable_dir_port() - enable a directed port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port enable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs to a directed port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_enable_dir_port(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_enable_dir_port_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_request,
+ unsigned int vdev_id);
+
+/**
+ * dlb2_hw_disable_dir_port() - disable a directed port for scheduling
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: port disable arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs to a directed port.
+ * Ports are enabled by default.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid or the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_disable_dir_port(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_disable_dir_port_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_request,
+ unsigned int vdev_id);
+
+/**
* dlb2_reset_domain() - reset a scheduling domain
* @hw: dlb2_hw handle for a particular device.
* @domain_id: domain ID.
diff --git a/include/uapi/linux/dlb2_user.h b/include/uapi/linux/dlb2_user.h
index 1fdc7627c393..00e9833bae1b 100644
--- a/include/uapi/linux/dlb2_user.h
+++ b/include/uapi/linux/dlb2_user.h
@@ -537,6 +537,78 @@ struct dlb2_pending_port_unmaps_args {
__u32 padding0;
};
+/*
+ * DLB2_DOMAIN_CMD_ENABLE_LDB_PORT: Enable scheduling to a load-balanced port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_enable_ldb_port_args {
+ /* Output parameters */
+ struct dlb2_cmd_response response;
+ /* Input parameters */
+ __u32 port_id;
+ __u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_ENABLE_DIR_PORT: Enable scheduling to a directed port.
+ * Input parameters:
+ * - port_id: Directed port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_enable_dir_port_args {
+ /* Output parameters */
+ struct dlb2_cmd_response response;
+ /* Input parameters */
+ __u32 port_id;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DISABLE_LDB_PORT: Disable scheduling to a load-balanced
+ * port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_disable_ldb_port_args {
+ /* Output parameters */
+ struct dlb2_cmd_response response;
+ /* Input parameters */
+ __u32 port_id;
+ __u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_DISABLE_DIR_PORT: Disable scheduling to a directed port.
+ * Input parameters:
+ * - port_id: Directed port ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_disable_dir_port_args {
+ /* Output parameters */
+ struct dlb2_cmd_response response;
+ /* Input parameters */
+ __u32 port_id;
+ __u32 padding0;
+};
+
enum dlb2_domain_user_interface_commands {
DLB2_DOMAIN_CMD_CREATE_LDB_QUEUE,
DLB2_DOMAIN_CMD_CREATE_DIR_QUEUE,
@@ -552,6 +624,10 @@ enum dlb2_domain_user_interface_commands {
DLB2_DOMAIN_CMD_MAP_QID,
DLB2_DOMAIN_CMD_UNMAP_QID,
DLB2_DOMAIN_CMD_PENDING_PORT_UNMAPS,
+ DLB2_DOMAIN_CMD_ENABLE_LDB_PORT,
+ DLB2_DOMAIN_CMD_ENABLE_DIR_PORT,
+ DLB2_DOMAIN_CMD_DISABLE_LDB_PORT,
+ DLB2_DOMAIN_CMD_DISABLE_DIR_PORT,
/* NUM_DLB2_DOMAIN_CMD must be last */
NUM_DLB2_DOMAIN_CMD,
@@ -642,5 +718,21 @@ enum dlb2_domain_user_interface_commands {
_IOWR(DLB2_IOC_MAGIC, \
DLB2_DOMAIN_CMD_PENDING_PORT_UNMAPS, \
struct dlb2_pending_port_unmaps_args)
+#define DLB2_IOC_ENABLE_LDB_PORT \
+ _IOWR(DLB2_IOC_MAGIC, \
+ DLB2_DOMAIN_CMD_ENABLE_LDB_PORT, \
+ struct dlb2_enable_ldb_port_args)
+#define DLB2_IOC_ENABLE_DIR_PORT \
+ _IOWR(DLB2_IOC_MAGIC, \
+ DLB2_DOMAIN_CMD_ENABLE_DIR_PORT, \
+ struct dlb2_enable_dir_port_args)
+#define DLB2_IOC_DISABLE_LDB_PORT \
+ _IOWR(DLB2_IOC_MAGIC, \
+ DLB2_DOMAIN_CMD_DISABLE_LDB_PORT, \
+ struct dlb2_disable_ldb_port_args)
+#define DLB2_IOC_DISABLE_DIR_PORT \
+ _IOWR(DLB2_IOC_MAGIC, \
+ DLB2_DOMAIN_CMD_DISABLE_DIR_PORT, \
+ struct dlb2_disable_dir_port_args)
#endif /* __DLB2_USER_H */
--
2.13.6
In order for a load-balanced DLB 2.0 queue to support ordered scheduling,
it must be configured with an allocation of sequence numbers (SNs) -- a
hardware resource used to re-order QEs.
The device evenly partitions its SNs across two groups. A queue is
allocated SNs by taking a 'slot' in one of these two groups, and the
number of slots is variable. A group can support as many as 16 slots (64
SNs per slot) and as few as 1 slot (1024 SNs per slot).
This commit adds ioctls to get and set a sequence number group's slot
configuration, as well as query the group's slot occupancy.
Signed-off-by: Gage Eads <[email protected]>
Reviewed-by: Björn Töpel <[email protected]>
---
drivers/misc/dlb2/dlb2_ioctl.c | 86 +++++++++++++++++++++++++++++++++++++++
drivers/misc/dlb2/dlb2_main.h | 3 ++
drivers/misc/dlb2/dlb2_pf_ops.c | 21 ++++++++++
drivers/misc/dlb2/dlb2_resource.c | 67 ++++++++++++++++++++++++++++++
drivers/misc/dlb2/dlb2_resource.h | 47 +++++++++++++++++++++
include/uapi/linux/dlb2_user.h | 84 ++++++++++++++++++++++++++++++++++++++
6 files changed, 308 insertions(+)
diff --git a/drivers/misc/dlb2/dlb2_ioctl.c b/drivers/misc/dlb2/dlb2_ioctl.c
index 2350d8ff823e..3e5ca65f9feb 100644
--- a/drivers/misc/dlb2/dlb2_ioctl.c
+++ b/drivers/misc/dlb2/dlb2_ioctl.c
@@ -681,6 +681,86 @@ static int dlb2_ioctl_query_cq_poll_mode(struct dlb2_dev *dev,
return ret;
}
+static int dlb2_ioctl_set_sn_allocation(struct dlb2_dev *dev,
+ unsigned long user_arg)
+{
+ struct dlb2_cmd_response response = {0};
+ struct dlb2_set_sn_allocation_args arg;
+ int ret;
+
+ if (copy_from_user(&arg, (void __user *)user_arg, sizeof(arg)))
+ return -EFAULT;
+
+ mutex_lock(&dev->resource_mutex);
+
+ ret = dev->ops->set_sn_allocation(&dev->hw, arg.group, arg.num);
+
+ mutex_unlock(&dev->resource_mutex);
+
+ BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
+
+ if (copy_to_user((void __user *)user_arg, &response, sizeof(response)))
+ return -EFAULT;
+
+ return ret;
+}
+
+static int dlb2_ioctl_get_sn_allocation(struct dlb2_dev *dev,
+ unsigned long user_arg)
+{
+ struct dlb2_cmd_response response = {0};
+ struct dlb2_get_sn_allocation_args arg;
+ int ret;
+
+ if (copy_from_user(&arg, (void __user *)user_arg, sizeof(arg)))
+ return -EFAULT;
+
+ mutex_lock(&dev->resource_mutex);
+
+ ret = dev->ops->get_sn_allocation(&dev->hw, arg.group);
+
+ response.id = ret;
+
+ ret = (ret > 0) ? 0 : ret;
+
+ mutex_unlock(&dev->resource_mutex);
+
+ BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
+
+ if (copy_to_user((void __user *)user_arg, &response, sizeof(response)))
+ return -EFAULT;
+
+ return ret;
+}
+
+static int dlb2_ioctl_get_sn_occupancy(struct dlb2_dev *dev,
+ unsigned long user_arg)
+{
+ struct dlb2_cmd_response response = {0};
+ struct dlb2_get_sn_occupancy_args arg;
+ int ret;
+
+ if (copy_from_user(&arg, (void __user *)user_arg, sizeof(arg)))
+ return -EFAULT;
+
+ mutex_lock(&dev->resource_mutex);
+
+ ret = dev->ops->get_sn_occupancy(&dev->hw, arg.group);
+
+ response.id = ret;
+
+ ret = (ret > 0) ? 0 : ret;
+
+ mutex_unlock(&dev->resource_mutex);
+
+ BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
+
+ if (copy_to_user((void __user *)user_arg, &response, sizeof(response)))
+ return -EFAULT;
+
+ return ret;
+}
+
long dlb2_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
{
struct dlb2_dev *dev;
@@ -696,6 +776,12 @@ long dlb2_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
return dlb2_ioctl_get_num_resources(dev, arg);
case DLB2_IOC_QUERY_CQ_POLL_MODE:
return dlb2_ioctl_query_cq_poll_mode(dev, arg);
+ case DLB2_IOC_SET_SN_ALLOCATION:
+ return dlb2_ioctl_set_sn_allocation(dev, arg);
+ case DLB2_IOC_GET_SN_ALLOCATION:
+ return dlb2_ioctl_get_sn_allocation(dev, arg);
+ case DLB2_IOC_GET_SN_OCCUPANCY:
+ return dlb2_ioctl_get_sn_occupancy(dev, arg);
default:
return -ENOTTY;
}
diff --git a/drivers/misc/dlb2/dlb2_main.h b/drivers/misc/dlb2/dlb2_main.h
index c1ae4267ff19..a7be9399255f 100644
--- a/drivers/misc/dlb2/dlb2_main.h
+++ b/drivers/misc/dlb2/dlb2_main.h
@@ -135,6 +135,9 @@ struct dlb2_device_ops {
int (*dir_port_owned_by_domain)(struct dlb2_hw *hw,
u32 domain_id,
u32 port_id);
+ int (*get_sn_allocation)(struct dlb2_hw *hw, u32 group_id);
+ int (*set_sn_allocation)(struct dlb2_hw *hw, u32 group_id, u32 num);
+ int (*get_sn_occupancy)(struct dlb2_hw *hw, u32 group_id);
int (*get_ldb_queue_depth)(struct dlb2_hw *hw,
u32 domain_id,
struct dlb2_get_ldb_queue_depth_args *args,
diff --git a/drivers/misc/dlb2/dlb2_pf_ops.c b/drivers/misc/dlb2/dlb2_pf_ops.c
index 23a1e9ba0226..337cc8a72a5c 100644
--- a/drivers/misc/dlb2/dlb2_pf_ops.c
+++ b/drivers/misc/dlb2/dlb2_pf_ops.c
@@ -604,6 +604,24 @@ dlb2_pf_dir_port_owned_by_domain(struct dlb2_hw *hw,
return dlb2_dir_port_owned_by_domain(hw, domain_id, port_id, false, 0);
}
+static int
+dlb2_pf_get_sn_allocation(struct dlb2_hw *hw, u32 group_id)
+{
+ return dlb2_get_group_sequence_numbers(hw, group_id);
+}
+
+static int
+dlb2_pf_set_sn_allocation(struct dlb2_hw *hw, u32 group_id, u32 num)
+{
+ return dlb2_set_group_sequence_numbers(hw, group_id, num);
+}
+
+static int
+dlb2_pf_get_sn_occupancy(struct dlb2_hw *hw, u32 group_id)
+{
+ return dlb2_get_group_sequence_number_occupancy(hw, group_id);
+}
+
/********************************/
/****** DLB2 PF Device Ops ******/
/********************************/
@@ -644,6 +662,9 @@ struct dlb2_device_ops dlb2_pf_ops = {
.reset_domain = dlb2_pf_reset_domain,
.ldb_port_owned_by_domain = dlb2_pf_ldb_port_owned_by_domain,
.dir_port_owned_by_domain = dlb2_pf_dir_port_owned_by_domain,
+ .get_sn_allocation = dlb2_pf_get_sn_allocation,
+ .set_sn_allocation = dlb2_pf_set_sn_allocation,
+ .get_sn_occupancy = dlb2_pf_get_sn_occupancy,
.get_ldb_queue_depth = dlb2_pf_get_ldb_queue_depth,
.get_dir_queue_depth = dlb2_pf_get_dir_queue_depth,
.init_hardware = dlb2_pf_init_hardware,
diff --git a/drivers/misc/dlb2/dlb2_resource.c b/drivers/misc/dlb2/dlb2_resource.c
index bfe20864515d..646f52a06842 100644
--- a/drivers/misc/dlb2/dlb2_resource.c
+++ b/drivers/misc/dlb2/dlb2_resource.c
@@ -4991,6 +4991,73 @@ void dlb2_ack_compressed_cq_intr(struct dlb2_hw *hw,
dlb2_ack_msix_interrupt(hw, DLB2_PF_COMPRESSED_MODE_CQ_VECTOR_ID);
}
+int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw, unsigned int group_id)
+{
+ if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+ return -EINVAL;
+
+ return hw->rsrcs.sn_groups[group_id].sequence_numbers_per_queue;
+}
+
+int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
+ unsigned int group_id)
+{
+ if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+ return -EINVAL;
+
+ return dlb2_sn_group_used_slots(&hw->rsrcs.sn_groups[group_id]);
+}
+
+static void dlb2_log_set_group_sequence_numbers(struct dlb2_hw *hw,
+ unsigned int group_id,
+ unsigned long val)
+{
+ DLB2_HW_DBG(hw, "DLB2 set group sequence numbers:\n");
+ DLB2_HW_DBG(hw, "\tGroup ID: %u\n", group_id);
+ DLB2_HW_DBG(hw, "\tValue: %lu\n", val);
+}
+
+int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
+ unsigned int group_id,
+ unsigned long val)
+{
+ u32 valid_allocations[] = {64, 128, 256, 512, 1024};
+ union dlb2_ro_pipe_grp_sn_mode r0 = { {0} };
+ struct dlb2_sn_group *group;
+ int mode;
+
+ if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
+ return -EINVAL;
+
+ group = &hw->rsrcs.sn_groups[group_id];
+
+ /*
+ * Once the first load-balanced queue using an SN group is configured,
+ * the group cannot be changed.
+ */
+ if (group->slot_use_bitmap != 0)
+ return -EPERM;
+
+ for (mode = 0; mode < DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES; mode++)
+ if (val == valid_allocations[mode])
+ break;
+
+ if (mode == DLB2_MAX_NUM_SEQUENCE_NUMBER_MODES)
+ return -EINVAL;
+
+ group->mode = mode;
+ group->sequence_numbers_per_queue = val;
+
+ r0.field.sn_mode_0 = hw->rsrcs.sn_groups[0].mode;
+ r0.field.sn_mode_1 = hw->rsrcs.sn_groups[1].mode;
+
+ DLB2_CSR_WR(hw, DLB2_RO_PIPE_GRP_SN_MODE, r0.val);
+
+ dlb2_log_set_group_sequence_numbers(hw, group_id, val);
+
+ return 0;
+}
+
static u32 dlb2_ldb_cq_inflight_count(struct dlb2_hw *hw,
struct dlb2_ldb_port *port)
{
diff --git a/drivers/misc/dlb2/dlb2_resource.h b/drivers/misc/dlb2/dlb2_resource.h
index 5fd7a30a1c1b..c1e1d677b2ae 100644
--- a/drivers/misc/dlb2/dlb2_resource.h
+++ b/drivers/misc/dlb2/dlb2_resource.h
@@ -573,6 +573,53 @@ void dlb2_ack_compressed_cq_intr(struct dlb2_hw *hw,
u32 *dir_interrupts);
/**
+ * dlb2_get_group_sequence_numbers() - return a group's number of SNs per queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ *
+ * This function returns the configured number of sequence numbers per queue
+ * for the specified group.
+ *
+ * Return:
+ * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
+ */
+int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw,
+ unsigned int group_id);
+
+/**
+ * dlb2_get_group_sequence_number_occupancy() - return a group's in-use slots
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ *
+ * This function returns the group's number of in-use slots (i.e. load-balanced
+ * queues using the specified group).
+ *
+ * Return:
+ * Returns -EINVAL if group_id is invalid, else the group's SNs per queue.
+ */
+int dlb2_get_group_sequence_number_occupancy(struct dlb2_hw *hw,
+ unsigned int group_id);
+
+/**
+ * dlb2_set_group_sequence_numbers() - assign a group's number of SNs per queue
+ * @hw: dlb2_hw handle for a particular device.
+ * @group_id: sequence number group ID.
+ * @val: requested amount of sequence numbers per queue.
+ *
+ * This function configures the group's number of sequence numbers per queue.
+ * val can be a power-of-two between 32 and 1024, inclusive. This setting can
+ * be configured until the first ordered load-balanced queue is configured, at
+ * which point the configuration is locked.
+ *
+ * Return:
+ * Returns 0 upon success; -EINVAL if group_id or val is invalid, -EPERM if an
+ * ordered queue is configured.
+ */
+int dlb2_set_group_sequence_numbers(struct dlb2_hw *hw,
+ unsigned int group_id,
+ unsigned long val);
+
+/**
* dlb2_reset_domain() - reset a scheduling domain
* @hw: dlb2_hw handle for a particular device.
* @domain_id: domain ID.
diff --git a/include/uapi/linux/dlb2_user.h b/include/uapi/linux/dlb2_user.h
index 48783a8e91c2..a117ca71dbbc 100644
--- a/include/uapi/linux/dlb2_user.h
+++ b/include/uapi/linux/dlb2_user.h
@@ -239,11 +239,83 @@ struct dlb2_query_cq_poll_mode_args {
struct dlb2_cmd_response response;
};
+/*
+ * DLB2_CMD_SET_SN_ALLOCATION: Configure a sequence number group (PF only)
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - num: Number of sequence numbers per queue.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_set_sn_allocation_args {
+ /* Output parameters */
+ struct dlb2_cmd_response response;
+ /* Input parameters */
+ __u32 group;
+ __u32 num;
+};
+
+/*
+ * DLB2_CMD_GET_SN_ALLOCATION: Get a sequence number group's configuration
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified group's number of sequence numbers per queue.
+ */
+struct dlb2_get_sn_allocation_args {
+ /* Output parameters */
+ struct dlb2_cmd_response response;
+ /* Input parameters */
+ __u32 group;
+ __u32 padding0;
+};
+
+/*
+ * DLB2_CMD_GET_SN_OCCUPANCY: Get a sequence number group's occupancy
+ *
+ * Each sequence number group has one or more slots, depending on its
+ * configuration. I.e.:
+ * - If configured for 1024 sequence numbers per queue, the group has 1 slot
+ * - If configured for 512 sequence numbers per queue, the group has 2 slots
+ * ...
+ * - If configured for 32 sequence numbers per queue, the group has 32 slots
+ *
+ * This ioctl returns the group's number of in-use slots. If its occupancy is
+ * 0, the group's sequence number allocation can be reconfigured.
+ *
+ * Input parameters:
+ * - group: Sequence number group ID.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ * - response.id: Specified group's number of used slots.
+ */
+struct dlb2_get_sn_occupancy_args {
+ /* Output parameters */
+ struct dlb2_cmd_response response;
+ /* Input parameters */
+ __u32 group;
+ __u32 padding0;
+};
+
enum dlb2_user_interface_commands {
DLB2_CMD_GET_DEVICE_VERSION,
DLB2_CMD_CREATE_SCHED_DOMAIN,
DLB2_CMD_GET_NUM_RESOURCES,
DLB2_CMD_QUERY_CQ_POLL_MODE,
+ DLB2_CMD_SET_SN_ALLOCATION,
+ DLB2_CMD_GET_SN_ALLOCATION,
+ DLB2_CMD_GET_SN_OCCUPANCY,
/* NUM_DLB2_CMD must be last */
NUM_DLB2_CMD,
@@ -825,6 +897,18 @@ enum dlb2_domain_user_interface_commands {
_IOR(DLB2_IOC_MAGIC, \
DLB2_CMD_QUERY_CQ_POLL_MODE, \
struct dlb2_query_cq_poll_mode_args)
+#define DLB2_IOC_SET_SN_ALLOCATION \
+ _IOWR(DLB2_IOC_MAGIC, \
+ DLB2_CMD_SET_SN_ALLOCATION, \
+ struct dlb2_set_sn_allocation_args)
+#define DLB2_IOC_GET_SN_ALLOCATION \
+ _IOWR(DLB2_IOC_MAGIC, \
+ DLB2_CMD_GET_SN_ALLOCATION, \
+ struct dlb2_get_sn_allocation_args)
+#define DLB2_IOC_GET_SN_OCCUPANCY \
+ _IOWR(DLB2_IOC_MAGIC, \
+ DLB2_CMD_GET_SN_OCCUPANCY, \
+ struct dlb2_get_sn_occupancy_args)
#define DLB2_IOC_CREATE_LDB_QUEUE \
_IOWR(DLB2_IOC_MAGIC, \
DLB2_DOMAIN_CMD_CREATE_LDB_QUEUE, \
--
2.13.6
Load-balanced queues can be "mapped" to any number of load-balanced ports.
Once mapped, the port becomes a candidate to which the device can schedule
queue entries from the queue. If a port is unmapped from a queue, it is no
longer a candidate for scheduling from that queue.
The process for map and unmap is different depending on whether or not the
domain is started, because the process requires updating hardware
structures that are dynamically modified by the device when it's in use.
If the domain is not started, its ports and queues are guaranteed not to be
in active use.
If the domain is started, the map process must (temporarily) disable the
queue and wait for it to quiesce. Similarly, the unmap process must
(temporarily) disable the port and wait for it to quiesce. 'Quiesce' here
means the user processes any in-flight QEs.
It's possible that the thread that requires the map/unmap is the same one
which is responsible for doing the processing that would quiesce the
queue/port, thus the driver may have to complete the operation
asynchronously. To that end, the driver uses a workqueue that periodically
checks whether any outstanding operations can be completed. This workqueue
function is only scheduled when there is at least one outstanding map/unmap
operation.
To support this asynchronous operation while also providing a reasonably
usable user-interface, the driver maintains two views of the queue map
state:
- The hardware view: the actual queue map state in the device
- The user/software view: the queue map state as though the operations were
synchronous.
While a map/unmap operation is inflight, these two views are out-of-sync.
When the user requests a new map/unmap operation, the driver verifies the
request against the software view, so the errors are synchronous from the
user’s perspective, then adds the request to the queue of in-progress
operations. When possible -- for example if the user requests to map a
queue and then immediately requests to unmap it -- the driver will coalesce
or cancel outstanding operations.
This commit also adds a pending-port-unmaps ioctl, that user-space can call
to query how many pending/outstanding unmap operations exist for a given
port.
Signed-off-by: Gage Eads <[email protected]>
Reviewed-by: Björn Töpel <[email protected]>
---
drivers/misc/dlb2/dlb2_ioctl.c | 9 +
drivers/misc/dlb2/dlb2_main.h | 14 +
drivers/misc/dlb2/dlb2_pf_ops.c | 30 ++
drivers/misc/dlb2/dlb2_resource.c | 789 ++++++++++++++++++++++++++++++++++++++
drivers/misc/dlb2/dlb2_resource.h | 111 ++++++
include/uapi/linux/dlb2_user.h | 79 ++++
6 files changed, 1032 insertions(+)
diff --git a/drivers/misc/dlb2/dlb2_ioctl.c b/drivers/misc/dlb2/dlb2_ioctl.c
index 5b6d3b4c3297..0ab4a95fb34a 100644
--- a/drivers/misc/dlb2/dlb2_ioctl.c
+++ b/drivers/misc/dlb2/dlb2_ioctl.c
@@ -52,6 +52,9 @@ DLB2_DOMAIN_IOCTL_CALLBACK_TEMPLATE(create_dir_queue)
DLB2_DOMAIN_IOCTL_CALLBACK_TEMPLATE(get_ldb_queue_depth)
DLB2_DOMAIN_IOCTL_CALLBACK_TEMPLATE(get_dir_queue_depth)
DLB2_DOMAIN_IOCTL_CALLBACK_TEMPLATE(start_domain)
+DLB2_DOMAIN_IOCTL_CALLBACK_TEMPLATE(map_qid)
+DLB2_DOMAIN_IOCTL_CALLBACK_TEMPLATE(unmap_qid)
+DLB2_DOMAIN_IOCTL_CALLBACK_TEMPLATE(pending_port_unmaps)
/*
* Port creation ioctls don't use the callback template macro because they have
@@ -346,6 +349,12 @@ long dlb2_domain_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
return dlb2_domain_ioctl_get_dir_port_cq_fd(dev, dom, arg);
case DLB2_IOC_START_DOMAIN:
return dlb2_domain_ioctl_start_domain(dev, dom, arg);
+ case DLB2_IOC_MAP_QID:
+ return dlb2_domain_ioctl_map_qid(dev, dom, arg);
+ case DLB2_IOC_UNMAP_QID:
+ return dlb2_domain_ioctl_unmap_qid(dev, dom, arg);
+ case DLB2_IOC_PENDING_PORT_UNMAPS:
+ return dlb2_domain_ioctl_pending_port_unmaps(dev, dom, arg);
default:
return -ENOTTY;
}
diff --git a/drivers/misc/dlb2/dlb2_main.h b/drivers/misc/dlb2/dlb2_main.h
index abe9808b4638..626a3311eb18 100644
--- a/drivers/misc/dlb2/dlb2_main.h
+++ b/drivers/misc/dlb2/dlb2_main.h
@@ -82,6 +82,18 @@ struct dlb2_device_ops {
u32 domain_id,
struct dlb2_start_domain_args *args,
struct dlb2_cmd_response *resp);
+ int (*map_qid)(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_map_qid_args *args,
+ struct dlb2_cmd_response *resp);
+ int (*unmap_qid)(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_unmap_qid_args *args,
+ struct dlb2_cmd_response *resp);
+ int (*pending_port_unmaps)(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_pending_port_unmaps_args *args,
+ struct dlb2_cmd_response *resp);
int (*get_num_resources)(struct dlb2_hw *hw,
struct dlb2_get_num_resources_args *args);
int (*reset_domain)(struct dlb2_hw *hw, u32 domain_id);
@@ -144,10 +156,12 @@ struct dlb2_dev {
* hardware registers.
*/
struct mutex resource_mutex;
+ struct work_struct work;
enum dlb2_device_type type;
int id;
dev_t dev_number;
u8 domain_reset_failed;
+ u8 worker_launched;
};
int dlb2_init_domain(struct dlb2_dev *dlb2_dev, u32 domain_id);
diff --git a/drivers/misc/dlb2/dlb2_pf_ops.c b/drivers/misc/dlb2/dlb2_pf_ops.c
index 4ab4d9fe3f61..b531448d1a25 100644
--- a/drivers/misc/dlb2/dlb2_pf_ops.c
+++ b/drivers/misc/dlb2/dlb2_pf_ops.c
@@ -272,6 +272,33 @@ dlb2_pf_start_domain(struct dlb2_hw *hw,
}
static int
+dlb2_pf_map_qid(struct dlb2_hw *hw,
+ u32 id,
+ struct dlb2_map_qid_args *args,
+ struct dlb2_cmd_response *resp)
+{
+ return dlb2_hw_map_qid(hw, id, args, resp, false, 0);
+}
+
+static int
+dlb2_pf_unmap_qid(struct dlb2_hw *hw,
+ u32 id,
+ struct dlb2_unmap_qid_args *args,
+ struct dlb2_cmd_response *resp)
+{
+ return dlb2_hw_unmap_qid(hw, id, args, resp, false, 0);
+}
+
+static int
+dlb2_pf_pending_port_unmaps(struct dlb2_hw *hw,
+ u32 id,
+ struct dlb2_pending_port_unmaps_args *args,
+ struct dlb2_cmd_response *resp)
+{
+ return dlb2_hw_pending_port_unmaps(hw, id, args, resp, false, 0);
+}
+
+static int
dlb2_pf_get_num_resources(struct dlb2_hw *hw,
struct dlb2_get_num_resources_args *args)
{
@@ -359,6 +386,9 @@ struct dlb2_device_ops dlb2_pf_ops = {
.create_ldb_port = dlb2_pf_create_ldb_port,
.create_dir_port = dlb2_pf_create_dir_port,
.start_domain = dlb2_pf_start_domain,
+ .map_qid = dlb2_pf_map_qid,
+ .unmap_qid = dlb2_pf_unmap_qid,
+ .pending_port_unmaps = dlb2_pf_pending_port_unmaps,
.get_num_resources = dlb2_pf_get_num_resources,
.reset_domain = dlb2_pf_reset_domain,
.ldb_port_owned_by_domain = dlb2_pf_ldb_port_owned_by_domain,
diff --git a/drivers/misc/dlb2/dlb2_resource.c b/drivers/misc/dlb2/dlb2_resource.c
index 4d8b56a4b437..6afac8c99c06 100644
--- a/drivers/misc/dlb2/dlb2_resource.c
+++ b/drivers/misc/dlb2/dlb2_resource.c
@@ -238,6 +238,32 @@ static struct dlb2_hw_domain *dlb2_get_domain_from_id(struct dlb2_hw *hw,
}
static struct dlb2_ldb_port *
+dlb2_get_domain_used_ldb_port(u32 id,
+ bool vdev_req,
+ struct dlb2_hw_domain *domain)
+{
+ struct dlb2_ldb_port *port;
+ int i;
+
+ if (id >= DLB2_MAX_NUM_LDB_PORTS)
+ return NULL;
+
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port)
+ if ((!vdev_req && port->id.phys_id == id) ||
+ (vdev_req && port->id.virt_id == id))
+ return port;
+
+ DLB2_DOM_LIST_FOR(domain->avail_ldb_ports[i], port)
+ if ((!vdev_req && port->id.phys_id == id) ||
+ (vdev_req && port->id.virt_id == id))
+ return port;
+ }
+
+ return NULL;
+}
+
+static struct dlb2_ldb_port *
dlb2_get_domain_ldb_port(u32 id,
bool vdev_req,
struct dlb2_hw_domain *domain)
@@ -1331,6 +1357,64 @@ static int dlb2_verify_start_domain_args(struct dlb2_hw *hw,
return 0;
}
+static int dlb2_verify_map_qid_args(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_map_qid_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_hw_domain *domain;
+ struct dlb2_ldb_port *port;
+ struct dlb2_ldb_queue *queue;
+ int id;
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+ if (!domain) {
+ resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+ return -EINVAL;
+ }
+
+ if (!domain->configured) {
+ resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+ return -EINVAL;
+ }
+
+ id = args->port_id;
+
+ port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+ if (!port || !port->configured) {
+ resp->status = DLB2_ST_INVALID_PORT_ID;
+ return -EINVAL;
+ }
+
+ if (args->priority >= DLB2_QID_PRIORITIES) {
+ resp->status = DLB2_ST_INVALID_PRIORITY;
+ return -EINVAL;
+ }
+
+ queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+
+ if (!queue || !queue->configured) {
+ resp->status = DLB2_ST_INVALID_QID;
+ return -EINVAL;
+ }
+
+ if (queue->domain_id.phys_id != domain->id.phys_id) {
+ resp->status = DLB2_ST_INVALID_QID;
+ return -EINVAL;
+ }
+
+ if (port->domain_id.phys_id != domain->id.phys_id) {
+ resp->status = DLB2_ST_INVALID_PORT_ID;
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
static bool dlb2_port_find_slot(struct dlb2_ldb_port *port,
enum dlb2_qid_map_state state,
int *slot)
@@ -1365,6 +1449,26 @@ static bool dlb2_port_find_slot_queue(struct dlb2_ldb_port *port,
return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
}
+static bool
+dlb2_port_find_slot_with_pending_map_queue(struct dlb2_ldb_port *port,
+ struct dlb2_ldb_queue *queue,
+ int *slot)
+{
+ int i;
+
+ for (i = 0; i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+ struct dlb2_ldb_port_qid_map *map = &port->qid_map[i];
+
+ if (map->state == DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP &&
+ map->pending_qid == queue->id.phys_id)
+ break;
+ }
+
+ *slot = i;
+
+ return (i < DLB2_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
static int dlb2_port_slot_state_transition(struct dlb2_hw *hw,
struct dlb2_ldb_port *port,
struct dlb2_ldb_queue *queue,
@@ -1492,6 +1596,117 @@ static int dlb2_port_slot_state_transition(struct dlb2_hw *hw,
return -EFAULT;
}
+static int dlb2_verify_map_qid_slot_available(struct dlb2_ldb_port *port,
+ struct dlb2_ldb_queue *queue,
+ struct dlb2_cmd_response *resp)
+{
+ enum dlb2_qid_map_state state;
+ int i;
+
+ /* Unused slot available? */
+ if (port->num_mappings < DLB2_MAX_NUM_QIDS_PER_LDB_CQ)
+ return 0;
+
+ /*
+ * If the queue is already mapped (from the application's perspective),
+ * this is simply a priority update.
+ */
+ state = DLB2_QUEUE_MAPPED;
+ if (dlb2_port_find_slot_queue(port, state, queue, &i))
+ return 0;
+
+ state = DLB2_QUEUE_MAP_IN_PROG;
+ if (dlb2_port_find_slot_queue(port, state, queue, &i))
+ return 0;
+
+ if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i))
+ return 0;
+
+ /*
+ * If the slot contains an unmap in progress, it's considered
+ * available.
+ */
+ state = DLB2_QUEUE_UNMAP_IN_PROG;
+ if (dlb2_port_find_slot(port, state, &i))
+ return 0;
+
+ state = DLB2_QUEUE_UNMAPPED;
+ if (dlb2_port_find_slot(port, state, &i))
+ return 0;
+
+ resp->status = DLB2_ST_NO_QID_SLOTS_AVAILABLE;
+ return -EINVAL;
+}
+
+static int dlb2_verify_unmap_qid_args(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_unmap_qid_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ enum dlb2_qid_map_state state;
+ struct dlb2_hw_domain *domain;
+ struct dlb2_ldb_queue *queue;
+ struct dlb2_ldb_port *port;
+ int slot;
+ int id;
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+ if (!domain) {
+ resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+ return -EINVAL;
+ }
+
+ if (!domain->configured) {
+ resp->status = DLB2_ST_DOMAIN_NOT_CONFIGURED;
+ return -EINVAL;
+ }
+
+ id = args->port_id;
+
+ port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+
+ if (!port || !port->configured) {
+ resp->status = DLB2_ST_INVALID_PORT_ID;
+ return -EINVAL;
+ }
+
+ if (port->domain_id.phys_id != domain->id.phys_id) {
+ resp->status = DLB2_ST_INVALID_PORT_ID;
+ return -EINVAL;
+ }
+
+ queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+
+ if (!queue || !queue->configured) {
+ DLB2_HW_ERR(hw, "[%s()] Can't unmap unconfigured queue %d\n",
+ __func__, args->qid);
+ resp->status = DLB2_ST_INVALID_QID;
+ return -EINVAL;
+ }
+
+ /*
+ * Verify that the port has the queue mapped. From the application's
+ * perspective a queue is mapped if it is actually mapped, the map is
+ * in progress, or the map is blocked pending an unmap.
+ */
+ state = DLB2_QUEUE_MAPPED;
+ if (dlb2_port_find_slot_queue(port, state, queue, &slot))
+ return 0;
+
+ state = DLB2_QUEUE_MAP_IN_PROG;
+ if (dlb2_port_find_slot_queue(port, state, queue, &slot))
+ return 0;
+
+ if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &slot))
+ return 0;
+
+ resp->status = DLB2_ST_INVALID_QID;
+ return -EINVAL;
+}
+
static void dlb2_configure_domain_credits(struct dlb2_hw *hw,
struct dlb2_hw_domain *domain)
{
@@ -2304,6 +2519,26 @@ static int dlb2_ldb_port_map_qid_static(struct dlb2_hw *hw,
return dlb2_port_slot_state_transition(hw, p, q, i, state);
}
+static void dlb2_ldb_port_change_qid_priority(struct dlb2_hw *hw,
+ struct dlb2_ldb_port *port,
+ int slot,
+ struct dlb2_map_qid_args *args)
+{
+ union dlb2_lsp_cq2priov r0;
+
+ /* Read-modify-write the priority and valid bit register */
+ r0.val = DLB2_CSR_RD(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id));
+
+ r0.field.v |= 1 << slot;
+ r0.field.prio |= (args->priority & 0x7) << slot * 3;
+
+ DLB2_CSR_WR(hw, DLB2_LSP_CQ2PRIOV(port->id.phys_id), r0.val);
+
+ dlb2_flush_csr(hw);
+
+ port->qid_map[slot].priority = args->priority;
+}
+
static int dlb2_ldb_port_set_has_work_bits(struct dlb2_hw *hw,
struct dlb2_ldb_port *port,
struct dlb2_ldb_queue *queue,
@@ -2543,6 +2778,62 @@ static int dlb2_ldb_port_finish_map_qid_dynamic(struct dlb2_hw *hw,
return 0;
}
+static unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw);
+static unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw);
+
+/*
+ * The workqueue callback runs until it completes all outstanding QID->CQ
+ * map and unmap requests. To prevent deadlock, this function gives other
+ * threads a chance to grab the resource mutex and configure hardware.
+ */
+static void dlb2_complete_queue_map_unmap(struct work_struct *work)
+{
+ struct dlb2_dev *dlb2_dev;
+ int ret;
+
+ dlb2_dev = container_of(work, struct dlb2_dev, work);
+
+ mutex_lock(&dlb2_dev->resource_mutex);
+
+ ret = dlb2_finish_unmap_qid_procedures(&dlb2_dev->hw);
+ ret += dlb2_finish_map_qid_procedures(&dlb2_dev->hw);
+
+ if (ret != 0)
+ /*
+ * Relinquish the CPU so the application can process its CQs,
+ * so this function doesn't deadlock.
+ */
+ schedule_work(&dlb2_dev->work);
+ else
+ dlb2_dev->worker_launched = false;
+
+ mutex_unlock(&dlb2_dev->resource_mutex);
+}
+
+/**
+ * dlb2_schedule_work() - launch a thread to process pending map and unmap work
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function launches a kernel thread that will run until all pending
+ * map and unmap procedures are complete.
+ */
+static void dlb2_schedule_work(struct dlb2_hw *hw)
+{
+ struct dlb2_dev *dlb2_dev;
+
+ dlb2_dev = container_of(hw, struct dlb2_dev, hw);
+
+ /* Nothing to do if the worker is already running */
+ if (dlb2_dev->worker_launched)
+ return;
+
+ INIT_WORK(&dlb2_dev->work, dlb2_complete_queue_map_unmap);
+
+ schedule_work(&dlb2_dev->work);
+
+ dlb2_dev->worker_launched = true;
+}
+
/**
* dlb2_ldb_port_map_qid_dynamic() - perform a "dynamic" QID->CQ mapping
* @hw: dlb2_hw handle for a particular device.
@@ -2600,6 +2891,20 @@ static int dlb2_ldb_port_map_qid_dynamic(struct dlb2_hw *hw,
if (ret)
return ret;
+ r0.val = DLB2_CSR_RD(hw,
+ DLB2_LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+ if (r0.field.count) {
+ /*
+ * The queue is owed completions so it's not safe to map it
+ * yet. Schedule a kernel thread to complete the mapping later,
+ * once software has completed all the queue's inflight events.
+ */
+ dlb2_schedule_work(hw);
+
+ return 1;
+ }
+
/*
* Disable the affected CQ, and the CQs already mapped to the QID,
* before reading the QID's inflight count a second time. There is an
@@ -2622,6 +2927,13 @@ static int dlb2_ldb_port_map_qid_dynamic(struct dlb2_hw *hw,
dlb2_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+ /*
+ * The queue is owed completions so it's not safe to map it
+ * yet. Schedule a kernel thread to complete the mapping later,
+ * once software has completed all the queue's inflight events.
+ */
+ dlb2_schedule_work(hw);
+
return 1;
}
@@ -3480,6 +3792,20 @@ dlb2_domain_finish_unmap_qid_procedures(struct dlb2_hw *hw,
return domain->num_pending_removals;
}
+static unsigned int dlb2_finish_unmap_qid_procedures(struct dlb2_hw *hw)
+{
+ int i, num = 0;
+
+ /* Finish queue unmap jobs for any domain that needs it */
+ for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+ struct dlb2_hw_domain *domain = &hw->domains[i];
+
+ num += dlb2_domain_finish_unmap_qid_procedures(hw, domain);
+ }
+
+ return num;
+}
+
static void dlb2_domain_finish_map_port(struct dlb2_hw *hw,
struct dlb2_hw_domain *domain,
struct dlb2_ldb_port *port)
@@ -3556,6 +3882,427 @@ dlb2_domain_finish_map_qid_procedures(struct dlb2_hw *hw,
return domain->num_pending_additions;
}
+static unsigned int dlb2_finish_map_qid_procedures(struct dlb2_hw *hw)
+{
+ int i, num = 0;
+
+ /* Finish queue map jobs for any domain that needs it */
+ for (i = 0; i < DLB2_MAX_NUM_DOMAINS; i++) {
+ struct dlb2_hw_domain *domain = &hw->domains[i];
+
+ num += dlb2_domain_finish_map_qid_procedures(hw, domain);
+ }
+
+ return num;
+}
+
+static void dlb2_log_map_qid(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_map_qid_args *args,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ DLB2_HW_DBG(hw, "DLB2 map QID arguments:\n");
+ if (vdev_req)
+ DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+ DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+ domain_id);
+ DLB2_HW_DBG(hw, "\tPort ID: %d\n",
+ args->port_id);
+ DLB2_HW_DBG(hw, "\tQueue ID: %d\n",
+ args->qid);
+ DLB2_HW_DBG(hw, "\tPriority: %d\n",
+ args->priority);
+}
+
+int dlb2_hw_map_qid(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_map_qid_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_hw_domain *domain;
+ struct dlb2_ldb_queue *queue;
+ enum dlb2_qid_map_state st;
+ struct dlb2_ldb_port *port;
+ int ret, i, id;
+ u8 prio;
+
+ dlb2_log_map_qid(hw, domain_id, args, vdev_req, vdev_id);
+
+ /*
+ * Verify that hardware resources are available before attempting to
+ * satisfy the request. This simplifies the error unwinding code.
+ */
+ ret = dlb2_verify_map_qid_args(hw,
+ domain_id,
+ args,
+ resp,
+ vdev_req,
+ vdev_id);
+ if (ret)
+ return ret;
+
+ prio = args->priority;
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+ if (!domain) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: domain not found\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ id = args->port_id;
+
+ port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+ if (!port) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: port not found\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+ if (!queue) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: queue not found\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ /*
+ * If there are any outstanding detach operations for this port,
+ * attempt to complete them. This may be necessary to free up a QID
+ * slot for this requested mapping.
+ */
+ if (port->num_pending_removals)
+ dlb2_domain_finish_unmap_port(hw, domain, port);
+
+ ret = dlb2_verify_map_qid_slot_available(port, queue, resp);
+ if (ret)
+ return ret;
+
+ /* Hardware requires disabling the CQ before mapping QIDs. */
+ if (port->enabled)
+ dlb2_ldb_port_cq_disable(hw, port);
+
+ /*
+ * If this is only a priority change, don't perform the full QID->CQ
+ * mapping procedure
+ */
+ st = DLB2_QUEUE_MAPPED;
+ if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+ if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: port slot tracking failed\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ if (prio != port->qid_map[i].priority) {
+ dlb2_ldb_port_change_qid_priority(hw, port, i, args);
+ DLB2_HW_DBG(hw, "DLB2 map: priority change\n");
+ }
+
+ st = DLB2_QUEUE_MAPPED;
+ ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+ if (ret)
+ return ret;
+
+ goto map_qid_done;
+ }
+
+ st = DLB2_QUEUE_UNMAP_IN_PROG;
+ if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+ if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: port slot tracking failed\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ if (prio != port->qid_map[i].priority) {
+ dlb2_ldb_port_change_qid_priority(hw, port, i, args);
+ DLB2_HW_DBG(hw, "DLB2 map: priority change\n");
+ }
+
+ st = DLB2_QUEUE_MAPPED;
+ ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+ if (ret)
+ return ret;
+
+ goto map_qid_done;
+ }
+
+ /*
+ * If this is a priority change on an in-progress mapping, don't
+ * perform the full QID->CQ mapping procedure.
+ */
+ st = DLB2_QUEUE_MAP_IN_PROG;
+ if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+ if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: port slot tracking failed\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ port->qid_map[i].priority = prio;
+
+ DLB2_HW_DBG(hw, "DLB2 map: priority change only\n");
+
+ goto map_qid_done;
+ }
+
+ /*
+ * If this is a priority change on a pending mapping, update the
+ * pending priority
+ */
+ if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+ if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: port slot tracking failed\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ port->qid_map[i].pending_priority = prio;
+
+ DLB2_HW_DBG(hw, "DLB2 map: priority change only\n");
+
+ goto map_qid_done;
+ }
+
+ /*
+ * If all the CQ's slots are in use, then there's an unmap in progress
+ * (guaranteed by dlb2_verify_map_qid_slot_available()), so add this
+ * mapping to pending_map and return. When the removal is completed for
+ * the slot's current occupant, this mapping will be performed.
+ */
+ if (!dlb2_port_find_slot(port, DLB2_QUEUE_UNMAPPED, &i)) {
+ if (dlb2_port_find_slot(port, DLB2_QUEUE_UNMAP_IN_PROG, &i)) {
+ enum dlb2_qid_map_state st;
+
+ if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: port slot tracking failed\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ port->qid_map[i].pending_qid = queue->id.phys_id;
+ port->qid_map[i].pending_priority = prio;
+
+ st = DLB2_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+ ret = dlb2_port_slot_state_transition(hw, port, queue,
+ i, st);
+ if (ret)
+ return ret;
+
+ DLB2_HW_DBG(hw, "DLB2 map: map pending removal\n");
+
+ goto map_qid_done;
+ }
+ }
+
+ /*
+ * If the domain has started, a special "dynamic" CQ->queue mapping
+ * procedure is required in order to safely update the CQ<->QID tables.
+ * The "static" procedure cannot be used when traffic is flowing,
+ * because the CQ<->QID tables cannot be updated atomically and the
+ * scheduler won't see the new mapping unless the queue's if_status
+ * changes, which isn't guaranteed.
+ */
+ ret = dlb2_ldb_port_map_qid(hw, domain, port, queue, prio);
+
+ /* If ret is less than zero, it's due to an internal error */
+ if (ret < 0)
+ return ret;
+
+map_qid_done:
+ if (port->enabled)
+ dlb2_ldb_port_cq_enable(hw, port);
+
+ resp->status = 0;
+
+ return 0;
+}
+
+static void dlb2_log_unmap_qid(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_unmap_qid_args *args,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ DLB2_HW_DBG(hw, "DLB2 unmap QID arguments:\n");
+ if (vdev_req)
+ DLB2_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+ DLB2_HW_DBG(hw, "\tDomain ID: %d\n",
+ domain_id);
+ DLB2_HW_DBG(hw, "\tPort ID: %d\n",
+ args->port_id);
+ DLB2_HW_DBG(hw, "\tQueue ID: %d\n",
+ args->qid);
+ if (args->qid < DLB2_MAX_NUM_LDB_QUEUES)
+ DLB2_HW_DBG(hw, "\tQueue's num mappings: %d\n",
+ hw->rsrcs.ldb_queues[args->qid].num_mappings);
+}
+
+int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_unmap_qid_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_hw_domain *domain;
+ struct dlb2_ldb_queue *queue;
+ enum dlb2_qid_map_state st;
+ struct dlb2_ldb_port *port;
+ bool unmap_complete;
+ int i, ret, id;
+
+ dlb2_log_unmap_qid(hw, domain_id, args, vdev_req, vdev_id);
+
+ /*
+ * Verify that hardware resources are available before attempting to
+ * satisfy the request. This simplifies the error unwinding code.
+ */
+ ret = dlb2_verify_unmap_qid_args(hw,
+ domain_id,
+ args,
+ resp,
+ vdev_req,
+ vdev_id);
+ if (ret)
+ return ret;
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+ if (!domain) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: domain not found\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ id = args->port_id;
+
+ port = dlb2_get_domain_used_ldb_port(id, vdev_req, domain);
+ if (!port) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: port not found\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ queue = dlb2_get_domain_ldb_queue(args->qid, vdev_req, domain);
+ if (!queue) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: queue not found\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ /*
+ * If the queue hasn't been mapped yet, we need to update the slot's
+ * state and re-enable the queue's inflights.
+ */
+ st = DLB2_QUEUE_MAP_IN_PROG;
+ if (dlb2_port_find_slot_queue(port, st, queue, &i)) {
+ if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: port slot tracking failed\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ /*
+ * Since the in-progress map was aborted, re-enable the QID's
+ * inflights.
+ */
+ if (queue->num_pending_additions == 0)
+ dlb2_ldb_queue_set_inflight_limit(hw, queue);
+
+ st = DLB2_QUEUE_UNMAPPED;
+ ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+ if (ret)
+ return ret;
+
+ goto unmap_qid_done;
+ }
+
+ /*
+ * If the queue mapping is on hold pending an unmap, we simply need to
+ * update the slot's state.
+ */
+ if (dlb2_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+ if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: port slot tracking failed\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ st = DLB2_QUEUE_UNMAP_IN_PROG;
+ ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+ if (ret)
+ return ret;
+
+ goto unmap_qid_done;
+ }
+
+ st = DLB2_QUEUE_MAPPED;
+ if (!dlb2_port_find_slot_queue(port, st, queue, &i)) {
+ DLB2_HW_ERR(hw,
+ "[%s()] Internal error: no available CQ slots\n",
+ __func__);
+ return -EFAULT;
+ }
+
+ if (i >= DLB2_MAX_NUM_QIDS_PER_LDB_CQ) {
+ DLB2_HW_ERR(hw,
+ "[%s():%d] Internal error: port slot tracking failed\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ /*
+ * QID->CQ mapping removal is an asychronous procedure. It requires
+ * stopping the DLB2 from scheduling this CQ, draining all inflights
+ * from the CQ, then unmapping the queue from the CQ. This function
+ * simply marks the port as needing the queue unmapped, and (if
+ * necessary) starts the unmapping worker thread.
+ */
+ dlb2_ldb_port_cq_disable(hw, port);
+
+ st = DLB2_QUEUE_UNMAP_IN_PROG;
+ ret = dlb2_port_slot_state_transition(hw, port, queue, i, st);
+ if (ret)
+ return ret;
+
+ /*
+ * Attempt to finish the unmapping now, in case the port has no
+ * outstanding inflights. If that's not the case, this will fail and
+ * the unmapping will be completed at a later time.
+ */
+ unmap_complete = dlb2_domain_finish_unmap_port(hw, domain, port);
+
+ /*
+ * If the unmapping couldn't complete immediately, launch the worker
+ * thread (if it isn't already launched) to finish it later.
+ */
+ if (!unmap_complete)
+ dlb2_schedule_work(hw);
+
+unmap_qid_done:
+ resp->status = 0;
+
+ return 0;
+}
+
static u32 dlb2_ldb_cq_inflight_count(struct dlb2_hw *hw,
struct dlb2_ldb_port *port)
{
@@ -3890,6 +4637,48 @@ int dlb2_hw_get_dir_queue_depth(struct dlb2_hw *hw,
return 0;
}
+static void
+dlb2_log_pending_port_unmaps_args(struct dlb2_hw *hw,
+ struct dlb2_pending_port_unmaps_args *args,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ DLB2_HW_DBG(hw, "DLB unmaps in progress arguments:\n");
+ if (vdev_req)
+ DLB2_HW_DBG(hw, "(Request from VF %d)\n", vdev_id);
+ DLB2_HW_DBG(hw, "\tPort ID: %d\n", args->port_id);
+}
+
+int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_pending_port_unmaps_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_hw_domain *domain;
+ struct dlb2_ldb_port *port;
+
+ dlb2_log_pending_port_unmaps_args(hw, args, vdev_req, vdev_id);
+
+ domain = dlb2_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
+
+ if (!domain) {
+ resp->status = DLB2_ST_INVALID_DOMAIN_ID;
+ return -EINVAL;
+ }
+
+ port = dlb2_get_domain_used_ldb_port(args->port_id, vdev_req, domain);
+ if (!port || !port->configured) {
+ resp->status = DLB2_ST_INVALID_PORT_ID;
+ return -EINVAL;
+ }
+
+ resp->id = port->num_pending_removals;
+
+ return 0;
+}
+
static u32 dlb2_ldb_queue_depth(struct dlb2_hw *hw,
struct dlb2_ldb_queue *queue)
{
diff --git a/drivers/misc/dlb2/dlb2_resource.h b/drivers/misc/dlb2/dlb2_resource.h
index d5b554b701f5..4843da49935f 100644
--- a/drivers/misc/dlb2/dlb2_resource.h
+++ b/drivers/misc/dlb2/dlb2_resource.h
@@ -239,6 +239,92 @@ int dlb2_hw_start_domain(struct dlb2_hw *hw,
unsigned int vdev_id);
/**
+ * dlb2_hw_map_qid() - map a load-balanced queue to a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: map QID arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to schedule QEs from the specified queue
+ * to the specified port. Each load-balanced port can be mapped to up to 8
+ * queues; each load-balanced queue can potentially map to all the
+ * load-balanced ports.
+ *
+ * A successful return does not necessarily mean the mapping was configured. If
+ * this function is unable to immediately map the queue to the port, it will
+ * add the requested operation to a per-port list of pending map/unmap
+ * operations, and (if it's not already running) launch a kernel thread that
+ * periodically attempts to process all pending operations. In a sense, this is
+ * an asynchronous function.
+ *
+ * This asynchronicity creates two views of the state of hardware: the actual
+ * hardware state and the requested state (as if every request completed
+ * immediately). If there are any pending map/unmap operations, the requested
+ * state will differ from the actual state. All validation is performed with
+ * respect to the pending state; for instance, if there are 8 pending map
+ * operations for port X, a request for a 9th will fail because a load-balanced
+ * port can only map up to 8 queues.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
+ * the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_map_qid(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_map_qid_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_request,
+ unsigned int vdev_id);
+
+/**
+ * dlb2_hw_unmap_qid() - Unmap a load-balanced queue from a load-balanced port
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: unmap QID arguments.
+ * @resp: response structure.
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function configures the DLB to stop scheduling QEs from the specified
+ * queue to the specified port.
+ *
+ * A successful return does not necessarily mean the mapping was removed. If
+ * this function is unable to immediately unmap the queue from the port, it
+ * will add the requested operation to a per-port list of pending map/unmap
+ * operations, and (if it's not already running) launch a kernel thread that
+ * periodically attempts to process all pending operations. See
+ * dlb2_hw_map_qid() for more details.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, invalid port or queue ID, or
+ * the domain is not configured.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb2_hw_unmap_qid(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_unmap_qid_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vdev_request,
+ unsigned int vdev_id);
+
+/**
* dlb2_reset_domain() - reset a scheduling domain
* @hw: dlb2_hw handle for a particular device.
* @domain_id: domain ID.
@@ -417,6 +503,31 @@ enum dlb2_virt_mode {
};
/**
+ * dlb2_hw_pending_port_unmaps() - returns the number of unmap operations in
+ * progress.
+ * @hw: dlb2_hw handle for a particular device.
+ * @domain_id: domain ID.
+ * @args: number of unmaps in progress args
+ * @resp: response structure.
+ * @vf_request: indicates whether this request came from a VF.
+ * @vf_id: If vf_request is true, this contains the VF's ID.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb2_error. If successful, resp->id
+ * contains the number of unmaps in progress.
+ *
+ * Errors:
+ * EINVAL - Invalid port ID.
+ */
+int dlb2_hw_pending_port_unmaps(struct dlb2_hw *hw,
+ u32 domain_id,
+ struct dlb2_pending_port_unmaps_args *args,
+ struct dlb2_cmd_response *resp,
+ bool vf_request,
+ unsigned int vf_id);
+
+/**
* dlb2_hw_enable_sparse_ldb_cq_mode() - enable sparse mode for load-balanced
* ports.
* @hw: dlb2_hw handle for a particular device.
diff --git a/include/uapi/linux/dlb2_user.h b/include/uapi/linux/dlb2_user.h
index 0eb9e14de11d..1fdc7627c393 100644
--- a/include/uapi/linux/dlb2_user.h
+++ b/include/uapi/linux/dlb2_user.h
@@ -473,6 +473,70 @@ struct dlb2_start_domain_args {
struct dlb2_cmd_response response;
};
+/*
+ * DLB2_DOMAIN_CMD_MAP_QID: Map a load-balanced queue to a load-balanced port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - qid: Load-balanced queue ID.
+ * - priority: Queue->port service priority.
+ * - padding0: Reserved for future use.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_map_qid_args {
+ /* Output parameters */
+ struct dlb2_cmd_response response;
+ /* Input parameters */
+ __u32 port_id;
+ __u32 qid;
+ __u32 priority;
+ __u32 padding0;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_UNMAP_QID: Unmap a load-balanced queue to a load-balanced
+ * port.
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ * - qid: Load-balanced queue ID.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_unmap_qid_args {
+ /* Output parameters */
+ struct dlb2_cmd_response response;
+ /* Input parameters */
+ __u32 port_id;
+ __u32 qid;
+};
+
+/*
+ * DLB2_DOMAIN_CMD_PENDING_PORT_UNMAPS: Get number of queue unmap operations in
+ * progress for a load-balanced port.
+ *
+ * Note: This is a snapshot; the number of unmap operations in progress
+ * is subject to change at any time.
+ *
+ * Input parameters:
+ * - port_id: Load-balanced port ID.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ * - response.id: number of unmaps in progress.
+ */
+struct dlb2_pending_port_unmaps_args {
+ /* Output parameters */
+ struct dlb2_cmd_response response;
+ /* Input parameters */
+ __u32 port_id;
+ __u32 padding0;
+};
+
enum dlb2_domain_user_interface_commands {
DLB2_DOMAIN_CMD_CREATE_LDB_QUEUE,
DLB2_DOMAIN_CMD_CREATE_DIR_QUEUE,
@@ -485,6 +549,9 @@ enum dlb2_domain_user_interface_commands {
DLB2_DOMAIN_CMD_GET_DIR_PORT_PP_FD,
DLB2_DOMAIN_CMD_GET_DIR_PORT_CQ_FD,
DLB2_DOMAIN_CMD_START_DOMAIN,
+ DLB2_DOMAIN_CMD_MAP_QID,
+ DLB2_DOMAIN_CMD_UNMAP_QID,
+ DLB2_DOMAIN_CMD_PENDING_PORT_UNMAPS,
/* NUM_DLB2_DOMAIN_CMD must be last */
NUM_DLB2_DOMAIN_CMD,
@@ -563,5 +630,17 @@ enum dlb2_domain_user_interface_commands {
_IOR(DLB2_IOC_MAGIC, \
DLB2_DOMAIN_CMD_START_DOMAIN, \
struct dlb2_start_domain_args)
+#define DLB2_IOC_MAP_QID \
+ _IOWR(DLB2_IOC_MAGIC, \
+ DLB2_DOMAIN_CMD_MAP_QID, \
+ struct dlb2_map_qid_args)
+#define DLB2_IOC_UNMAP_QID \
+ _IOWR(DLB2_IOC_MAGIC, \
+ DLB2_DOMAIN_CMD_UNMAP_QID, \
+ struct dlb2_unmap_qid_args)
+#define DLB2_IOC_PENDING_PORT_UNMAPS \
+ _IOWR(DLB2_IOC_MAGIC, \
+ DLB2_DOMAIN_CMD_PENDING_PORT_UNMAPS, \
+ struct dlb2_pending_port_unmaps_args)
#endif /* __DLB2_USER_H */
--
2.13.6
When DLB 2.0 detects that an application enqueued an illegal QE, it fires
an ingress error interrupt. The driver processes these errors and notifies
user-space through a domain alert.
A malicious application or virtual machine could attempt a
denial-of-service attack by enqueueing illegal QEs at a high rate and tying
up the PF driver, so the driver tracks the rate of ingress error alarms per
second. If they exceed a threshold of 1000/second, the driver simply
disables the interrupt. The interrupt is re-enabled when the device is
reset, the driver is reloaded, or by a user through a sysfs interface.
Signed-off-by: Gage Eads <[email protected]>
Reviewed-by: Björn Töpel <[email protected]>
---
Documentation/ABI/testing/sysfs-driver-dlb2 | 15 ++
drivers/misc/dlb2/dlb2_hw_types.h | 17 ++
drivers/misc/dlb2/dlb2_main.h | 16 ++
drivers/misc/dlb2/dlb2_pf_ops.c | 177 ++++++++++++++++++++
drivers/misc/dlb2/dlb2_resource.c | 251 +++++++++++++++++++++++++++-
drivers/misc/dlb2/dlb2_resource.h | 45 +++++
6 files changed, 520 insertions(+), 1 deletion(-)
diff --git a/Documentation/ABI/testing/sysfs-driver-dlb2 b/Documentation/ABI/testing/sysfs-driver-dlb2
index c5cb1cbb70f4..3598ce9caa03 100644
--- a/Documentation/ABI/testing/sysfs-driver-dlb2
+++ b/Documentation/ABI/testing/sysfs-driver-dlb2
@@ -185,3 +185,18 @@ Description: Device ID used in /dev, i.e. /dev/dlb<device ID>
/dev directory: /dev/dlb<device ID>. This sysfs file can be read
to determine a device's ID, which allows the user to map a
device file to a PCI BDF.
+
+What: /sys/bus/pci/devices/.../ingress_err_en
+Date: June 22, 2020
+KernelVersion: 5.9
+Contact: [email protected]
+Description: Interface for re-enabling ingress error interrupts
+
+ If the PF driver detects an overload of ingress error
+ interrupts, it will -- to prevent a possible denial-of-service
+ attack -- disable them.
+
+ If the ingress error interrupt is disabled, they will be
+ re-enabled when the device is reset or the driver is reloaded.
+ This interface can be used to re-enable the interrupt as well,
+ and can be read to determine whether the interrupt is enabled.
diff --git a/drivers/misc/dlb2/dlb2_hw_types.h b/drivers/misc/dlb2/dlb2_hw_types.h
index 986d35cbeaf9..2a103e3e958e 100644
--- a/drivers/misc/dlb2/dlb2_hw_types.h
+++ b/drivers/misc/dlb2/dlb2_hw_types.h
@@ -61,6 +61,23 @@
#define DLB2_PF_COMPRESSED_MODE_CQ_VECTOR_ID \
DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS
+/* DLB non-CQ interrupts (alarm, mailbox, WDT) */
+#define DLB2_INT_NON_CQ 0
+
+#define DLB2_ALARM_HW_SOURCE_SYS 0
+#define DLB2_ALARM_HW_SOURCE_DLB 1
+
+#define DLB2_ALARM_HW_UNIT_CHP 4
+
+#define DLB2_ALARM_SYS_AID_ILLEGAL_QID 3
+#define DLB2_ALARM_SYS_AID_DISABLED_QID 4
+#define DLB2_ALARM_SYS_AID_ILLEGAL_HCW 5
+#define DLB2_ALARM_HW_CHP_AID_ILLEGAL_ENQ 1
+#define DLB2_ALARM_HW_CHP_AID_EXCESS_TOKEN_POPS 2
+
+#define DLB2_VF_NUM_NON_CQ_INTERRUPT_VECTORS 1
+#define DLB2_VF_NUM_CQ_INTERRUPT_VECTORS 31
+
/*
* Hardware-defined base addresses. Those prefixed 'DLB2_DRV' are only used by
* the PF driver.
diff --git a/drivers/misc/dlb2/dlb2_main.h b/drivers/misc/dlb2/dlb2_main.h
index ab8cebe6220f..cf7e187d11c6 100644
--- a/drivers/misc/dlb2/dlb2_main.h
+++ b/drivers/misc/dlb2/dlb2_main.h
@@ -217,6 +217,21 @@ struct dlb2_intr {
int mode;
};
+/*
+ * ISR overload is defined as more than DLB2_ISR_OVERLOAD_THRESH interrupts
+ * (of a particular type) occurring in a 1s period. If overload is detected,
+ * the driver blocks that interrupt (exact mechanism depending on the
+ * interrupt) from overloading the PF driver.
+ */
+#define DLB2_ISR_OVERLOAD_THRESH 1000
+#define DLB2_ISR_OVERLOAD_PERIOD_S 1
+
+struct dlb2_alarm {
+ ktime_t ts;
+ unsigned int enabled;
+ u32 count;
+};
+
struct dlb2_dev {
struct pci_dev *pdev;
struct dlb2_hw hw;
@@ -239,6 +254,7 @@ struct dlb2_dev {
*/
struct mutex resource_mutex;
struct work_struct work;
+ struct dlb2_alarm ingress_err;
enum dlb2_device_type type;
int id;
dev_t dev_number;
diff --git a/drivers/misc/dlb2/dlb2_pf_ops.c b/drivers/misc/dlb2/dlb2_pf_ops.c
index 4f0f0e41ab8d..d719a977a763 100644
--- a/drivers/misc/dlb2/dlb2_pf_ops.c
+++ b/drivers/misc/dlb2/dlb2_pf_ops.c
@@ -82,6 +82,109 @@ dlb2_pf_map_pci_bar_space(struct dlb2_dev *dlb2_dev,
/****** Interrupt management ******/
/**********************************/
+static void dlb2_detect_ingress_err_overload(struct dlb2_dev *dev)
+{
+ s64 delta_us;
+
+ if (dev->ingress_err.count == 0)
+ dev->ingress_err.ts = ktime_get();
+
+ dev->ingress_err.count++;
+
+ /* Don't check for overload until OVERLOAD_THRESH ISRs have run */
+ if (dev->ingress_err.count < DLB2_ISR_OVERLOAD_THRESH)
+ return;
+
+ delta_us = ktime_us_delta(ktime_get(), dev->ingress_err.ts);
+
+ /* Reset stats for next measurement period */
+ dev->ingress_err.count = 0;
+ dev->ingress_err.ts = ktime_get();
+
+ /* Check for overload during this measurement period */
+ if (delta_us > DLB2_ISR_OVERLOAD_PERIOD_S * USEC_PER_SEC)
+ return;
+
+ /*
+ * Alarm interrupt overload: disable software-generated alarms,
+ * so only hardware problems (e.g. ECC errors) interrupt the PF.
+ */
+ dlb2_disable_ingress_error_alarms(&dev->hw);
+
+ dev->ingress_err.enabled = false;
+
+ dev_err(dev->dlb2_device,
+ "[%s()] Overloaded detected: disabling ingress error interrupts",
+ __func__);
+}
+
+/*
+ * The alarm handler logs the alarm syndrome and, for user-caused errors,
+ * reports the alarm to user-space through the per-domain device file interface.
+ *
+ * This function runs as a bottom-half handler because it can call printk
+ * and/or acquire a mutex. These alarms don't need to be handled immediately --
+ * they represent a serious, unexpected error (either in hardware or software)
+ * that can't be recovered without restarting the application or resetting the
+ * device. The VF->PF operations are also non-trivial and require running in a
+ * bottom-half handler.
+ */
+static irqreturn_t
+dlb2_service_intr_handler(int irq, void *hdlr_ptr)
+{
+ struct dlb2_dev *dev = (struct dlb2_dev *)hdlr_ptr;
+ union dlb2_sys_alarm_hw_synd r0;
+
+ dev_dbg(dev->dlb2_device, "DLB service interrupt fired\n");
+
+ mutex_lock(&dev->resource_mutex);
+
+ r0.val = DLB2_CSR_RD(&dev->hw, DLB2_SYS_ALARM_HW_SYND);
+
+ /*
+ * Clear the MSI-X ack bit before processing the VF->PF or watchdog
+ * timer interrupts. This order is necessary so that if an interrupt
+ * event arrives after reading the corresponding bit vector, the event
+ * won't be lost.
+ */
+ dlb2_ack_msix_interrupt(&dev->hw, DLB2_INT_NON_CQ);
+
+ if (r0.field.alarm & r0.field.valid)
+ dlb2_process_alarm_interrupt(&dev->hw);
+
+ if (dlb2_process_ingress_error_interrupt(&dev->hw))
+ dlb2_detect_ingress_err_overload(dev);
+
+ mutex_unlock(&dev->resource_mutex);
+
+ return IRQ_HANDLED;
+}
+
+static int
+dlb2_init_alarm_interrupts(struct dlb2_dev *dev,
+ struct pci_dev *pdev)
+{
+ int i, ret;
+
+ for (i = 0; i < DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS; i++) {
+ ret = devm_request_threaded_irq(&pdev->dev,
+ pci_irq_vector(pdev, i),
+ NULL,
+ dlb2_service_intr_handler,
+ IRQF_ONESHOT,
+ "dlb2_alarm",
+ dev);
+ if (ret)
+ return ret;
+
+ dev->intr.isr_registered[i] = true;
+ }
+
+ dlb2_enable_ingress_error_alarms(&dev->hw);
+
+ return 0;
+}
+
static irqreturn_t
dlb2_compressed_cq_intr_handler(int irq, void *hdlr_ptr)
{
@@ -178,6 +281,12 @@ dlb2_pf_init_interrupts(struct dlb2_dev *dev, struct pci_dev *pdev)
dev->intr.num_vectors = ret;
dev->intr.base_vector = pci_irq_vector(pdev, 0);
+ ret = dlb2_init_alarm_interrupts(dev, pdev);
+ if (ret) {
+ dlb2_pf_free_interrupts(dev, pdev);
+ return ret;
+ }
+
ret = dlb2_init_compressed_mode_interrupts(dev, pdev);
if (ret) {
dlb2_pf_free_interrupts(dev, pdev);
@@ -208,6 +317,16 @@ dlb2_pf_init_interrupts(struct dlb2_dev *dev, struct pci_dev *pdev)
static void
dlb2_pf_reinit_interrupts(struct dlb2_dev *dev)
{
+ /* Re-enable alarms after device reset */
+ dlb2_enable_ingress_error_alarms(&dev->hw);
+
+ if (!dev->ingress_err.enabled)
+ dev_err(dev->dlb2_device,
+ "[%s()] Re-enabling ingress error interrupts",
+ __func__);
+
+ dev->ingress_err.enabled = true;
+
dlb2_set_msix_mode(&dev->hw, DLB2_MSIX_MODE_COMPRESSED);
}
@@ -289,6 +408,9 @@ dlb2_pf_arm_cq_interrupt(struct dlb2_dev *dev,
static int
dlb2_pf_init_driver_state(struct dlb2_dev *dlb2_dev)
{
+ dlb2_dev->ingress_err.count = 0;
+ dlb2_dev->ingress_err.enabled = true;
+
mutex_init(&dlb2_dev->resource_mutex);
return 0;
@@ -761,6 +883,54 @@ static ssize_t dev_id_show(struct device *dev,
static DEVICE_ATTR_RO(dev_id);
+static ssize_t ingress_err_en_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct dlb2_dev *dlb2_dev = dev_get_drvdata(dev);
+ ssize_t ret;
+
+ mutex_lock(&dlb2_dev->resource_mutex);
+
+ ret = scnprintf(buf, PAGE_SIZE, "%d\n", dlb2_dev->ingress_err.enabled);
+
+ mutex_unlock(&dlb2_dev->resource_mutex);
+
+ return ret;
+}
+
+static ssize_t ingress_err_en_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf,
+ size_t count)
+{
+ struct dlb2_dev *dlb2_dev = dev_get_drvdata(dev);
+ unsigned long num;
+ ssize_t ret;
+
+ ret = kstrtoul(buf, 0, &num);
+ if (ret)
+ return -1;
+
+ mutex_lock(&dlb2_dev->resource_mutex);
+
+ if (!dlb2_dev->ingress_err.enabled && num) {
+ dlb2_enable_ingress_error_alarms(&dlb2_dev->hw);
+
+ dev_err(dlb2_dev->dlb2_device,
+ "[%s()] Re-enabling ingress error interrupts",
+ __func__);
+
+ dlb2_dev->ingress_err.enabled = true;
+ }
+
+ mutex_unlock(&dlb2_dev->resource_mutex);
+
+ return (ret == 0) ? count : ret;
+}
+
+static DEVICE_ATTR_RW(ingress_err_en);
+
static int
dlb2_pf_sysfs_create(struct dlb2_dev *dlb2_dev)
{
@@ -769,6 +939,10 @@ dlb2_pf_sysfs_create(struct dlb2_dev *dlb2_dev)
kobj = &dlb2_dev->pdev->dev.kobj;
+ ret = sysfs_create_file(kobj, &dev_attr_ingress_err_en.attr);
+ if (ret)
+ goto dlb2_ingress_err_en_attr_group_fail;
+
ret = sysfs_create_file(kobj, &dev_attr_dev_id.attr);
if (ret)
goto dlb2_dev_id_attr_group_fail;
@@ -800,6 +974,8 @@ dlb2_pf_sysfs_create(struct dlb2_dev *dlb2_dev)
dlb2_total_attr_group_fail:
sysfs_remove_file(kobj, &dev_attr_dev_id.attr);
dlb2_dev_id_attr_group_fail:
+ sysfs_remove_file(kobj, &dev_attr_ingress_err_en.attr);
+dlb2_ingress_err_en_attr_group_fail:
return ret;
}
@@ -815,6 +991,7 @@ dlb2_pf_sysfs_destroy(struct dlb2_dev *dlb2_dev)
sysfs_remove_group(kobj, &dlb2_avail_attr_group);
sysfs_remove_group(kobj, &dlb2_total_attr_group);
sysfs_remove_file(kobj, &dev_attr_dev_id.attr);
+ sysfs_remove_file(kobj, &dev_attr_ingress_err_en.attr);
}
static void
diff --git a/drivers/misc/dlb2/dlb2_resource.c b/drivers/misc/dlb2/dlb2_resource.c
index 4d751a0ce1db..2e0399ed455d 100644
--- a/drivers/misc/dlb2/dlb2_resource.c
+++ b/drivers/misc/dlb2/dlb2_resource.c
@@ -4917,7 +4917,7 @@ void dlb2_read_compressed_cq_intr_status(struct dlb2_hw *hw,
DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS);
}
-static void dlb2_ack_msix_interrupt(struct dlb2_hw *hw, int vector)
+void dlb2_ack_msix_interrupt(struct dlb2_hw *hw, int vector)
{
union dlb2_sys_msix_ack r0 = { {0} };
@@ -4994,6 +4994,255 @@ void dlb2_ack_compressed_cq_intr(struct dlb2_hw *hw,
dlb2_ack_msix_interrupt(hw, DLB2_PF_COMPRESSED_MODE_CQ_VECTOR_ID);
}
+void dlb2_enable_ingress_error_alarms(struct dlb2_hw *hw)
+{
+ union dlb2_sys_ingress_alarm_enbl r0;
+
+ r0.val = DLB2_CSR_RD(hw, DLB2_SYS_INGRESS_ALARM_ENBL);
+
+ r0.field.illegal_hcw = 1;
+ r0.field.illegal_pp = 1;
+ r0.field.illegal_pasid = 1;
+ r0.field.illegal_qid = 1;
+ r0.field.disabled_qid = 1;
+ r0.field.illegal_ldb_qid_cfg = 1;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_INGRESS_ALARM_ENBL, r0.val);
+}
+
+void dlb2_disable_ingress_error_alarms(struct dlb2_hw *hw)
+{
+ union dlb2_sys_ingress_alarm_enbl r0;
+
+ r0.val = DLB2_CSR_RD(hw, DLB2_SYS_INGRESS_ALARM_ENBL);
+
+ r0.field.illegal_hcw = 0;
+ r0.field.illegal_pp = 0;
+ r0.field.illegal_pasid = 0;
+ r0.field.illegal_qid = 0;
+ r0.field.disabled_qid = 0;
+ r0.field.illegal_ldb_qid_cfg = 0;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_INGRESS_ALARM_ENBL, r0.val);
+}
+
+static void dlb2_log_alarm_syndrome(struct dlb2_hw *hw,
+ const char *str,
+ union dlb2_sys_alarm_hw_synd r0)
+{
+ DLB2_HW_ERR(hw, "%s:\n", str);
+ DLB2_HW_ERR(hw, "\tsyndrome: 0x%x\n", r0.field.syndrome);
+ DLB2_HW_ERR(hw, "\trtype: 0x%x\n", r0.field.rtype);
+ DLB2_HW_ERR(hw, "\talarm: 0x%x\n", r0.field.alarm);
+ DLB2_HW_ERR(hw, "\tcwd: 0x%x\n", r0.field.cwd);
+ DLB2_HW_ERR(hw, "\tvf_pf_mb: 0x%x\n", r0.field.vf_pf_mb);
+ DLB2_HW_ERR(hw, "\tcls: 0x%x\n", r0.field.cls);
+ DLB2_HW_ERR(hw, "\taid: 0x%x\n", r0.field.aid);
+ DLB2_HW_ERR(hw, "\tunit: 0x%x\n", r0.field.unit);
+ DLB2_HW_ERR(hw, "\tsource: 0x%x\n", r0.field.source);
+ DLB2_HW_ERR(hw, "\tmore: 0x%x\n", r0.field.more);
+ DLB2_HW_ERR(hw, "\tvalid: 0x%x\n", r0.field.valid);
+}
+
+/* Note: this array's contents must match dlb2_alert_id() */
+static const char dlb2_alert_strings[NUM_DLB2_DOMAIN_ALERTS][128] = {
+ [DLB2_DOMAIN_ALERT_PP_ILLEGAL_ENQ] = "Illegal enqueue",
+ [DLB2_DOMAIN_ALERT_PP_EXCESS_TOKEN_POPS] = "Excess token pops",
+ [DLB2_DOMAIN_ALERT_ILLEGAL_HCW] = "Illegal HCW",
+ [DLB2_DOMAIN_ALERT_ILLEGAL_QID] = "Illegal QID",
+ [DLB2_DOMAIN_ALERT_DISABLED_QID] = "Disabled QID",
+};
+
+static void dlb2_log_pf_vf_syndrome(struct dlb2_hw *hw,
+ const char *str,
+ union dlb2_sys_alarm_pf_synd0 r0,
+ union dlb2_sys_alarm_pf_synd1 r1,
+ union dlb2_sys_alarm_pf_synd2 r2,
+ u32 alert_id)
+{
+ DLB2_HW_ERR(hw, "%s:\n", str);
+ if (alert_id < NUM_DLB2_DOMAIN_ALERTS)
+ DLB2_HW_ERR(hw, "Alert: %s\n", dlb2_alert_strings[alert_id]);
+ DLB2_HW_ERR(hw, "\tsyndrome: 0x%x\n", r0.field.syndrome);
+ DLB2_HW_ERR(hw, "\trtype: 0x%x\n", r0.field.rtype);
+ DLB2_HW_ERR(hw, "\tis_ldb: 0x%x\n", r0.field.is_ldb);
+ DLB2_HW_ERR(hw, "\tcls: 0x%x\n", r0.field.cls);
+ DLB2_HW_ERR(hw, "\taid: 0x%x\n", r0.field.aid);
+ DLB2_HW_ERR(hw, "\tunit: 0x%x\n", r0.field.unit);
+ DLB2_HW_ERR(hw, "\tsource: 0x%x\n", r0.field.source);
+ DLB2_HW_ERR(hw, "\tmore: 0x%x\n", r0.field.more);
+ DLB2_HW_ERR(hw, "\tvalid: 0x%x\n", r0.field.valid);
+ DLB2_HW_ERR(hw, "\tdsi: 0x%x\n", r1.field.dsi);
+ DLB2_HW_ERR(hw, "\tqid: 0x%x\n", r1.field.qid);
+ DLB2_HW_ERR(hw, "\tqtype: 0x%x\n", r1.field.qtype);
+ DLB2_HW_ERR(hw, "\tqpri: 0x%x\n", r1.field.qpri);
+ DLB2_HW_ERR(hw, "\tmsg_type: 0x%x\n", r1.field.msg_type);
+ DLB2_HW_ERR(hw, "\tlock_id: 0x%x\n", r2.field.lock_id);
+ DLB2_HW_ERR(hw, "\tmeas: 0x%x\n", r2.field.meas);
+ DLB2_HW_ERR(hw, "\tdebug: 0x%x\n", r2.field.debug);
+ DLB2_HW_ERR(hw, "\tcq_pop: 0x%x\n", r2.field.cq_pop);
+ DLB2_HW_ERR(hw, "\tqe_uhl: 0x%x\n", r2.field.qe_uhl);
+ DLB2_HW_ERR(hw, "\tqe_orsp: 0x%x\n", r2.field.qe_orsp);
+ DLB2_HW_ERR(hw, "\tqe_valid: 0x%x\n", r2.field.qe_valid);
+ DLB2_HW_ERR(hw, "\tcq_int_rearm: 0x%x\n", r2.field.cq_int_rearm);
+ DLB2_HW_ERR(hw, "\tdsi_error: 0x%x\n", r2.field.dsi_error);
+}
+
+static void dlb2_clear_syndrome_register(struct dlb2_hw *hw, u32 offset)
+{
+ union dlb2_sys_alarm_hw_synd r0 = { {0} };
+
+ r0.field.valid = 1;
+ r0.field.more = 1;
+
+ DLB2_CSR_WR(hw, offset, r0.val);
+}
+
+void dlb2_process_alarm_interrupt(struct dlb2_hw *hw)
+{
+ union dlb2_sys_alarm_hw_synd r0;
+
+ DLB2_HW_DBG(hw, "Processing alarm interrupt\n");
+
+ r0.val = DLB2_CSR_RD(hw, DLB2_SYS_ALARM_HW_SYND);
+
+ dlb2_log_alarm_syndrome(hw, "HW alarm syndrome", r0);
+
+ dlb2_clear_syndrome_register(hw, DLB2_SYS_ALARM_HW_SYND);
+}
+
+static void dlb2_process_ingress_error(struct dlb2_hw *hw,
+ union dlb2_sys_alarm_pf_synd0 r0,
+ u32 alert_id,
+ bool vf_error,
+ unsigned int vf_id)
+{
+ struct dlb2_dev *dev;
+ u32 domain_id;
+ bool is_ldb;
+ u8 port_id;
+ int ret;
+
+ port_id = r0.field.syndrome & 0x7F;
+ if (r0.field.source == DLB2_ALARM_HW_SOURCE_SYS)
+ is_ldb = r0.field.is_ldb;
+ else
+ is_ldb = (r0.field.syndrome & 0x80) != 0;
+
+ /* Get the domain ID and, if it's a VF domain, the virtual port ID */
+ if (is_ldb) {
+ struct dlb2_ldb_port *port;
+
+ port = dlb2_get_ldb_port_from_id(hw, port_id, vf_error, vf_id);
+ if (!port) {
+ DLB2_HW_ERR(hw,
+ "[%s()]: Internal error: unable to find LDB port\n\tport: %u, vf_error: %u, vf_id: %u\n",
+ __func__, port_id, vf_error, vf_id);
+ return;
+ }
+
+ domain_id = port->domain_id.phys_id;
+ } else {
+ struct dlb2_dir_pq_pair *port;
+
+ port = dlb2_get_dir_pq_from_id(hw, port_id, vf_error, vf_id);
+ if (!port) {
+ DLB2_HW_ERR(hw,
+ "[%s()]: Internal error: unable to find DIR port\n\tport: %u, vf_error: %u, vf_id: %u\n",
+ __func__, port_id, vf_error, vf_id);
+ return;
+ }
+
+ domain_id = port->domain_id.phys_id;
+ }
+
+ dev = container_of(hw, struct dlb2_dev, hw);
+
+ ret = dlb2_write_domain_alert(dev,
+ dev->sched_domains[domain_id],
+ alert_id,
+ (is_ldb << 8) | port_id);
+ if (ret)
+ DLB2_HW_ERR(hw,
+ "[%s()] Internal error: failed to notify\n",
+ __func__);
+}
+
+static u32 dlb2_alert_id(union dlb2_sys_alarm_pf_synd0 r0)
+{
+ if (r0.field.unit == DLB2_ALARM_HW_UNIT_CHP &&
+ r0.field.aid == DLB2_ALARM_HW_CHP_AID_ILLEGAL_ENQ)
+ return DLB2_DOMAIN_ALERT_PP_ILLEGAL_ENQ;
+ else if (r0.field.unit == DLB2_ALARM_HW_UNIT_CHP &&
+ r0.field.aid == DLB2_ALARM_HW_CHP_AID_EXCESS_TOKEN_POPS)
+ return DLB2_DOMAIN_ALERT_PP_EXCESS_TOKEN_POPS;
+ else if (r0.field.source == DLB2_ALARM_HW_SOURCE_SYS &&
+ r0.field.aid == DLB2_ALARM_SYS_AID_ILLEGAL_HCW)
+ return DLB2_DOMAIN_ALERT_ILLEGAL_HCW;
+ else if (r0.field.source == DLB2_ALARM_HW_SOURCE_SYS &&
+ r0.field.aid == DLB2_ALARM_SYS_AID_ILLEGAL_QID)
+ return DLB2_DOMAIN_ALERT_ILLEGAL_QID;
+ else if (r0.field.source == DLB2_ALARM_HW_SOURCE_SYS &&
+ r0.field.aid == DLB2_ALARM_SYS_AID_DISABLED_QID)
+ return DLB2_DOMAIN_ALERT_DISABLED_QID;
+ else
+ return NUM_DLB2_DOMAIN_ALERTS;
+}
+
+bool dlb2_process_ingress_error_interrupt(struct dlb2_hw *hw)
+{
+ union dlb2_sys_alarm_pf_synd0 r0;
+ union dlb2_sys_alarm_pf_synd1 r1;
+ union dlb2_sys_alarm_pf_synd2 r2;
+ u32 alert_id;
+ bool valid;
+ int i;
+
+ r0.val = DLB2_CSR_RD(hw, DLB2_SYS_ALARM_PF_SYND0);
+
+ valid = r0.field.valid;
+
+ if (r0.field.valid) {
+ r1.val = DLB2_CSR_RD(hw, DLB2_SYS_ALARM_PF_SYND1);
+ r2.val = DLB2_CSR_RD(hw, DLB2_SYS_ALARM_PF_SYND2);
+
+ alert_id = dlb2_alert_id(r0);
+
+ dlb2_log_pf_vf_syndrome(hw,
+ "PF Ingress error alarm",
+ r0, r1, r2, alert_id);
+
+ dlb2_clear_syndrome_register(hw, DLB2_SYS_ALARM_PF_SYND0);
+
+ dlb2_process_ingress_error(hw, r0, alert_id, false, 0);
+ }
+
+ for (i = 0; i < DLB2_MAX_NUM_VDEVS; i++) {
+ r0.val = DLB2_CSR_RD(hw, DLB2_SYS_ALARM_VF_SYND0(i));
+
+ valid |= r0.field.valid;
+
+ if (!r0.field.valid)
+ continue;
+
+ r1.val = DLB2_CSR_RD(hw, DLB2_SYS_ALARM_VF_SYND1(i));
+ r2.val = DLB2_CSR_RD(hw, DLB2_SYS_ALARM_VF_SYND2(i));
+
+ alert_id = dlb2_alert_id(r0);
+
+ dlb2_log_pf_vf_syndrome(hw,
+ "VF Ingress error alarm",
+ r0, r1, r2, alert_id);
+
+ dlb2_clear_syndrome_register(hw,
+ DLB2_SYS_ALARM_VF_SYND0(i));
+
+ dlb2_process_ingress_error(hw, r0, alert_id, true, i);
+ }
+
+ return valid;
+}
+
int dlb2_get_group_sequence_numbers(struct dlb2_hw *hw, unsigned int group_id)
{
if (group_id >= DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS)
diff --git a/drivers/misc/dlb2/dlb2_resource.h b/drivers/misc/dlb2/dlb2_resource.h
index 258ec1c11032..ac7347852662 100644
--- a/drivers/misc/dlb2/dlb2_resource.h
+++ b/drivers/misc/dlb2/dlb2_resource.h
@@ -517,6 +517,18 @@ int dlb2_configure_dir_cq_interrupt(struct dlb2_hw *hw,
u16 threshold);
/**
+ * dlb2_enable_ingress_error_alarms() - enable ingress error alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ */
+void dlb2_enable_ingress_error_alarms(struct dlb2_hw *hw);
+
+/**
+ * dlb2_disable_ingress_error_alarms() - disable ingress error alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ */
+void dlb2_disable_ingress_error_alarms(struct dlb2_hw *hw);
+
+/**
* dlb2_set_msix_mode() - enable certain hardware alarm interrupts
* @hw: dlb2_hw handle for a particular device.
* @mode: MSI-X mode (DLB2_MSIX_MODE_PACKED or DLB2_MSIX_MODE_COMPRESSED)
@@ -527,6 +539,16 @@ int dlb2_configure_dir_cq_interrupt(struct dlb2_hw *hw,
void dlb2_set_msix_mode(struct dlb2_hw *hw, int mode);
/**
+ * dlb2_ack_msix_interrupt() - Ack an MSI-X interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @vector: interrupt vector.
+ *
+ * Note: Only needed for PF service interrupts (vector 0). CQ interrupts are
+ * acked in dlb2_ack_compressed_cq_intr().
+ */
+void dlb2_ack_msix_interrupt(struct dlb2_hw *hw, int vector);
+
+/**
* dlb2_arm_cq_interrupt() - arm a CQ's interrupt
* @hw: dlb2_hw handle for a particular device.
* @port_id: port ID
@@ -582,6 +604,29 @@ void dlb2_ack_compressed_cq_intr(struct dlb2_hw *hw,
u32 *dir_interrupts);
/**
+ * dlb2_process_alarm_interrupt() - process an alarm interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads the alarm syndrome, logs its, and acks the interrupt.
+ * This function should be called from the alarm interrupt handler when
+ * interrupt vector DLB2_INT_ALARM fires.
+ */
+void dlb2_process_alarm_interrupt(struct dlb2_hw *hw);
+
+/**
+ * dlb2_process_ingress_error_interrupt() - process ingress error interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ *
+ * This function reads the alarm syndrome, logs it, notifies user-space, and
+ * acks the interrupt. This function should be called from the alarm interrupt
+ * handler when interrupt vector DLB2_INT_INGRESS_ERROR fires.
+ *
+ * Return:
+ * Returns true if an ingress error interrupt occurred, false otherwise
+ */
+bool dlb2_process_ingress_error_interrupt(struct dlb2_hw *hw);
+
+/**
* dlb2_get_group_sequence_numbers() - return a group's number of SNs per queue
* @hw: dlb2_hw handle for a particular device.
* @group_id: sequence number group ID.
--
2.13.6
These interfaces include files for reading the total and available device
resources, getting and setting sequence number group allocations, and
reading the device ID.
Signed-off-by: Gage Eads <[email protected]>
Reviewed-by: Björn Töpel <[email protected]>
---
Documentation/ABI/testing/sysfs-driver-dlb2 | 165 +++++++++++
drivers/misc/dlb2/dlb2_main.c | 11 +
drivers/misc/dlb2/dlb2_main.h | 3 +
drivers/misc/dlb2/dlb2_pf_ops.c | 423 ++++++++++++++++++++++++++++
4 files changed, 602 insertions(+)
diff --git a/Documentation/ABI/testing/sysfs-driver-dlb2 b/Documentation/ABI/testing/sysfs-driver-dlb2
index 38d0d3d92670..c5cb1cbb70f4 100644
--- a/Documentation/ABI/testing/sysfs-driver-dlb2
+++ b/Documentation/ABI/testing/sysfs-driver-dlb2
@@ -20,3 +20,168 @@ Description: Interface for setting the driver's reset timeout.
their device mappings can interfere with processes that use the
device after the reset completes. To ensure that user processes
have enough time to clean up, reset_timeout_s can be increased.
+
+What: /sys/bus/pci/devices/.../sequence_numbers/group<N>_sns_per_queue
+Date: June 22, 2020
+KernelVersion: 5.9
+Contact: [email protected]
+Description: Interface for configuring dlb2 load-balanced sequence numbers.
+
+ The DLB 2.0 has a fixed number of sequence numbers used for
+ ordered scheduling. They are divided among two sequence number
+ groups. A group can be configured to contain one queue with
+ 1,024 sequence numbers, or two queues with 512 sequence numbers
+ each, and so on, down to 16 queues with 64 sequence numbers
+ each.
+
+ When a load-balanced queue is configured with non-zero sequence
+ numbers, the driver finds a group configured for the same
+ number of sequence numbers and an available slot. If no such
+ groups are found, the queue cannot be configured.
+
+ Once the first ordered queue is configured, the sequence number
+ configurations are locked. The driver returns an error on writes
+ to locked sequence number configurations. When all ordered
+ queues are unconfigured, the sequence number configurations can
+ be changed again.
+
+ This file is only accessible for physical function DLB 2.0
+ devices.
+
+What: /sys/bus/pci/devices/.../total_resources/num_atomic_inflights
+What: /sys/bus/pci/devices/.../total_resources/num_dir_credits
+What: /sys/bus/pci/devices/.../total_resources/num_dir_ports
+What: /sys/bus/pci/devices/.../total_resources/num_hist_list_entries
+What: /sys/bus/pci/devices/.../total_resources/num_ldb_credits
+What: /sys/bus/pci/devices/.../total_resources/num_ldb_ports
+What: /sys/bus/pci/devices/.../total_resources/num_cos0_ldb_ports
+What: /sys/bus/pci/devices/.../total_resources/num_cos1_ldb_ports
+What: /sys/bus/pci/devices/.../total_resources/num_cos2_ldb_ports
+What: /sys/bus/pci/devices/.../total_resources/num_cos3_ldb_ports
+What: /sys/bus/pci/devices/.../total_resources/num_ldb_queues
+What: /sys/bus/pci/devices/.../total_resources/num_sched_domains
+Date: June 22, 2020
+KernelVersion: 5.9
+Contact: [email protected]
+Description:
+ The total_resources subdirectory contains read-only files that
+ indicate the total number of resources in the device.
+
+ num_atomic_inflights: Total number of atomic inflights in the
+ device. Atomic inflights refers to the
+ on-device storage used by the atomic
+ scheduler.
+
+ num_dir_credits: Total number of directed credits in the
+ device.
+
+ num_dir_ports: Total number of directed ports (and
+ queues) in the device.
+
+ num_hist_list_entries: Total number of history list entries in
+ the device.
+
+ num_ldb_credits: Total number of load-balanced credits in
+ the device.
+
+ num_ldb_ports: Total number of load-balanced ports in
+ the device.
+
+ num_cos<M>_ldb_ports: Total number of load-balanced ports
+ belonging to class-of-service M in the
+ device.
+
+ num_ldb_queues: Total number of load-balanced queues in
+ the device.
+
+ num_sched_domains: Total number of scheduling domains in the
+ device.
+
+What: /sys/bus/pci/devices/.../avail_resources/num_atomic_inflights
+What: /sys/bus/pci/devices/.../avail_resources/num_dir_credits
+What: /sys/bus/pci/devices/.../avail_resources/num_dir_ports
+What: /sys/bus/pci/devices/.../avail_resources/num_hist_list_entries
+What: /sys/bus/pci/devices/.../avail_resources/num_ldb_credits
+What: /sys/bus/pci/devices/.../avail_resources/num_ldb_ports
+What: /sys/bus/pci/devices/.../avail_resources/num_cos0_ldb_ports
+What: /sys/bus/pci/devices/.../avail_resources/num_cos1_ldb_ports
+What: /sys/bus/pci/devices/.../avail_resources/num_cos2_ldb_ports
+What: /sys/bus/pci/devices/.../avail_resources/num_cos3_ldb_ports
+What: /sys/bus/pci/devices/.../avail_resources/num_ldb_queues
+What: /sys/bus/pci/devices/.../avail_resources/num_sched_domains
+What: /sys/bus/pci/devices/.../avail_resources/max_ctg_hl_entries
+Date: June 22, 2020
+KernelVersion: 5.9
+Contact: [email protected]
+Description:
+ The avail_resources subdirectory contains read-only files that
+ indicate the available number of resources in the device.
+ "Available" here means resources that are not currently in use
+ by an application or, in the case of a physical function
+ device, assigned to a virtual function.
+
+ num_atomic_inflights: Available number of atomic inflights in
+ the device.
+
+ num_dir_ports: Available number of directed ports (and
+ queues) in the device.
+
+ num_hist_list_entries: Available number of history list entries
+ in the device.
+
+ num_ldb_credits: Available number of load-balanced credits
+ in the device.
+
+ num_ldb_ports: Available number of load-balanced ports
+ in the device.
+
+ num_cos<M>_ldb_ports: Available number of load-balanced ports
+ belonging to class-of-service M in the
+ device.
+
+ num_ldb_queues: Available number of load-balanced queues
+ in the device.
+
+ num_sched_domains: Available number of scheduling domains in
+ the device.
+
+ max_ctg_hl_entries: Maximum contiguous history list entries
+ available in the device.
+
+ Each scheduling domain is created with
+ an allocation of history list entries,
+ and each domain's allocation of entries
+ must be contiguous.
+
+What: /sys/bus/pci/devices/.../cos_bw/cos<N>_bw_percent
+Date: June 22, 2020
+KernelVersion: 5.9
+Contact: [email protected]
+Description: Interface for configuring dlb2 class-of-service bandwidths.
+
+ The DLB 2.0 supports four load-balanced port classes of service
+ (CoS). Each CoS receives a guaranteed percentage of the
+ load-balanced scheduler's bandwidth, and any unreserved
+ bandwidth is divided among the four CoS.
+
+ By default, each CoS is guaranteed 25% of the bandwidth. Writing
+ these files allows the user to change the CoS, subject to the
+ constraint that the total reserved bandwidth of all CoS cannot
+ exceed 100%.
+
+ Bandwidth reservations can be modified during run-time.
+
+ These files are only accessible for physical function DLB 2.0
+ devices.
+
+What: /sys/bus/pci/devices/.../dev_id
+Date: June 22, 2020
+KernelVersion: 5.9
+Contact: [email protected]
+Description: Device ID used in /dev, i.e. /dev/dlb<device ID>
+
+ Each DLB 2.0 PF and VF device is granted a unique ID by the
+ kernel driver, and this ID is used to construct the device's
+ /dev directory: /dev/dlb<device ID>. This sysfs file can be read
+ to determine a device's ID, which allows the user to map a
+ device file to a PCI BDF.
diff --git a/drivers/misc/dlb2/dlb2_main.c b/drivers/misc/dlb2/dlb2_main.c
index bc065788061a..b000319599aa 100644
--- a/drivers/misc/dlb2/dlb2_main.c
+++ b/drivers/misc/dlb2/dlb2_main.c
@@ -603,6 +603,10 @@ static int dlb2_probe(struct pci_dev *pdev,
if (ret)
goto dma_set_mask_fail;
+ ret = dlb2_dev->ops->sysfs_create(dlb2_dev);
+ if (ret)
+ goto sysfs_create_fail;
+
/*
* PM enable must be done before any other MMIO accesses, and this
* setting is persistent across device reset.
@@ -653,6 +657,8 @@ static int dlb2_probe(struct pci_dev *pdev,
init_interrupts_fail:
dlb2_reset_fail:
wait_for_device_ready_fail:
+ dlb2_dev->ops->sysfs_destroy(dlb2_dev);
+sysfs_create_fail:
dma_set_mask_fail:
dlb2_dev->ops->device_destroy(dlb2_dev, dlb2_class);
device_add_fail:
@@ -694,6 +700,8 @@ static void dlb2_remove(struct pci_dev *pdev)
dlb2_release_device_memory(dlb2_dev);
+ dlb2_dev->ops->sysfs_destroy(dlb2_dev);
+
dlb2_dev->ops->device_destroy(dlb2_dev, dlb2_class);
dlb2_dev->ops->cdev_del(dlb2_dev);
@@ -719,6 +727,9 @@ static void dlb2_reset_hardware_state(struct dlb2_dev *dev, bool issue_flr)
/* Reinitialize interrupt configuration */
dev->ops->reinit_interrupts(dev);
+ /* Reset configuration done through the sysfs */
+ dev->ops->sysfs_reapply(dev);
+
/* Reinitialize any other hardware state */
dev->ops->init_hardware(dev);
}
diff --git a/drivers/misc/dlb2/dlb2_main.h b/drivers/misc/dlb2/dlb2_main.h
index 45050403a4be..ab8cebe6220f 100644
--- a/drivers/misc/dlb2/dlb2_main.h
+++ b/drivers/misc/dlb2/dlb2_main.h
@@ -57,6 +57,9 @@ struct dlb2_device_ops {
dev_t base,
const struct file_operations *fops);
void (*cdev_del)(struct dlb2_dev *dlb2_dev);
+ int (*sysfs_create)(struct dlb2_dev *dlb2_dev);
+ void (*sysfs_destroy)(struct dlb2_dev *dlb2_dev);
+ void (*sysfs_reapply)(struct dlb2_dev *dev);
int (*init_interrupts)(struct dlb2_dev *dev, struct pci_dev *pdev);
int (*enable_ldb_cq_interrupts)(struct dlb2_dev *dev,
int domain_id,
diff --git a/drivers/misc/dlb2/dlb2_pf_ops.c b/drivers/misc/dlb2/dlb2_pf_ops.c
index 771fd793870b..4f0f0e41ab8d 100644
--- a/drivers/misc/dlb2/dlb2_pf_ops.c
+++ b/drivers/misc/dlb2/dlb2_pf_ops.c
@@ -416,6 +416,426 @@ dlb2_pf_init_hardware(struct dlb2_dev *dlb2_dev)
}
/*****************************/
+/****** Sysfs callbacks ******/
+/*****************************/
+
+#define DLB2_TOTAL_SYSFS_SHOW(name, macro) \
+static ssize_t total_##name##_show( \
+ struct device *dev, \
+ struct device_attribute *attr, \
+ char *buf) \
+{ \
+ int val = DLB2_MAX_NUM_##macro; \
+ \
+ return scnprintf(buf, PAGE_SIZE, "%d\n", val); \
+}
+
+DLB2_TOTAL_SYSFS_SHOW(num_sched_domains, DOMAINS)
+DLB2_TOTAL_SYSFS_SHOW(num_ldb_queues, LDB_QUEUES)
+DLB2_TOTAL_SYSFS_SHOW(num_ldb_ports, LDB_PORTS)
+DLB2_TOTAL_SYSFS_SHOW(num_cos0_ldb_ports, LDB_PORTS / DLB2_NUM_COS_DOMAINS)
+DLB2_TOTAL_SYSFS_SHOW(num_cos1_ldb_ports, LDB_PORTS / DLB2_NUM_COS_DOMAINS)
+DLB2_TOTAL_SYSFS_SHOW(num_cos2_ldb_ports, LDB_PORTS / DLB2_NUM_COS_DOMAINS)
+DLB2_TOTAL_SYSFS_SHOW(num_cos3_ldb_ports, LDB_PORTS / DLB2_NUM_COS_DOMAINS)
+DLB2_TOTAL_SYSFS_SHOW(num_dir_ports, DIR_PORTS)
+DLB2_TOTAL_SYSFS_SHOW(num_ldb_credits, LDB_CREDITS)
+DLB2_TOTAL_SYSFS_SHOW(num_dir_credits, DIR_CREDITS)
+DLB2_TOTAL_SYSFS_SHOW(num_atomic_inflights, AQED_ENTRIES)
+DLB2_TOTAL_SYSFS_SHOW(num_hist_list_entries, HIST_LIST_ENTRIES)
+
+#define DLB2_AVAIL_SYSFS_SHOW(name) \
+static ssize_t avail_##name##_show( \
+ struct device *dev, \
+ struct device_attribute *attr, \
+ char *buf) \
+{ \
+ struct dlb2_dev *dlb2_dev = dev_get_drvdata(dev); \
+ struct dlb2_get_num_resources_args arg; \
+ struct dlb2_hw *hw = &dlb2_dev->hw; \
+ int val; \
+ \
+ mutex_lock(&dlb2_dev->resource_mutex); \
+ \
+ val = dlb2_hw_get_num_resources(hw, &arg, false, 0); \
+ \
+ mutex_unlock(&dlb2_dev->resource_mutex); \
+ \
+ if (val) \
+ return -1; \
+ \
+ val = arg.name; \
+ \
+ return scnprintf(buf, PAGE_SIZE, "%d\n", val); \
+}
+
+#define DLB2_AVAIL_SYSFS_SHOW_COS(name, idx) \
+static ssize_t avail_##name##_show( \
+ struct device *dev, \
+ struct device_attribute *attr, \
+ char *buf) \
+{ \
+ struct dlb2_dev *dlb2_dev = dev_get_drvdata(dev); \
+ struct dlb2_get_num_resources_args arg; \
+ struct dlb2_hw *hw = &dlb2_dev->hw; \
+ int val; \
+ \
+ mutex_lock(&dlb2_dev->resource_mutex); \
+ \
+ val = dlb2_hw_get_num_resources(hw, &arg, false, 0); \
+ \
+ mutex_unlock(&dlb2_dev->resource_mutex); \
+ \
+ if (val) \
+ return -1; \
+ \
+ val = arg.num_cos_ldb_ports[idx]; \
+ \
+ return scnprintf(buf, PAGE_SIZE, "%d\n", val); \
+}
+
+DLB2_AVAIL_SYSFS_SHOW(num_sched_domains)
+DLB2_AVAIL_SYSFS_SHOW(num_ldb_queues)
+DLB2_AVAIL_SYSFS_SHOW(num_ldb_ports)
+DLB2_AVAIL_SYSFS_SHOW_COS(num_cos0_ldb_ports, 0)
+DLB2_AVAIL_SYSFS_SHOW_COS(num_cos1_ldb_ports, 1)
+DLB2_AVAIL_SYSFS_SHOW_COS(num_cos2_ldb_ports, 2)
+DLB2_AVAIL_SYSFS_SHOW_COS(num_cos3_ldb_ports, 3)
+DLB2_AVAIL_SYSFS_SHOW(num_dir_ports)
+DLB2_AVAIL_SYSFS_SHOW(num_ldb_credits)
+DLB2_AVAIL_SYSFS_SHOW(num_dir_credits)
+DLB2_AVAIL_SYSFS_SHOW(num_atomic_inflights)
+DLB2_AVAIL_SYSFS_SHOW(num_hist_list_entries)
+
+static ssize_t max_ctg_hl_entries_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct dlb2_dev *dlb2_dev = dev_get_drvdata(dev);
+ struct dlb2_get_num_resources_args arg;
+ struct dlb2_hw *hw = &dlb2_dev->hw;
+ int val;
+
+ mutex_lock(&dlb2_dev->resource_mutex);
+
+ val = dlb2_hw_get_num_resources(hw, &arg, false, 0);
+
+ mutex_unlock(&dlb2_dev->resource_mutex);
+
+ if (val)
+ return -1;
+
+ val = arg.max_contiguous_hist_list_entries;
+
+ return scnprintf(buf, PAGE_SIZE, "%d\n", val);
+}
+
+/*
+ * Device attribute name doesn't match the show function name, so we define our
+ * own DEVICE_ATTR macro.
+ */
+#define DLB2_DEVICE_ATTR_RO(_prefix, _name) \
+struct device_attribute dev_attr_##_prefix##_##_name = {\
+ .attr = { .name = __stringify(_name), .mode = 0444 },\
+ .show = _prefix##_##_name##_show,\
+}
+
+static DLB2_DEVICE_ATTR_RO(total, num_sched_domains);
+static DLB2_DEVICE_ATTR_RO(total, num_ldb_queues);
+static DLB2_DEVICE_ATTR_RO(total, num_ldb_ports);
+static DLB2_DEVICE_ATTR_RO(total, num_cos0_ldb_ports);
+static DLB2_DEVICE_ATTR_RO(total, num_cos1_ldb_ports);
+static DLB2_DEVICE_ATTR_RO(total, num_cos2_ldb_ports);
+static DLB2_DEVICE_ATTR_RO(total, num_cos3_ldb_ports);
+static DLB2_DEVICE_ATTR_RO(total, num_dir_ports);
+static DLB2_DEVICE_ATTR_RO(total, num_ldb_credits);
+static DLB2_DEVICE_ATTR_RO(total, num_dir_credits);
+static DLB2_DEVICE_ATTR_RO(total, num_atomic_inflights);
+static DLB2_DEVICE_ATTR_RO(total, num_hist_list_entries);
+
+static struct attribute *dlb2_total_attrs[] = {
+ &dev_attr_total_num_sched_domains.attr,
+ &dev_attr_total_num_ldb_queues.attr,
+ &dev_attr_total_num_ldb_ports.attr,
+ &dev_attr_total_num_cos0_ldb_ports.attr,
+ &dev_attr_total_num_cos1_ldb_ports.attr,
+ &dev_attr_total_num_cos2_ldb_ports.attr,
+ &dev_attr_total_num_cos3_ldb_ports.attr,
+ &dev_attr_total_num_dir_ports.attr,
+ &dev_attr_total_num_ldb_credits.attr,
+ &dev_attr_total_num_dir_credits.attr,
+ &dev_attr_total_num_atomic_inflights.attr,
+ &dev_attr_total_num_hist_list_entries.attr,
+ NULL
+};
+
+static const struct attribute_group dlb2_total_attr_group = {
+ .attrs = dlb2_total_attrs,
+ .name = "total_resources",
+};
+
+static DLB2_DEVICE_ATTR_RO(avail, num_sched_domains);
+static DLB2_DEVICE_ATTR_RO(avail, num_ldb_queues);
+static DLB2_DEVICE_ATTR_RO(avail, num_ldb_ports);
+static DLB2_DEVICE_ATTR_RO(avail, num_cos0_ldb_ports);
+static DLB2_DEVICE_ATTR_RO(avail, num_cos1_ldb_ports);
+static DLB2_DEVICE_ATTR_RO(avail, num_cos2_ldb_ports);
+static DLB2_DEVICE_ATTR_RO(avail, num_cos3_ldb_ports);
+static DLB2_DEVICE_ATTR_RO(avail, num_dir_ports);
+static DLB2_DEVICE_ATTR_RO(avail, num_ldb_credits);
+static DLB2_DEVICE_ATTR_RO(avail, num_dir_credits);
+static DLB2_DEVICE_ATTR_RO(avail, num_atomic_inflights);
+static DLB2_DEVICE_ATTR_RO(avail, num_hist_list_entries);
+static DEVICE_ATTR_RO(max_ctg_hl_entries);
+
+static struct attribute *dlb2_avail_attrs[] = {
+ &dev_attr_avail_num_sched_domains.attr,
+ &dev_attr_avail_num_ldb_queues.attr,
+ &dev_attr_avail_num_ldb_ports.attr,
+ &dev_attr_avail_num_cos0_ldb_ports.attr,
+ &dev_attr_avail_num_cos1_ldb_ports.attr,
+ &dev_attr_avail_num_cos2_ldb_ports.attr,
+ &dev_attr_avail_num_cos3_ldb_ports.attr,
+ &dev_attr_avail_num_dir_ports.attr,
+ &dev_attr_avail_num_ldb_credits.attr,
+ &dev_attr_avail_num_dir_credits.attr,
+ &dev_attr_avail_num_atomic_inflights.attr,
+ &dev_attr_avail_num_hist_list_entries.attr,
+ &dev_attr_max_ctg_hl_entries.attr,
+ NULL
+};
+
+static const struct attribute_group dlb2_avail_attr_group = {
+ .attrs = dlb2_avail_attrs,
+ .name = "avail_resources",
+};
+
+#define DLB2_GROUP_SNS_PER_QUEUE_SHOW(id) \
+static ssize_t group##id##_sns_per_queue_show( \
+ struct device *dev, \
+ struct device_attribute *attr, \
+ char *buf) \
+{ \
+ struct dlb2_dev *dlb2_dev = \
+ dev_get_drvdata(dev); \
+ struct dlb2_hw *hw = &dlb2_dev->hw; \
+ int val; \
+ \
+ mutex_lock(&dlb2_dev->resource_mutex); \
+ \
+ val = dlb2_get_group_sequence_numbers(hw, id); \
+ \
+ mutex_unlock(&dlb2_dev->resource_mutex); \
+ \
+ return scnprintf(buf, PAGE_SIZE, "%d\n", val); \
+}
+
+DLB2_GROUP_SNS_PER_QUEUE_SHOW(0)
+DLB2_GROUP_SNS_PER_QUEUE_SHOW(1)
+
+#define DLB2_GROUP_SNS_PER_QUEUE_STORE(id) \
+static ssize_t group##id##_sns_per_queue_store( \
+ struct device *dev, \
+ struct device_attribute *attr, \
+ const char *buf, \
+ size_t count) \
+{ \
+ struct dlb2_dev *dlb2_dev = dev_get_drvdata(dev); \
+ struct dlb2_hw *hw = &dlb2_dev->hw; \
+ unsigned long val; \
+ int err; \
+ \
+ err = kstrtoul(buf, 0, &val); \
+ if (err) \
+ return -1; \
+ \
+ mutex_lock(&dlb2_dev->resource_mutex); \
+ \
+ err = dlb2_set_group_sequence_numbers(hw, id, val); \
+ \
+ mutex_unlock(&dlb2_dev->resource_mutex); \
+ \
+ return err ? err : count; \
+}
+
+DLB2_GROUP_SNS_PER_QUEUE_STORE(0)
+DLB2_GROUP_SNS_PER_QUEUE_STORE(1)
+
+/* RW sysfs files in the sequence_numbers/ subdirectory */
+static DEVICE_ATTR_RW(group0_sns_per_queue);
+static DEVICE_ATTR_RW(group1_sns_per_queue);
+
+static struct attribute *dlb2_sequence_number_attrs[] = {
+ &dev_attr_group0_sns_per_queue.attr,
+ &dev_attr_group1_sns_per_queue.attr,
+ NULL
+};
+
+static const struct attribute_group dlb2_sequence_number_attr_group = {
+ .attrs = dlb2_sequence_number_attrs,
+ .name = "sequence_numbers"
+};
+
+#define DLB2_COS_BW_PERCENT_SHOW(id) \
+static ssize_t cos##id##_bw_percent_show( \
+ struct device *dev, \
+ struct device_attribute *attr, \
+ char *buf) \
+{ \
+ struct dlb2_dev *dlb2_dev = \
+ dev_get_drvdata(dev); \
+ struct dlb2_hw *hw = &dlb2_dev->hw; \
+ int val; \
+ \
+ mutex_lock(&dlb2_dev->resource_mutex); \
+ \
+ val = dlb2_hw_get_cos_bandwidth(hw, id); \
+ \
+ mutex_unlock(&dlb2_dev->resource_mutex); \
+ \
+ return scnprintf(buf, PAGE_SIZE, "%d\n", val); \
+}
+
+DLB2_COS_BW_PERCENT_SHOW(0)
+DLB2_COS_BW_PERCENT_SHOW(1)
+DLB2_COS_BW_PERCENT_SHOW(2)
+DLB2_COS_BW_PERCENT_SHOW(3)
+
+#define DLB2_COS_BW_PERCENT_STORE(id) \
+static ssize_t cos##id##_bw_percent_store( \
+ struct device *dev, \
+ struct device_attribute *attr, \
+ const char *buf, \
+ size_t count) \
+{ \
+ struct dlb2_dev *dlb2_dev = \
+ dev_get_drvdata(dev); \
+ struct dlb2_hw *hw = &dlb2_dev->hw; \
+ unsigned long val; \
+ int err; \
+ \
+ err = kstrtoul(buf, 0, &val); \
+ if (err) \
+ return -1; \
+ \
+ mutex_lock(&dlb2_dev->resource_mutex); \
+ \
+ err = dlb2_hw_set_cos_bandwidth(hw, id, val); \
+ \
+ mutex_unlock(&dlb2_dev->resource_mutex); \
+ \
+ return err ? err : count; \
+}
+
+DLB2_COS_BW_PERCENT_STORE(0)
+DLB2_COS_BW_PERCENT_STORE(1)
+DLB2_COS_BW_PERCENT_STORE(2)
+DLB2_COS_BW_PERCENT_STORE(3)
+
+/* RW sysfs files in the sequence_numbers/ subdirectory */
+static DEVICE_ATTR_RW(cos0_bw_percent);
+static DEVICE_ATTR_RW(cos1_bw_percent);
+static DEVICE_ATTR_RW(cos2_bw_percent);
+static DEVICE_ATTR_RW(cos3_bw_percent);
+
+static struct attribute *dlb2_cos_bw_percent_attrs[] = {
+ &dev_attr_cos0_bw_percent.attr,
+ &dev_attr_cos1_bw_percent.attr,
+ &dev_attr_cos2_bw_percent.attr,
+ &dev_attr_cos3_bw_percent.attr,
+ NULL
+};
+
+static const struct attribute_group dlb2_cos_bw_percent_attr_group = {
+ .attrs = dlb2_cos_bw_percent_attrs,
+ .name = "cos_bw"
+};
+
+static ssize_t dev_id_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct dlb2_dev *dlb2_dev = dev_get_drvdata(dev);
+
+ return scnprintf(buf, PAGE_SIZE, "%d\n", dlb2_dev->id);
+}
+
+static DEVICE_ATTR_RO(dev_id);
+
+static int
+dlb2_pf_sysfs_create(struct dlb2_dev *dlb2_dev)
+{
+ struct kobject *kobj;
+ int ret;
+
+ kobj = &dlb2_dev->pdev->dev.kobj;
+
+ ret = sysfs_create_file(kobj, &dev_attr_dev_id.attr);
+ if (ret)
+ goto dlb2_dev_id_attr_group_fail;
+
+ ret = sysfs_create_group(kobj, &dlb2_total_attr_group);
+ if (ret)
+ goto dlb2_total_attr_group_fail;
+
+ ret = sysfs_create_group(kobj, &dlb2_avail_attr_group);
+ if (ret)
+ goto dlb2_avail_attr_group_fail;
+
+ ret = sysfs_create_group(kobj, &dlb2_sequence_number_attr_group);
+ if (ret)
+ goto dlb2_sn_attr_group_fail;
+
+ ret = sysfs_create_group(kobj, &dlb2_cos_bw_percent_attr_group);
+ if (ret)
+ goto dlb2_cos_bw_percent_attr_group_fail;
+
+ return 0;
+
+dlb2_cos_bw_percent_attr_group_fail:
+ sysfs_remove_group(kobj, &dlb2_sequence_number_attr_group);
+dlb2_sn_attr_group_fail:
+ sysfs_remove_group(kobj, &dlb2_avail_attr_group);
+dlb2_avail_attr_group_fail:
+ sysfs_remove_group(kobj, &dlb2_total_attr_group);
+dlb2_total_attr_group_fail:
+ sysfs_remove_file(kobj, &dev_attr_dev_id.attr);
+dlb2_dev_id_attr_group_fail:
+ return ret;
+}
+
+static void
+dlb2_pf_sysfs_destroy(struct dlb2_dev *dlb2_dev)
+{
+ struct kobject *kobj;
+
+ kobj = &dlb2_dev->pdev->dev.kobj;
+
+ sysfs_remove_group(kobj, &dlb2_cos_bw_percent_attr_group);
+ sysfs_remove_group(kobj, &dlb2_sequence_number_attr_group);
+ sysfs_remove_group(kobj, &dlb2_avail_attr_group);
+ sysfs_remove_group(kobj, &dlb2_total_attr_group);
+ sysfs_remove_file(kobj, &dev_attr_dev_id.attr);
+}
+
+static void
+dlb2_pf_sysfs_reapply_configuration(struct dlb2_dev *dev)
+{
+ int i;
+
+ for (i = 0; i < DLB2_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+ int num_sns = dlb2_get_group_sequence_numbers(&dev->hw, i);
+
+ dlb2_set_group_sequence_numbers(&dev->hw, i, num_sns);
+ }
+
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ int bw = dlb2_hw_get_cos_bandwidth(&dev->hw, i);
+
+ dlb2_hw_set_cos_bandwidth(&dev->hw, i, bw);
+ }
+}
+
+/*****************************/
/****** IOCTL callbacks ******/
/*****************************/
@@ -649,6 +1069,9 @@ struct dlb2_device_ops dlb2_pf_ops = {
.device_destroy = dlb2_pf_device_destroy,
.cdev_add = dlb2_pf_cdev_add,
.cdev_del = dlb2_pf_cdev_del,
+ .sysfs_create = dlb2_pf_sysfs_create,
+ .sysfs_destroy = dlb2_pf_sysfs_destroy,
+ .sysfs_reapply = dlb2_pf_sysfs_reapply_configuration,
.init_interrupts = dlb2_pf_init_interrupts,
.enable_ldb_cq_interrupts = dlb2_pf_enable_ldb_cq_interrupts,
.enable_dir_cq_interrupts = dlb2_pf_enable_dir_cq_interrupts,
--
2.13.6
DLB 2.0 supports interrupt-driven applications with per-CQ interrupts. A CQ
interrupt is armed by user-space software by enqueueing a special command
to the device through the port's MMIO window, and the interrupt fires when
the armed CQ becomes non-empty.
All CQ interrupts use a single MSI-X interrupt vector, and the ISR reads
bitmap registers to determine which CQ(s)'s interrupt fired. For each of
these CQs, the driver wakes up a wait-queue -- on which user-space threads
may be blocking waiting for the interrupt. User-space software calls a
block-on-CQ-interrupt ioctl in order to block on the wait queue.
A CQ's interrupt is enabled when its port is configured, and interrupts are
enabled/disabled when a port is enabled/disabled. If a port is disabled and
a thread is blocked on the wait queue, the thread is woken and returned to
user-space.
Signed-off-by: Gage Eads <[email protected]>
Reviewed-by: Björn Töpel <[email protected]>
---
drivers/misc/dlb2/Makefile | 1 +
drivers/misc/dlb2/dlb2_hw_types.h | 13 ++
drivers/misc/dlb2/dlb2_intr.c | 130 ++++++++++++++++
drivers/misc/dlb2/dlb2_intr.h | 29 ++++
drivers/misc/dlb2/dlb2_ioctl.c | 70 +++++++++
drivers/misc/dlb2/dlb2_main.c | 20 ++-
drivers/misc/dlb2/dlb2_main.h | 43 ++++++
drivers/misc/dlb2/dlb2_pf_ops.c | 211 ++++++++++++++++++++++++++
drivers/misc/dlb2/dlb2_resource.c | 306 ++++++++++++++++++++++++++++++++++++++
drivers/misc/dlb2/dlb2_resource.h | 128 ++++++++++++++++
include/uapi/linux/dlb2_user.h | 37 +++++
11 files changed, 987 insertions(+), 1 deletion(-)
create mode 100644 drivers/misc/dlb2/dlb2_intr.c
create mode 100644 drivers/misc/dlb2/dlb2_intr.h
diff --git a/drivers/misc/dlb2/Makefile b/drivers/misc/dlb2/Makefile
index 12361461dcff..64ec27489b73 100644
--- a/drivers/misc/dlb2/Makefile
+++ b/drivers/misc/dlb2/Makefile
@@ -6,6 +6,7 @@ obj-$(CONFIG_INTEL_DLB2) := dlb2.o
dlb2-objs := \
dlb2_main.o \
+ dlb2_intr.o \
dlb2_file.o \
dlb2_ioctl.o \
dlb2_pf_ops.o \
diff --git a/drivers/misc/dlb2/dlb2_hw_types.h b/drivers/misc/dlb2/dlb2_hw_types.h
index 7db59157da15..986d35cbeaf9 100644
--- a/drivers/misc/dlb2/dlb2_hw_types.h
+++ b/drivers/misc/dlb2/dlb2_hw_types.h
@@ -48,6 +48,19 @@
#define DLB2_MAX_QID_EMPTY_CHECK_LOOPS (32 * 64 * 1024 * (800 / 30))
#define DLB2_HZ 800000000
+/* Interrupt related macros */
+#define DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS 1
+#define DLB2_PF_NUM_CQ_INTERRUPT_VECTORS 64
+#define DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS \
+ (DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + \
+ DLB2_PF_NUM_CQ_INTERRUPT_VECTORS)
+#define DLB2_PF_NUM_COMPRESSED_MODE_VECTORS \
+ (DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS + 1)
+#define DLB2_PF_NUM_PACKED_MODE_VECTORS \
+ DLB2_PF_TOTAL_NUM_INTERRUPT_VECTORS
+#define DLB2_PF_COMPRESSED_MODE_CQ_VECTOR_ID \
+ DLB2_PF_NUM_NON_CQ_INTERRUPT_VECTORS
+
/*
* Hardware-defined base addresses. Those prefixed 'DLB2_DRV' are only used by
* the PF driver.
diff --git a/drivers/misc/dlb2/dlb2_intr.c b/drivers/misc/dlb2/dlb2_intr.c
new file mode 100644
index 000000000000..0e20197e96fb
--- /dev/null
+++ b/drivers/misc/dlb2/dlb2_intr.c
@@ -0,0 +1,130 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(c) 2017-2020 Intel Corporation */
+
+#include <linux/interrupt.h>
+#include <linux/uaccess.h>
+
+#include "dlb2_intr.h"
+#include "dlb2_main.h"
+#include "dlb2_resource.h"
+
+void dlb2_wake_thread(struct dlb2_dev *dev,
+ struct dlb2_cq_intr *intr,
+ enum dlb2_wake_reason reason)
+{
+ switch (reason) {
+ case WAKE_CQ_INTR:
+ WRITE_ONCE(intr->wake, true);
+ break;
+ case WAKE_PORT_DISABLED:
+ WRITE_ONCE(intr->disabled, true);
+ break;
+ default:
+ break;
+ }
+
+ wake_up_interruptible(&intr->wq_head);
+}
+
+static inline bool wake_condition(struct dlb2_cq_intr *intr,
+ struct dlb2_dev *dev,
+ struct dlb2_domain *domain)
+{
+ return (READ_ONCE(intr->wake) || READ_ONCE(intr->disabled));
+}
+
+struct dlb2_dequeue_qe {
+ u8 rsvd0[15];
+ u8 cq_gen:1;
+ u8 rsvd1:7;
+} __packed;
+
+/**
+ * dlb2_cq_empty() - determine whether a CQ is empty
+ * @dev: struct dlb2_dev pointer.
+ * @user_cq_va: User VA pointing to next CQ entry.
+ * @cq_gen: Current CQ generation bit.
+ *
+ * Return:
+ * Returns 1 if empty, 0 if non-empty, or < 0 if an error occurs.
+ */
+static int dlb2_cq_empty(struct dlb2_dev *dev, u64 user_cq_va, u8 cq_gen)
+{
+ struct dlb2_dequeue_qe qe;
+
+ if (copy_from_user(&qe, (void __user *)user_cq_va, sizeof(qe)))
+ return -EFAULT;
+
+ return qe.cq_gen != cq_gen;
+}
+
+int dlb2_block_on_cq_interrupt(struct dlb2_dev *dev,
+ struct dlb2_domain *dom,
+ int port_id,
+ bool is_ldb,
+ u64 cq_va,
+ u8 cq_gen,
+ bool arm)
+{
+ struct dlb2_cq_intr *intr;
+ int ret = 0;
+
+ if (is_ldb && port_id >= DLB2_MAX_NUM_LDB_PORTS)
+ return -EINVAL;
+ if (!is_ldb && port_id >= DLB2_MAX_NUM_DIR_PORTS)
+ return -EINVAL;
+
+ if (is_ldb)
+ intr = &dev->intr.ldb_cq_intr[port_id];
+ else
+ intr = &dev->intr.dir_cq_intr[port_id];
+
+ if (!intr->configured || intr->domain_id != dom->id)
+ return -EINVAL;
+
+ /*
+ * This function requires that only one thread process the CQ at a time.
+ * Otherwise, the wake condition could become false in the time between
+ * the ISR calling wake_up_interruptible() and the thread checking its
+ * wake condition.
+ */
+ mutex_lock(&intr->mutex);
+
+ /* Return early if the port's interrupt is disabled */
+ if (READ_ONCE(intr->disabled)) {
+ mutex_unlock(&intr->mutex);
+ return -EACCES;
+ }
+
+ dev_dbg(dev->dlb2_device,
+ "Thread is blocking on %s port %d's interrupt\n",
+ (is_ldb) ? "LDB" : "DIR", port_id);
+
+ /* Don't block if the CQ is non-empty */
+ ret = dlb2_cq_empty(dev, cq_va, cq_gen);
+ if (ret != 1)
+ goto error;
+
+ if (arm) {
+ ret = dev->ops->arm_cq_interrupt(dev, dom->id, port_id, is_ldb);
+ if (ret)
+ goto error;
+ }
+
+ ret = wait_event_interruptible(intr->wq_head,
+ wake_condition(intr, dev, dom));
+
+ if (ret == 0 && READ_ONCE(intr->disabled))
+ ret = -EACCES;
+
+ WRITE_ONCE(intr->wake, false);
+
+ dev_dbg(dev->dlb2_device,
+ "Thread is unblocked from %s port %d's interrupt\n",
+ (is_ldb) ? "LDB" : "DIR", port_id);
+
+error:
+ mutex_unlock(&intr->mutex);
+
+ return ret;
+}
diff --git a/drivers/misc/dlb2/dlb2_intr.h b/drivers/misc/dlb2/dlb2_intr.h
new file mode 100644
index 000000000000..613179795d8f
--- /dev/null
+++ b/drivers/misc/dlb2/dlb2_intr.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ * Copyright(c) 2017-2020 Intel Corporation
+ */
+
+#ifndef __DLB2_INTR_H
+#define __DLB2_INTR_H
+
+#include <linux/pci.h>
+
+#include "dlb2_main.h"
+
+int dlb2_block_on_cq_interrupt(struct dlb2_dev *dev,
+ struct dlb2_domain *domain,
+ int port_id,
+ bool is_ldb,
+ u64 cq_va,
+ u8 cq_gen,
+ bool arm);
+
+enum dlb2_wake_reason {
+ WAKE_CQ_INTR,
+ WAKE_PORT_DISABLED,
+};
+
+void dlb2_wake_thread(struct dlb2_dev *dev,
+ struct dlb2_cq_intr *intr,
+ enum dlb2_wake_reason reason);
+
+#endif /* __DLB2_INTR_H */
diff --git a/drivers/misc/dlb2/dlb2_ioctl.c b/drivers/misc/dlb2/dlb2_ioctl.c
index 490c380670cc..010e67941cf9 100644
--- a/drivers/misc/dlb2/dlb2_ioctl.c
+++ b/drivers/misc/dlb2/dlb2_ioctl.c
@@ -7,6 +7,7 @@
#include <uapi/linux/dlb2_user.h>
#include "dlb2_file.h"
+#include "dlb2_intr.h"
#include "dlb2_ioctl.h"
#include "dlb2_main.h"
@@ -75,6 +76,10 @@ static int dlb2_domain_ioctl_enable_ldb_port(struct dlb2_dev *dev,
ret = dev->ops->enable_ldb_port(&dev->hw, domain->id, &arg, &response);
+ /* Allow threads to block on this port's CQ interrupt */
+ if (!ret)
+ WRITE_ONCE(dev->intr.ldb_cq_intr[arg.port_id].disabled, false);
+
mutex_unlock(&dev->resource_mutex);
BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
@@ -100,6 +105,10 @@ static int dlb2_domain_ioctl_enable_dir_port(struct dlb2_dev *dev,
ret = dev->ops->enable_dir_port(&dev->hw, domain->id, &arg, &response);
+ /* Allow threads to block on this port's CQ interrupt */
+ if (!ret)
+ WRITE_ONCE(dev->intr.dir_cq_intr[arg.port_id].disabled, false);
+
mutex_unlock(&dev->resource_mutex);
BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
@@ -125,6 +134,15 @@ static int dlb2_domain_ioctl_disable_ldb_port(struct dlb2_dev *dev,
ret = dev->ops->disable_ldb_port(&dev->hw, domain->id, &arg, &response);
+ /*
+ * Wake threads blocked on this port's CQ interrupt, and prevent
+ * subsequent attempts to block on it.
+ */
+ if (!ret)
+ dlb2_wake_thread(dev,
+ &dev->intr.ldb_cq_intr[arg.port_id],
+ WAKE_PORT_DISABLED);
+
mutex_unlock(&dev->resource_mutex);
BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
@@ -150,6 +168,15 @@ static int dlb2_domain_ioctl_disable_dir_port(struct dlb2_dev *dev,
ret = dev->ops->disable_dir_port(&dev->hw, domain->id, &arg, &response);
+ /*
+ * Wake threads blocked on this port's CQ interrupt, and prevent
+ * subsequent attempts to block on it.
+ */
+ if (!ret)
+ dlb2_wake_thread(dev,
+ &dev->intr.dir_cq_intr[arg.port_id],
+ WAKE_PORT_DISABLED);
+
mutex_unlock(&dev->resource_mutex);
BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
@@ -197,6 +224,13 @@ static int dlb2_domain_ioctl_create_ldb_port(struct dlb2_dev *dev,
if (ret)
goto unlock;
+ ret = dev->ops->enable_ldb_cq_interrupts(dev,
+ domain->id,
+ response.id,
+ arg.cq_depth_threshold);
+ if (ret)
+ goto unlock; /* Internal error, don't unwind port creation */
+
/* Fill out the per-port data structure */
dev->ldb_port[response.id].id = response.id;
dev->ldb_port[response.id].is_ldb = true;
@@ -255,6 +289,13 @@ static int dlb2_domain_ioctl_create_dir_port(struct dlb2_dev *dev,
if (ret)
goto unlock;
+ dev->ops->enable_dir_cq_interrupts(dev,
+ domain->id,
+ response.id,
+ arg.cq_depth_threshold);
+ if (ret)
+ goto unlock; /* Internal error, don't unwind port creation */
+
/* Fill out the per-port data structure */
dev->dir_port[response.id].id = response.id;
dev->dir_port[response.id].is_ldb = false;
@@ -280,6 +321,33 @@ static int dlb2_domain_ioctl_create_dir_port(struct dlb2_dev *dev,
return ret;
}
+static int dlb2_domain_ioctl_block_on_cq_interrupt(struct dlb2_dev *dev,
+ struct dlb2_domain *domain,
+ unsigned long user_arg)
+{
+ struct dlb2_block_on_cq_interrupt_args arg;
+ struct dlb2_cmd_response response = {0};
+ int ret = 0;
+
+ if (copy_from_user(&arg, (void __user *)user_arg, sizeof(arg)))
+ return -EFAULT;
+
+ ret = dlb2_block_on_cq_interrupt(dev,
+ domain,
+ arg.port_id,
+ arg.is_ldb,
+ arg.cq_va,
+ arg.cq_gen,
+ arg.arm);
+
+ BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
+
+ if (copy_to_user((void __user *)user_arg, &response, sizeof(response)))
+ return -EFAULT;
+
+ return ret;
+}
+
static int dlb2_create_port_fd(struct dlb2_dev *dev,
struct dlb2_domain *domain,
const char *prefix,
@@ -467,6 +535,8 @@ long dlb2_domain_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
return dlb2_domain_ioctl_disable_ldb_port(dev, dom, arg);
case DLB2_IOC_DISABLE_DIR_PORT:
return dlb2_domain_ioctl_disable_dir_port(dev, dom, arg);
+ case DLB2_IOC_BLOCK_ON_CQ_INTERRUPT:
+ return dlb2_domain_ioctl_block_on_cq_interrupt(dev, dom, arg);
default:
return -ENOTTY;
}
diff --git a/drivers/misc/dlb2/dlb2_main.c b/drivers/misc/dlb2/dlb2_main.c
index 3fe59e99b402..b542c2c081a5 100644
--- a/drivers/misc/dlb2/dlb2_main.c
+++ b/drivers/misc/dlb2/dlb2_main.c
@@ -174,13 +174,20 @@ static void dlb2_release_device_memory(struct dlb2_dev *dev)
static int __dlb2_free_domain(struct dlb2_dev *dev, struct dlb2_domain *domain)
{
- int ret = 0;
+ int i, ret = 0;
ret = dev->ops->reset_domain(&dev->hw, domain->id);
/* Unpin and free all memory pages associated with the domain */
dlb2_release_domain_memory(dev, domain->id);
+ for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++)
+ if (dev->intr.ldb_cq_intr[i].domain_id == domain->id)
+ dev->intr.ldb_cq_intr[i].configured = false;
+ for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++)
+ if (dev->intr.dir_cq_intr[i].domain_id == domain->id)
+ dev->intr.dir_cq_intr[i].configured = false;
+
if (ret) {
dev->domain_reset_failed = true;
dev_err(dev->dlb2_device,
@@ -430,6 +437,10 @@ static int dlb2_probe(struct pci_dev *pdev,
if (ret)
goto dlb2_reset_fail;
+ ret = dlb2_dev->ops->init_interrupts(dlb2_dev, pdev);
+ if (ret)
+ goto init_interrupts_fail;
+
ret = dlb2_resource_init(&dlb2_dev->hw);
if (ret)
goto resource_init_fail;
@@ -458,6 +469,8 @@ static int dlb2_probe(struct pci_dev *pdev,
init_driver_state_fail:
dlb2_resource_free(&dlb2_dev->hw);
resource_init_fail:
+ dlb2_dev->ops->free_interrupts(dlb2_dev, pdev);
+init_interrupts_fail:
dlb2_reset_fail:
wait_for_device_ready_fail:
dma_set_mask_fail:
@@ -497,6 +510,8 @@ static void dlb2_remove(struct pci_dev *pdev)
dlb2_resource_free(&dlb2_dev->hw);
+ dlb2_dev->ops->free_interrupts(dlb2_dev, pdev);
+
dlb2_release_device_memory(dlb2_dev);
dlb2_dev->ops->device_destroy(dlb2_dev, dlb2_class);
@@ -521,6 +536,9 @@ static void dlb2_reset_hardware_state(struct dlb2_dev *dev)
{
dlb2_reset_device(dev->pdev);
+ /* Reinitialize interrupt configuration */
+ dev->ops->reinit_interrupts(dev);
+
/* Reinitialize any other hardware state */
dev->ops->init_hardware(dev);
}
diff --git a/drivers/misc/dlb2/dlb2_main.h b/drivers/misc/dlb2/dlb2_main.h
index 92e182603445..db462209fa6a 100644
--- a/drivers/misc/dlb2/dlb2_main.h
+++ b/drivers/misc/dlb2/dlb2_main.h
@@ -54,6 +54,21 @@ struct dlb2_device_ops {
dev_t base,
const struct file_operations *fops);
void (*cdev_del)(struct dlb2_dev *dlb2_dev);
+ int (*init_interrupts)(struct dlb2_dev *dev, struct pci_dev *pdev);
+ int (*enable_ldb_cq_interrupts)(struct dlb2_dev *dev,
+ int domain_id,
+ int port_id,
+ u16 thresh);
+ int (*enable_dir_cq_interrupts)(struct dlb2_dev *dev,
+ int domain_id,
+ int port_id,
+ u16 thresh);
+ int (*arm_cq_interrupt)(struct dlb2_dev *dev,
+ int domain_id,
+ int port_id,
+ bool is_ldb);
+ void (*reinit_interrupts)(struct dlb2_dev *dev);
+ void (*free_interrupts)(struct dlb2_dev *dev, struct pci_dev *pdev);
void (*enable_pm)(struct dlb2_dev *dev);
int (*wait_for_device_ready)(struct dlb2_dev *dev,
struct pci_dev *pdev);
@@ -152,6 +167,33 @@ struct dlb2_domain {
u8 id;
};
+struct dlb2_cq_intr {
+ wait_queue_head_t wq_head;
+ /*
+ * The CQ interrupt mutex guarantees one thread is blocking on a CQ's
+ * interrupt at a time.
+ */
+ struct mutex mutex;
+ u8 wake;
+ u8 configured;
+ u8 domain_id;
+ /*
+ * disabled is true if the port is disabled. In that
+ * case, the driver doesn't allow applications to block on the
+ * port's interrupt.
+ */
+ u8 disabled;
+} ____cacheline_aligned;
+
+struct dlb2_intr {
+ struct dlb2_cq_intr ldb_cq_intr[DLB2_MAX_NUM_LDB_PORTS];
+ struct dlb2_cq_intr dir_cq_intr[DLB2_MAX_NUM_DIR_PORTS];
+ u8 isr_registered[DLB2_PF_NUM_CQ_INTERRUPT_VECTORS];
+ int num_vectors;
+ int base_vector;
+ int mode;
+};
+
struct dlb2_dev {
struct pci_dev *pdev;
struct dlb2_hw hw;
@@ -167,6 +209,7 @@ struct dlb2_dev {
* device file mappings.
*/
struct inode *inode;
+ struct dlb2_intr intr;
/*
* The resource mutex serializes access to driver data structures and
* hardware registers.
diff --git a/drivers/misc/dlb2/dlb2_pf_ops.c b/drivers/misc/dlb2/dlb2_pf_ops.c
index 8b1aac196073..23a1e9ba0226 100644
--- a/drivers/misc/dlb2/dlb2_pf_ops.c
+++ b/drivers/misc/dlb2/dlb2_pf_ops.c
@@ -4,6 +4,7 @@
#include <linux/delay.h>
#include <linux/pm_runtime.h>
+#include "dlb2_intr.h"
#include "dlb2_main.h"
#include "dlb2_regs.h"
#include "dlb2_resource.h"
@@ -77,6 +78,210 @@ dlb2_pf_map_pci_bar_space(struct dlb2_dev *dlb2_dev,
return 0;
}
+/**********************************/
+/****** Interrupt management ******/
+/**********************************/
+
+static irqreturn_t
+dlb2_compressed_cq_intr_handler(int irq, void *hdlr_ptr)
+{
+ struct dlb2_dev *dev = (struct dlb2_dev *)hdlr_ptr;
+ u32 ldb_cq_interrupts[DLB2_MAX_NUM_LDB_PORTS / 32];
+ u32 dir_cq_interrupts[DLB2_MAX_NUM_DIR_PORTS / 32];
+ int i;
+
+ dlb2_read_compressed_cq_intr_status(&dev->hw,
+ ldb_cq_interrupts,
+ dir_cq_interrupts);
+
+ dlb2_ack_compressed_cq_intr(&dev->hw,
+ ldb_cq_interrupts,
+ dir_cq_interrupts);
+
+ for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+ if (!(ldb_cq_interrupts[i / 32] & (1 << (i % 32))))
+ continue;
+
+ dlb2_wake_thread(dev, &dev->intr.ldb_cq_intr[i], WAKE_CQ_INTR);
+ }
+
+ for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++) {
+ if (!(dir_cq_interrupts[i / 32] & (1 << (i % 32))))
+ continue;
+
+ dlb2_wake_thread(dev, &dev->intr.dir_cq_intr[i], WAKE_CQ_INTR);
+ }
+
+ return IRQ_HANDLED;
+}
+
+static int
+dlb2_init_compressed_mode_interrupts(struct dlb2_dev *dev,
+ struct pci_dev *pdev)
+{
+ int ret, irq;
+
+ irq = pci_irq_vector(pdev, DLB2_PF_COMPRESSED_MODE_CQ_VECTOR_ID);
+
+ ret = devm_request_irq(&pdev->dev,
+ irq,
+ dlb2_compressed_cq_intr_handler,
+ 0,
+ "dlb2_compressed_cq",
+ dev);
+ if (ret)
+ return ret;
+
+ dev->intr.isr_registered[DLB2_PF_COMPRESSED_MODE_CQ_VECTOR_ID] = true;
+
+ dev->intr.mode = DLB2_MSIX_MODE_COMPRESSED;
+
+ dlb2_set_msix_mode(&dev->hw, DLB2_MSIX_MODE_COMPRESSED);
+
+ return 0;
+}
+
+static void
+dlb2_pf_free_interrupts(struct dlb2_dev *dev,
+ struct pci_dev *pdev)
+{
+ int i;
+
+ for (i = 0; i < dev->intr.num_vectors; i++) {
+ if (dev->intr.isr_registered[i])
+ devm_free_irq(&pdev->dev, pci_irq_vector(pdev, i), dev);
+ }
+
+ pci_free_irq_vectors(pdev);
+}
+
+static int
+dlb2_pf_init_interrupts(struct dlb2_dev *dev, struct pci_dev *pdev)
+{
+ int ret, i;
+
+ /*
+ * DLB supports two modes for CQ interrupts:
+ * - "compressed mode": all CQ interrupts are packed into a single
+ * vector. The ISR reads six interrupt status registers to
+ * determine the source(s).
+ * - "packed mode" (unused): the hardware supports up to 64 vectors.
+ */
+
+ ret = pci_alloc_irq_vectors(pdev,
+ DLB2_PF_NUM_COMPRESSED_MODE_VECTORS,
+ DLB2_PF_NUM_COMPRESSED_MODE_VECTORS,
+ PCI_IRQ_MSIX);
+ if (ret < 0)
+ return ret;
+
+ dev->intr.num_vectors = ret;
+ dev->intr.base_vector = pci_irq_vector(pdev, 0);
+
+ ret = dlb2_init_compressed_mode_interrupts(dev, pdev);
+ if (ret) {
+ dlb2_pf_free_interrupts(dev, pdev);
+ return ret;
+ }
+
+ /*
+ * Initialize per-CQ interrupt structures, such as wait queues that
+ * threads will wait on until the CQ's interrupt fires.
+ */
+ for (i = 0; i < DLB2_MAX_NUM_LDB_PORTS; i++) {
+ init_waitqueue_head(&dev->intr.ldb_cq_intr[i].wq_head);
+ mutex_init(&dev->intr.ldb_cq_intr[i].mutex);
+ }
+
+ for (i = 0; i < DLB2_MAX_NUM_DIR_PORTS; i++) {
+ init_waitqueue_head(&dev->intr.dir_cq_intr[i].wq_head);
+ mutex_init(&dev->intr.dir_cq_intr[i].mutex);
+ }
+
+ return 0;
+}
+
+/*
+ * If the device is reset during use, its interrupt registers need to be
+ * reinitialized.
+ */
+static void
+dlb2_pf_reinit_interrupts(struct dlb2_dev *dev)
+{
+ dlb2_set_msix_mode(&dev->hw, DLB2_MSIX_MODE_COMPRESSED);
+}
+
+static int
+dlb2_pf_enable_ldb_cq_interrupts(struct dlb2_dev *dev,
+ int domain_id,
+ int id,
+ u16 thresh)
+{
+ int mode, vec;
+
+ if (dev->intr.mode == DLB2_MSIX_MODE_COMPRESSED) {
+ mode = DLB2_CQ_ISR_MODE_MSIX;
+ vec = 0;
+ } else {
+ mode = DLB2_CQ_ISR_MODE_MSIX;
+ vec = id % 64;
+ }
+
+ dev->intr.ldb_cq_intr[id].disabled = false;
+ dev->intr.ldb_cq_intr[id].configured = true;
+ dev->intr.ldb_cq_intr[id].domain_id = domain_id;
+
+ return dlb2_configure_ldb_cq_interrupt(&dev->hw, id, vec,
+ mode, 0, 0, thresh);
+}
+
+static int
+dlb2_pf_enable_dir_cq_interrupts(struct dlb2_dev *dev,
+ int domain_id,
+ int id,
+ u16 thresh)
+{
+ int mode, vec;
+
+ if (dev->intr.mode == DLB2_MSIX_MODE_COMPRESSED) {
+ mode = DLB2_CQ_ISR_MODE_MSIX;
+ vec = 0;
+ } else {
+ mode = DLB2_CQ_ISR_MODE_MSIX;
+ vec = id % 64;
+ }
+
+ dev->intr.dir_cq_intr[id].disabled = false;
+ dev->intr.dir_cq_intr[id].configured = true;
+ dev->intr.dir_cq_intr[id].domain_id = domain_id;
+
+ return dlb2_configure_dir_cq_interrupt(&dev->hw, id, vec,
+ mode, 0, 0, thresh);
+}
+
+static int
+dlb2_pf_arm_cq_interrupt(struct dlb2_dev *dev,
+ int domain_id,
+ int port_id,
+ bool is_ldb)
+{
+ int ret;
+
+ if (is_ldb)
+ ret = dev->ops->ldb_port_owned_by_domain(&dev->hw,
+ domain_id,
+ port_id);
+ else
+ ret = dev->ops->dir_port_owned_by_domain(&dev->hw,
+ domain_id,
+ port_id);
+
+ if (ret != 1)
+ return -EINVAL;
+
+ return dlb2_arm_cq_interrupt(&dev->hw, port_id, is_ldb, false, 0);
+}
+
/*******************************/
/****** Driver management ******/
/*******************************/
@@ -414,6 +619,12 @@ struct dlb2_device_ops dlb2_pf_ops = {
.device_destroy = dlb2_pf_device_destroy,
.cdev_add = dlb2_pf_cdev_add,
.cdev_del = dlb2_pf_cdev_del,
+ .init_interrupts = dlb2_pf_init_interrupts,
+ .enable_ldb_cq_interrupts = dlb2_pf_enable_ldb_cq_interrupts,
+ .enable_dir_cq_interrupts = dlb2_pf_enable_dir_cq_interrupts,
+ .arm_cq_interrupt = dlb2_pf_arm_cq_interrupt,
+ .reinit_interrupts = dlb2_pf_reinit_interrupts,
+ .free_interrupts = dlb2_pf_free_interrupts,
.enable_pm = dlb2_pf_enable_pm,
.wait_for_device_ready = dlb2_pf_wait_for_device_ready,
.create_sched_domain = dlb2_pf_create_sched_domain,
diff --git a/drivers/misc/dlb2/dlb2_resource.c b/drivers/misc/dlb2/dlb2_resource.c
index 665841ca7919..bfe20864515d 100644
--- a/drivers/misc/dlb2/dlb2_resource.c
+++ b/drivers/misc/dlb2/dlb2_resource.c
@@ -237,6 +237,41 @@ static struct dlb2_hw_domain *dlb2_get_domain_from_id(struct dlb2_hw *hw,
return NULL;
}
+static struct dlb2_ldb_port *dlb2_get_ldb_port_from_id(struct dlb2_hw *hw,
+ u32 id,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_function_resources *rsrcs;
+ struct dlb2_hw_domain *domain;
+ struct dlb2_ldb_port *port;
+ int i;
+
+ if (id >= DLB2_MAX_NUM_LDB_PORTS)
+ return NULL;
+
+ rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+ if (!vdev_req)
+ return &hw->rsrcs.ldb_ports[id];
+
+ DLB2_FUNC_LIST_FOR(rsrcs->used_domains, domain) {
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ DLB2_DOM_LIST_FOR(domain->used_ldb_ports[i], port)
+ if (port->id.virt_id == id)
+ return port;
+ }
+ }
+
+ for (i = 0; i < DLB2_NUM_COS_DOMAINS; i++) {
+ DLB2_FUNC_LIST_FOR(rsrcs->avail_ldb_ports[i], port)
+ if (port->id.virt_id == id)
+ return port;
+ }
+
+ return NULL;
+}
+
static struct dlb2_ldb_port *
dlb2_get_domain_used_ldb_port(u32 id,
bool vdev_req,
@@ -289,6 +324,36 @@ dlb2_get_domain_ldb_port(u32 id,
return NULL;
}
+static struct dlb2_dir_pq_pair *dlb2_get_dir_pq_from_id(struct dlb2_hw *hw,
+ u32 id,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb2_function_resources *rsrcs;
+ struct dlb2_dir_pq_pair *port;
+ struct dlb2_hw_domain *domain;
+
+ if (id >= DLB2_MAX_NUM_DIR_PORTS)
+ return NULL;
+
+ rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+ if (!vdev_req)
+ return &hw->rsrcs.dir_pq_pairs[id];
+
+ DLB2_FUNC_LIST_FOR(rsrcs->used_domains, domain) {
+ DLB2_DOM_LIST_FOR(domain->used_dir_pq_pairs, port)
+ if (port->id.virt_id == id)
+ return port;
+ }
+
+ DLB2_FUNC_LIST_FOR(rsrcs->avail_dir_pq_pairs, port)
+ if (port->id.virt_id == id)
+ return port;
+
+ return NULL;
+}
+
static struct dlb2_dir_pq_pair *
dlb2_get_domain_used_dir_pq(u32 id,
bool vdev_req,
@@ -4685,6 +4750,247 @@ int dlb2_hw_disable_dir_port(struct dlb2_hw *hw,
return 0;
}
+void dlb2_set_msix_mode(struct dlb2_hw *hw, int mode)
+{
+ union dlb2_sys_msix_mode r0 = { {0} };
+
+ r0.field.mode = mode;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_MSIX_MODE, r0.val);
+}
+
+int dlb2_configure_ldb_cq_interrupt(struct dlb2_hw *hw,
+ int port_id,
+ int vector,
+ int mode,
+ unsigned int vf,
+ unsigned int owner_vf,
+ u16 threshold)
+{
+ union dlb2_chp_ldb_cq_int_depth_thrsh r0 = { {0} };
+ union dlb2_chp_ldb_cq_int_enb r1 = { {0} };
+ union dlb2_sys_ldb_cq_isr r2 = { {0} };
+ struct dlb2_ldb_port *port;
+ bool vdev_req;
+
+ vdev_req = (mode == DLB2_CQ_ISR_MODE_MSI ||
+ mode == DLB2_CQ_ISR_MODE_ADI);
+
+ port = dlb2_get_ldb_port_from_id(hw, port_id, vdev_req, vf);
+ if (!port) {
+ DLB2_HW_ERR(hw,
+ "[%s()]: Internal error: failed to enable LDB CQ int\n\tport_id: %u, vdev_req: %u, vdev: %u\n",
+ __func__, port_id, vdev_req, vf);
+ return -EINVAL;
+ }
+
+ /* Trigger the interrupt when threshold or more QEs arrive in the CQ */
+ r0.field.depth_threshold = threshold - 1;
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_LDB_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+ r0.val);
+
+ r1.field.en_depth = 1;
+
+ DLB2_CSR_WR(hw, DLB2_CHP_LDB_CQ_INT_ENB(port->id.phys_id), r1.val);
+
+ r2.field.vector = vector;
+ r2.field.vf = owner_vf;
+ r2.field.en_code = mode;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_LDB_CQ_ISR(port->id.phys_id), r2.val);
+
+ return 0;
+}
+
+int dlb2_configure_dir_cq_interrupt(struct dlb2_hw *hw,
+ int port_id,
+ int vector,
+ int mode,
+ unsigned int vf,
+ unsigned int owner_vf,
+ u16 threshold)
+{
+ union dlb2_chp_dir_cq_int_depth_thrsh r0 = { {0} };
+ union dlb2_chp_dir_cq_int_enb r1 = { {0} };
+ union dlb2_sys_dir_cq_isr r2 = { {0} };
+ struct dlb2_dir_pq_pair *port;
+ bool vdev_req;
+
+ vdev_req = (mode == DLB2_CQ_ISR_MODE_MSI ||
+ mode == DLB2_CQ_ISR_MODE_ADI);
+
+ port = dlb2_get_dir_pq_from_id(hw, port_id, vdev_req, vf);
+ if (!port) {
+ DLB2_HW_ERR(hw,
+ "[%s()]: Internal error: failed to enable DIR CQ int\n\tport_id: %u, vdev_req: %u, vdev: %u\n",
+ __func__, port_id, vdev_req, vf);
+ return -EINVAL;
+ }
+
+ /* Trigger the interrupt when threshold or more QEs arrive in the CQ */
+ r0.field.depth_threshold = threshold - 1;
+
+ DLB2_CSR_WR(hw,
+ DLB2_CHP_DIR_CQ_INT_DEPTH_THRSH(port->id.phys_id),
+ r0.val);
+
+ r1.field.en_depth = 1;
+
+ DLB2_CSR_WR(hw, DLB2_CHP_DIR_CQ_INT_ENB(port->id.phys_id), r1.val);
+
+ r2.field.vector = vector;
+ r2.field.vf = owner_vf;
+ r2.field.en_code = mode;
+
+ DLB2_CSR_WR(hw, DLB2_SYS_DIR_CQ_ISR(port->id.phys_id), r2.val);
+
+ return 0;
+}
+
+int dlb2_arm_cq_interrupt(struct dlb2_hw *hw,
+ int port_id,
+ bool is_ldb,
+ bool vdev_req,
+ unsigned int vdev_id)
+{
+ u32 val;
+ u32 reg;
+
+ if (vdev_req && is_ldb) {
+ struct dlb2_ldb_port *ldb_port;
+
+ ldb_port = dlb2_get_ldb_port_from_id(hw, port_id,
+ true, vdev_id);
+
+ if (!ldb_port || !ldb_port->configured)
+ return -EINVAL;
+
+ port_id = ldb_port->id.phys_id;
+ } else if (vdev_req && !is_ldb) {
+ struct dlb2_dir_pq_pair *dir_port;
+
+ dir_port = dlb2_get_dir_pq_from_id(hw, port_id, true, vdev_id);
+
+ if (!dir_port || !dir_port->port_configured)
+ return -EINVAL;
+
+ port_id = dir_port->id.phys_id;
+ }
+
+ val = 1 << (port_id % 32);
+
+ if (is_ldb && port_id < 32)
+ reg = DLB2_CHP_LDB_CQ_INTR_ARMED0;
+ else if (is_ldb && port_id < 64)
+ reg = DLB2_CHP_LDB_CQ_INTR_ARMED1;
+ else if (!is_ldb && port_id < 32)
+ reg = DLB2_CHP_DIR_CQ_INTR_ARMED0;
+ else
+ reg = DLB2_CHP_DIR_CQ_INTR_ARMED1;
+
+ DLB2_CSR_WR(hw, reg, val);
+
+ dlb2_flush_csr(hw);
+
+ return 0;
+}
+
+void dlb2_read_compressed_cq_intr_status(struct dlb2_hw *hw,
+ u32 *ldb_interrupts,
+ u32 *dir_interrupts)
+{
+ /* Read every CQ's interrupt status */
+
+ ldb_interrupts[0] = DLB2_CSR_RD(hw,
+ DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS);
+ ldb_interrupts[1] = DLB2_CSR_RD(hw,
+ DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS);
+
+ dir_interrupts[0] = DLB2_CSR_RD(hw,
+ DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS);
+ dir_interrupts[1] = DLB2_CSR_RD(hw,
+ DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS);
+}
+
+static void dlb2_ack_msix_interrupt(struct dlb2_hw *hw, int vector)
+{
+ union dlb2_sys_msix_ack r0 = { {0} };
+
+ switch (vector) {
+ case 0:
+ r0.field.msix_0_ack = 1;
+ break;
+ case 1:
+ r0.field.msix_1_ack = 1;
+ /*
+ * CSSY-1650
+ * workaround h/w bug for lost MSI-X interrupts
+ *
+ * The recommended workaround for acknowledging
+ * vector 1 interrupts is :
+ * 1: set MSI-X mask
+ * 2: set MSIX_PASSTHROUGH
+ * 3: clear MSIX_ACK
+ * 4: clear MSIX_PASSTHROUGH
+ * 5: clear MSI-X mask
+ *
+ * The MSIX-ACK (step 3) is cleared for all vectors
+ * below. We handle steps 1 & 2 for vector 1 here.
+ *
+ * The bitfields for MSIX_ACK and MSIX_PASSTHRU are
+ * defined the same, so we just use the MSIX_ACK
+ * value when writing to PASSTHRU.
+ */
+
+ /* set MSI-X mask and passthrough for vector 1 */
+ DLB2_FUNC_WR(hw, DLB2_MSIX_MEM_VECTOR_CTRL(1), 1);
+ DLB2_CSR_WR(hw, DLB2_SYS_MSIX_PASSTHRU, r0.val);
+ break;
+ }
+
+ /* clear MSIX_ACK (write one to clear) */
+ DLB2_CSR_WR(hw, DLB2_SYS_MSIX_ACK, r0.val);
+
+ if (vector == 1) {
+ /*
+ * finish up steps 4 & 5 of the workaround -
+ * clear pasthrough and mask
+ */
+ DLB2_CSR_WR(hw, DLB2_SYS_MSIX_PASSTHRU, 0);
+ DLB2_FUNC_WR(hw, DLB2_MSIX_MEM_VECTOR_CTRL(1), 0);
+ }
+
+ dlb2_flush_csr(hw);
+}
+
+void dlb2_ack_compressed_cq_intr(struct dlb2_hw *hw,
+ u32 *ldb_interrupts,
+ u32 *dir_interrupts)
+{
+ /* Write back the status regs to ack the interrupts */
+ if (ldb_interrupts[0])
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_LDB_CQ_31_0_OCC_INT_STS,
+ ldb_interrupts[0]);
+ if (ldb_interrupts[1])
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_LDB_CQ_63_32_OCC_INT_STS,
+ ldb_interrupts[1]);
+
+ if (dir_interrupts[0])
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_DIR_CQ_31_0_OCC_INT_STS,
+ dir_interrupts[0]);
+ if (dir_interrupts[1])
+ DLB2_CSR_WR(hw,
+ DLB2_SYS_DIR_CQ_63_32_OCC_INT_STS,
+ dir_interrupts[1]);
+
+ dlb2_ack_msix_interrupt(hw, DLB2_PF_COMPRESSED_MODE_CQ_VECTOR_ID);
+}
+
static u32 dlb2_ldb_cq_inflight_count(struct dlb2_hw *hw,
struct dlb2_ldb_port *port)
{
diff --git a/drivers/misc/dlb2/dlb2_resource.h b/drivers/misc/dlb2/dlb2_resource.h
index a78a8664eabc..5fd7a30a1c1b 100644
--- a/drivers/misc/dlb2/dlb2_resource.h
+++ b/drivers/misc/dlb2/dlb2_resource.h
@@ -445,6 +445,134 @@ int dlb2_hw_disable_dir_port(struct dlb2_hw *hw,
unsigned int vdev_id);
/**
+ * dlb2_configure_ldb_cq_interrupt() - configure load-balanced CQ for
+ * interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: load-balanced port ID.
+ * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
+ * else a value up to 64.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
+ * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
+ * virtual port ID to a physical port ID. Ignored if mode is not MSI.
+ * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
+ * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
+ * greater than 0.
+ *
+ * This function configures the DLB registers for load-balanced CQ's
+ * interrupts. This doesn't enable the CQ's interrupt; that can be done with
+ * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid.
+ */
+int dlb2_configure_ldb_cq_interrupt(struct dlb2_hw *hw,
+ int port_id,
+ int vector,
+ int mode,
+ unsigned int vf,
+ unsigned int owner_vf,
+ u16 threshold);
+
+/**
+ * dlb2_configure_dir_cq_interrupt() - configure directed CQ for interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: load-balanced port ID.
+ * @vector: interrupt vector ID. Should be 0 for MSI or compressed MSI-X mode,
+ * else a value up to 64.
+ * @mode: interrupt type (DLB2_CQ_ISR_MODE_MSI or DLB2_CQ_ISR_MODE_MSIX)
+ * @vf: If the port is VF-owned, the VF's ID. This is used for translating the
+ * virtual port ID to a physical port ID. Ignored if mode is not MSI.
+ * @owner_vf: the VF to route the interrupt to. Ignore if mode is not MSI.
+ * @threshold: the minimum CQ depth at which the interrupt can fire. Must be
+ * greater than 0.
+ *
+ * This function configures the DLB registers for directed CQ's interrupts.
+ * This doesn't enable the CQ's interrupt; that can be done with
+ * dlb2_arm_cq_interrupt() or through an interrupt arm QE.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - The port ID is invalid.
+ */
+int dlb2_configure_dir_cq_interrupt(struct dlb2_hw *hw,
+ int port_id,
+ int vector,
+ int mode,
+ unsigned int vf,
+ unsigned int owner_vf,
+ u16 threshold);
+
+/**
+ * dlb2_set_msix_mode() - enable certain hardware alarm interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @mode: MSI-X mode (DLB2_MSIX_MODE_PACKED or DLB2_MSIX_MODE_COMPRESSED)
+ *
+ * This function configures the hardware to use either packed or compressed
+ * mode. This function should not be called if using MSI interrupts.
+ */
+void dlb2_set_msix_mode(struct dlb2_hw *hw, int mode);
+
+/**
+ * dlb2_arm_cq_interrupt() - arm a CQ's interrupt
+ * @hw: dlb2_hw handle for a particular device.
+ * @port_id: port ID
+ * @is_ldb: true for load-balanced port, false for a directed port
+ * @vdev_request: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_request is true, this contains the vdev's ID.
+ *
+ * This function arms the CQ's interrupt. The CQ must be configured prior to
+ * calling this function.
+ *
+ * The function does no parameter validation; that is the caller's
+ * responsibility.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return: returns 0 upon success, <0 otherwise.
+ *
+ * EINVAL - Invalid port ID.
+ */
+int dlb2_arm_cq_interrupt(struct dlb2_hw *hw,
+ int port_id,
+ bool is_ldb,
+ bool vdev_request,
+ unsigned int vdev_id);
+
+/**
+ * dlb2_read_compressed_cq_intr_status() - read compressed CQ interrupt status
+ * @hw: dlb2_hw handle for a particular device.
+ * @ldb_interrupts: 2-entry array of u32 bitmaps
+ * @dir_interrupts: 4-entry array of u32 bitmaps
+ *
+ * This function can be called from a compressed CQ interrupt handler to
+ * determine which CQ interrupts have fired. The caller should take appropriate
+ * (such as waking threads blocked on a CQ's interrupt) then ack the interrupts
+ * with dlb2_ack_compressed_cq_intr().
+ */
+void dlb2_read_compressed_cq_intr_status(struct dlb2_hw *hw,
+ u32 *ldb_interrupts,
+ u32 *dir_interrupts);
+
+/**
+ * dlb2_ack_compressed_cq_intr_status() - ack compressed CQ interrupts
+ * @hw: dlb2_hw handle for a particular device.
+ * @ldb_interrupts: 2-entry array of u32 bitmaps
+ * @dir_interrupts: 4-entry array of u32 bitmaps
+ *
+ * This function ACKs compressed CQ interrupts. Its arguments should be the
+ * same ones passed to dlb2_read_compressed_cq_intr_status().
+ */
+void dlb2_ack_compressed_cq_intr(struct dlb2_hw *hw,
+ u32 *ldb_interrupts,
+ u32 *dir_interrupts);
+
+/**
* dlb2_reset_domain() - reset a scheduling domain
* @hw: dlb2_hw handle for a particular device.
* @domain_id: domain ID.
diff --git a/include/uapi/linux/dlb2_user.h b/include/uapi/linux/dlb2_user.h
index 00e9833bae1b..9edeff826e15 100644
--- a/include/uapi/linux/dlb2_user.h
+++ b/include/uapi/linux/dlb2_user.h
@@ -609,6 +609,38 @@ struct dlb2_disable_dir_port_args {
__u32 padding0;
};
+/*
+ * DLB2_DOMAIN_CMD_BLOCK_ON_CQ_INTERRUPT: Block on a CQ interrupt until a QE
+ * arrives for the specified port. If a QE is already present, the ioctl
+ * will immediately return.
+ *
+ * Note: Only one thread can block on a CQ's interrupt at a time. Doing
+ * otherwise can result in hung threads.
+ *
+ * Input parameters:
+ * - port_id: Port ID.
+ * - is_ldb: True if the port is load-balanced, false otherwise.
+ * - arm: Tell the driver to arm the interrupt.
+ * - cq_gen: Current CQ generation bit.
+ * - padding0: Reserved for future use.
+ * - cq_va: VA of the CQ entry where the next QE will be placed.
+ *
+ * Output parameters:
+ * - response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ */
+struct dlb2_block_on_cq_interrupt_args {
+ /* Output parameters */
+ struct dlb2_cmd_response response;
+ /* Input parameters */
+ __u32 port_id;
+ __u8 is_ldb;
+ __u8 arm;
+ __u8 cq_gen;
+ __u8 padding0;
+ __u64 cq_va;
+};
+
enum dlb2_domain_user_interface_commands {
DLB2_DOMAIN_CMD_CREATE_LDB_QUEUE,
DLB2_DOMAIN_CMD_CREATE_DIR_QUEUE,
@@ -628,6 +660,7 @@ enum dlb2_domain_user_interface_commands {
DLB2_DOMAIN_CMD_ENABLE_DIR_PORT,
DLB2_DOMAIN_CMD_DISABLE_LDB_PORT,
DLB2_DOMAIN_CMD_DISABLE_DIR_PORT,
+ DLB2_DOMAIN_CMD_BLOCK_ON_CQ_INTERRUPT,
/* NUM_DLB2_DOMAIN_CMD must be last */
NUM_DLB2_DOMAIN_CMD,
@@ -734,5 +767,9 @@ enum dlb2_domain_user_interface_commands {
_IOWR(DLB2_IOC_MAGIC, \
DLB2_DOMAIN_CMD_DISABLE_DIR_PORT, \
struct dlb2_disable_dir_port_args)
+#define DLB2_IOC_BLOCK_ON_CQ_INTERRUPT \
+ _IOWR(DLB2_IOC_MAGIC, \
+ DLB2_DOMAIN_CMD_BLOCK_ON_CQ_INTERRUPT, \
+ struct dlb2_block_on_cq_interrupt_args)
#endif /* __DLB2_USER_H */
--
2.13.6
On Tue, Aug 11, 2020 at 11:27:14AM -0500, Gage Eads wrote:
> diff --git a/drivers/misc/dlb2/dlb2_main.c b/drivers/misc/dlb2/dlb2_main.c
> new file mode 100644
> index 000000000000..ffd6df788e2e
> --- /dev/null
> +++ b/drivers/misc/dlb2/dlb2_main.c
> @@ -0,0 +1,208 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/* Copyright(c) 2018-2020 Intel Corporation */
> +
> +#include <linux/aer.h>
> +#include <linux/cdev.h>
> +#include <linux/delay.h>
> +#include <linux/fs.h>
> +#include <linux/init.h>
> +#include <linux/list.h>
> +#include <linux/module.h>
> +#include <linux/pci.h>
> +#include <linux/uaccess.h>
> +
> +#include "dlb2_main.h"
> +
> +static const char
> +dlb2_driver_copyright[] = "Copyright(c) 2018-2020 Intel Corporation";
Why do you have this static string that you never use?
> +
> +MODULE_LICENSE("GPL v2");
> +MODULE_AUTHOR("Copyright(c) 2018-2020 Intel Corporation");
> +MODULE_DESCRIPTION("Intel(R) Dynamic Load Balancer 2.0 Driver");
> +
> +/* The driver lock protects data structures that used by multiple devices. */
> +static DEFINE_MUTEX(dlb2_driver_lock);
> +static struct list_head dlb2_dev_list = LIST_HEAD_INIT(dlb2_dev_list);
A static list of devices that you constantly walk? Why? This isn't the
1990's anymore, that should not be needed anywhere at all.
> +
> +static struct class *dlb2_class;
> +static dev_t dlb2_dev_number_base;
static dev_t my_major; perhaps?
Why have "dlb2_" as a prefix for static variables?
> +
> +/*****************************/
> +/****** Devfs callbacks ******/
there is nothing in the kernel called "devfs", and has not been for 16+
years I think. Where did you get that term from???
> +/*****************************/
> +
> +static int dlb2_open(struct inode *i, struct file *f)
> +{
> + return 0;
> +}
> +
> +static int dlb2_close(struct inode *i, struct file *f)
> +{
> + return 0;
If you do not need an open/close function, do not include it.
> +}
> +
> +static const struct file_operations dlb2_fops = {
> + .owner = THIS_MODULE,
> + .open = dlb2_open,
> + .release = dlb2_close,
> +};
You never use this structure in this patch :(
> +
> +/**********************************/
> +/****** PCI driver callbacks ******/
> +/**********************************/
> +
> +static DEFINE_IDA(dlb2_ids);
This is not a pci driver callback :)
Why have comments if they instantly are wrong? :(
> +
> +static int dlb2_alloc_id(void)
> +{
> + return ida_alloc_max(&dlb2_ids, DLB2_MAX_NUM_DEVICES - 1, GFP_KERNEL);
> +}
> +
> +static void dlb2_free_id(int id)
> +{
> + ida_free(&dlb2_ids, id);
No locking needed for accessing the ida?
> +}
> +
> +static int dlb2_probe(struct pci_dev *pdev,
> + const struct pci_device_id *pdev_id)
> +{
> + struct dlb2_dev *dlb2_dev;
> + int ret;
> +
> + dlb2_dev = devm_kzalloc(&pdev->dev, sizeof(*dlb2_dev), GFP_KERNEL);
Why are you calling devm_kzalloc() if you manually call devm_kfree() for
this chunk of memory? Either rely on the devm api or don't use it at
all.
> + if (!dlb2_dev)
> + return -ENOMEM;
> +
> + pci_set_drvdata(pdev, dlb2_dev);
> +
> + dlb2_dev->pdev = pdev;
> +
> + dlb2_dev->id = dlb2_alloc_id();
> + if (dlb2_dev->id < 0) {
> + dev_err(&pdev->dev, "probe: device ID allocation failed\n");
> +
> + ret = dlb2_dev->id;
> + goto alloc_id_fail;
> + }
> +
> + ret = pci_enable_device(pdev);
> + if (ret != 0) {
> + dev_err(&pdev->dev, "pci_enable_device() returned %d\n", ret);
> +
> + goto pci_enable_device_fail;
> + }
> +
> + ret = pci_request_regions(pdev, dlb2_driver_name);
> + if (ret != 0) {
> + dev_err(&pdev->dev,
> + "pci_request_regions(): returned %d\n", ret);
> +
> + goto pci_request_regions_fail;
> + }
> +
> + pci_set_master(pdev);
> +
> + if (pci_enable_pcie_error_reporting(pdev))
> + dev_info(&pdev->dev, "[%s()] Failed to enable AER\n", __func__);
> +
> + mutex_lock(&dlb2_driver_lock);
> + list_add(&dlb2_dev->list, &dlb2_dev_list);
> + mutex_unlock(&dlb2_driver_lock);
> +
> + return 0;
> +
> +pci_request_regions_fail:
> + pci_disable_device(pdev);
> +pci_enable_device_fail:
> + dlb2_free_id(dlb2_dev->id);
> +alloc_id_fail:
> + devm_kfree(&pdev->dev, dlb2_dev);
> + return ret;
> +}
> +
> +static void dlb2_remove(struct pci_dev *pdev)
> +{
> + struct dlb2_dev *dlb2_dev;
> +
> + /* Undo all the dlb2_probe() operations */
> + dlb2_dev = pci_get_drvdata(pdev);
> +
> + mutex_lock(&dlb2_driver_lock);
> + list_del(&dlb2_dev->list);
> + mutex_unlock(&dlb2_driver_lock);
> +
> + pci_disable_pcie_error_reporting(pdev);
> +
> + pci_release_regions(pdev);
> +
> + pci_disable_device(pdev);
> +
> + dlb2_free_id(dlb2_dev->id);
> +
> + devm_kfree(&pdev->dev, dlb2_dev);
> +}
> +
> +static struct pci_device_id dlb2_id_table[] = {
> + { PCI_DEVICE_DATA(INTEL, DLB2_PF, NULL) },
> + { 0 }
> +};
> +MODULE_DEVICE_TABLE(pci, dlb2_id_table);
> +
> +static struct pci_driver dlb2_pci_driver = {
> + .name = (char *)dlb2_driver_name,
Why is this cast needed? That's a hint that something is wrong...
> + .id_table = dlb2_id_table,
> + .probe = dlb2_probe,
> + .remove = dlb2_remove,
> +};
> +
> +static int __init dlb2_init_module(void)
> +{
> + int err;
> +
> + dlb2_class = class_create(THIS_MODULE, dlb2_driver_name);
> +
> + if (IS_ERR(dlb2_class)) {
blank line was not needed.
> + pr_err("%s: class_create() returned %ld\n",
> + dlb2_driver_name, PTR_ERR(dlb2_class));
> +
> + return PTR_ERR(dlb2_class);
> + }
> +
> + /* Allocate one minor number per domain */
> + err = alloc_chrdev_region(&dlb2_dev_number_base,
> + 0,
> + DLB2_MAX_NUM_DEVICES,
> + dlb2_driver_name);
> +
> + if (err < 0) {
> + pr_err("%s: alloc_chrdev_region() returned %d\n",
> + dlb2_driver_name, err);
> +
> + return err;
> + }
> +
> + err = pci_register_driver(&dlb2_pci_driver);
> + if (err < 0) {
> + pr_err("%s: pci_register_driver() returned %d\n",
> + dlb2_driver_name, err);
> + return err;
> + }
> +
> + return 0;
> +}
> +
> +static void __exit dlb2_exit_module(void)
> +{
> + pci_unregister_driver(&dlb2_pci_driver);
> +
> + unregister_chrdev_region(dlb2_dev_number_base,
> + DLB2_MAX_NUM_DEVICES);
> +
> + if (dlb2_class) {
> + class_destroy(dlb2_class);
> + dlb2_class = NULL;
Why set this?
> + }
No freeing of allocated ida structures? Or is that not needed anymore,
I can never remember...
> +}
> +
> +module_init(dlb2_init_module);
> +module_exit(dlb2_exit_module);
> diff --git a/drivers/misc/dlb2/dlb2_main.h b/drivers/misc/dlb2/dlb2_main.h
> new file mode 100644
> index 000000000000..cc05546fba13
> --- /dev/null
> +++ b/drivers/misc/dlb2/dlb2_main.h
> @@ -0,0 +1,37 @@
> +/* SPDX-License-Identifier: GPL-2.0-only
> + * Copyright(c) 2017-2020 Intel Corporation
> + */
> +
> +#ifndef __DLB2_MAIN_H
> +#define __DLB2_MAIN_H
> +
> +#include <linux/cdev.h>
> +#include <linux/device.h>
> +#include <linux/ktime.h>
> +#include <linux/list.h>
> +#include <linux/mutex.h>
> +#include <linux/pci.h>
> +#include <linux/types.h>
> +
> +#include "dlb2_hw_types.h"
> +
> +static const char dlb2_driver_name[] = KBUILD_MODNAME;
> +
> +/*
> + * The dlb2 driver uses a different minor number for each device file, of which
> + * there are:
> + * - 33 per device (PF or VF/VDEV): 1 for the device, 32 for scheduling domains
> + * - Up to 17 devices per PF: 1 PF and up to 16 VFs/VDEVs
> + * - Up to 16 PFs per system
> + */
> +#define DLB2_MAX_NUM_PFS 16
> +#define DLB2_NUM_FUNCS_PER_DEVICE (1 + DLB2_MAX_NUM_VDEVS)
> +#define DLB2_MAX_NUM_DEVICES (DLB2_MAX_NUM_PFS * DLB2_NUM_FUNCS_PER_DEVICE)
> +
> +struct dlb2_dev {
> + struct pci_dev *pdev;
> + struct list_head list;
Why is list needed?
> + int id;
u32? u64? u8? Pick something :)
> +};
> +
> +#endif /* __DLB2_MAIN_H */
> diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
> index 5c709a1450b1..eb865b4eb900 100644
> --- a/include/linux/pci_ids.h
> +++ b/include/linux/pci_ids.h
> @@ -2809,6 +2809,8 @@
> #define PCI_DEVICE_ID_INTEL_ESB2_14 0x2698
> #define PCI_DEVICE_ID_INTEL_ESB2_17 0x269b
> #define PCI_DEVICE_ID_INTEL_ESB2_18 0x269e
> +#define PCI_DEVICE_ID_INTEL_DLB2_PF 0x2710
> +#define PCI_DEVICE_ID_INTEL_DLB2_VF 0x2711
Did you read the top of this file? How does this patch justify the
request there?
{sigh}
I'm not reviewing beyond this patch, sorry.
greg k-h
On Tue, Aug 11, 2020 at 11:27:15AM -0500, Gage Eads wrote:
> The driver detects the device type (PF/VF) at probe time, and assigns the
> corresponding 'ops' callbacks from that. These callbacks include mapping
> and unmapping the PCI BAR space, creating/destroying the device, and
> adding/deleting a char device.
>
> Signed-off-by: Gage Eads <[email protected]>
> Reviewed-by: Magnus Karlsson <[email protected]>
> ---
> drivers/misc/dlb2/Makefile | 5 +-
> drivers/misc/dlb2/dlb2_hw_types.h | 28 +++++++++
> drivers/misc/dlb2/dlb2_main.c | 49 ++++++++++++++-
> drivers/misc/dlb2/dlb2_main.h | 30 +++++++++
> drivers/misc/dlb2/dlb2_pf_ops.c | 126 ++++++++++++++++++++++++++++++++++++++
> 5 files changed, 235 insertions(+), 3 deletions(-)
> create mode 100644 drivers/misc/dlb2/dlb2_pf_ops.c
>
> diff --git a/drivers/misc/dlb2/Makefile b/drivers/misc/dlb2/Makefile
> index 90ae953d2a8f..95e67e5bd8ff 100644
> --- a/drivers/misc/dlb2/Makefile
> +++ b/drivers/misc/dlb2/Makefile
> @@ -4,5 +4,6 @@
>
> obj-$(CONFIG_INTEL_DLB2) := dlb2.o
>
> -dlb2-objs := \
> - dlb2_main.o \
> +dlb2-objs := \
> + dlb2_main.o \
> + dlb2_pf_ops.o \
> diff --git a/drivers/misc/dlb2/dlb2_hw_types.h b/drivers/misc/dlb2/dlb2_hw_types.h
> index a0f908c603eb..10b26356535f 100644
> --- a/drivers/misc/dlb2/dlb2_hw_types.h
> +++ b/drivers/misc/dlb2/dlb2_hw_types.h
> @@ -5,6 +5,25 @@
> #ifndef __DLB2_HW_TYPES_H
> #define __DLB2_HW_TYPES_H
>
> +#include <linux/io.h>
> +
> +#define DLB2_PCI_REG_READ(addr) ioread32(addr)
> +#define DLB2_PCI_REG_WRITE(reg, value) iowrite32(value, reg)
Why create a macro that is larger than the native call, this saves
nothing except make the reviewer wonder what in the world is happening.
Just use the real call.
> > +#include <linux/io.h>
> > +
> > +#define DLB2_PCI_REG_READ(addr) ioread32(addr)
> > +#define DLB2_PCI_REG_WRITE(reg, value) iowrite32(value, reg)
>
> Why create a macro that is larger than the native call, this saves
> nothing except make the reviewer wonder what in the world is happening.
>
> Just use the real call.
Sure, I'll change this in v3. I'll wait for more reviews before re-submitting.
Thanks,
Gage