Introduce a new misc device driver for the Intel(r) Dynamic Load Balancer
(Intel(r) DLB). The Intel DLB is a PCIe device that provides
load-balanced, prioritized scheduling of core-to-core communication.
Intel DLB is an accelerator for the event-driven programming model of
DPDK's Event Device Library[2]. The library is used in packet processing
pipelines that arrange for multi-core scalability, dynamic load-balancing,
and variety of packet distribution and synchronization schemes
These distribution schemes include "parallel" (packets are load-balanced
across multiple cores and processed in parallel), "ordered" (similar to
"parallel" but packets are reordered into ingress order by the device), and
"atomic" (packet flows are scheduled to a single core at a time such that
locks are not required to access per-flow data, and dynamically migrated to
ensure load-balance).
This submission supports Intel DLB 2.0 only.
The Intel DLB consists of queues and arbiters that connect producer
cores and consumer cores. The device implements load-balanced queueing
features including:
- Lock-free multi-producer/multi-consumer operation.
- Multiple priority levels for varying traffic types.
- 'Direct' traffic (i.e. multi-producer/single-consumer)
- Simple unordered load-balanced distribution.
- Atomic lock free load balancing across multiple consumers.
- Queue element reordering feature allowing ordered load-balanced
distribution.
The fundamental unit of communication through the device is a queue entry
(QE), which consists of 8B of data and 8B of metadata (destination queue,
priority, etc.). The data field can be any type that fits within 8B.
A core's interface to the device, a "port," consists of a memory-mappable
region through which the core enqueues a queue entry, and an in-memory
queue (the "consumer queue") to which the device schedules QEs. Each QE
is enqueued to a device-managed queue, and from there scheduled to a port.
Software specifies the "linking" of queues and ports; i.e. which ports the
device is allowed to schedule to for a given queue. The device uses a
credit scheme to prevent overflow of the on-device queue storage.
Applications can interface directly with the device by mapping the port's
memory and MMIO regions into the application's address space for enqueue
and dequeue operations, but call into the kernel driver for configuration
operations. An application can also be polling- or interrupt-driven;
Intel DLB supports both modes of operation.
Device resources -- i.e. ports, queues, and credits -- are contained within
a scheduling domain. Scheduling domains are isolated from one another; a
port can only enqueue to and dequeue from queues within its scheduling
domain. A scheduling domain's resources are configured through a scheduling
domain file, which is acquired through an ioctl.
Intel DLB supports SR-IOV and Scalable IOV, and allows for a flexible
division of its resources among the PF and its virtual devices. The virtual
devices are incapable of configuring the device directly; they use a
hardware mailbox to proxy configuration requests to the PF driver. This
driver supports both PF and virtual devices, as there is significant code
re-use between the two, with device-specific behavior handled through a
callback interface. Virtualization support will be added in a later patch
set.
The dlb driver uses ioctls as its primary interface (it makes use of sysfs
as well, to a lesser extent). The dlb device file supports a different
ioctl interface than the scheduling domain file; the dlb device file
is used for device-wide operations (including scheduling domain creation),
and the scheduling domain file supports operations on the scheduling
domain's resources (primarily resource configuration). Scheduling domains
are created dynamically (using a dlb device file ioctl) by user-space
software, and the scheduling domain file is created from an anonymous file
that is installed in the ioctl's calling process's file descriptor table.
[1] https://builders.intel.com/docs/networkbuilders/SKU-343247-001US-queue-management-and-load-balancing-on-intel-architecture.pdf
[2] https://doc.dpdk.org/guides/prog_guide/eventdev.html
v9:
- Addressed all of Greg's feecback on v8, including
-- Remove function name (__func__) from dev_err() messages, that could spam log.
-- Replace list and function pointer calls in dlb_ioctl() with switch-case
and real function callsi for ioctl.
-- Drop the compat_ptr_ioctl in dlb_ops (struct file_operations).
-- Change ioctl magic number for DLB to unused 0x81 (from 'h').
-- Remove all placeholder/dummy functions in the patch set.
-- Re-arrange the comments in dlb.h so that the order is consistent with that
of data structures referred.
-- Correct the comments on SPDX License and DLB versions in dlb.h.
-- Replace BIT_SET() and BITS_CLR() marcos with direct coding.
-- Remove NULL pointer checking (f->private_data) in dlb_ioctl().
-- Use whole line whenever possible and not wrapping lines unnecessarily.
-- Remove __attribute__((unused)).
-- Merge dlb_ioctl.h and dlb_file.h into dlb_main.h
v8:
- Add a functional block diagram in dlb.rst
- Modify change logs to reflect the links between patches and DPDK
eventdev library.
- Add a check of power-of-2 for CQ depth.
- Move call to INIT_WORK() to dlb_open().
- Clean dlb workqueue by calling flush_scheduled_work().
- Add unmap_mapping_range() in dlb_port_close().
v7 (Intel internal version):
- Address all of Dan's feedback, including
-- Drop DLB 2.0 throughout the patch set, use DLB only.
-- Fix license and copyright statements
-- Use pcim_enable_device() and pcim_iomap_regions(), instead of
unmanaged version.
-- Move cdev_add() to dlb_init() and add all devices at once.
-- Fix Makefile, using "+=" style.
-- Remove FLR description and mention movdir64/enqcmd usage in doc.
-- Make the permission for the domain same as that for device for
ioctl access.
-- Use idr instead of ida.
-- Add a lock in dlb_close() to prevent driver unbinding while ioctl
coomands are in progress.
-- Remove wrappers that are used for code sharing between kernel driver
and DPDK.
- Address Pierre-Louis' feedback, including
-- Clean the warinings from checkpatch
-- Fix the warnings from "make W=1"
v6 (Intel internal version):
- Change the module name to dlb(from dlb2), which currently supports Intel
DLB 2.0 only.
- Address all of Pierre-Louis' feedback on v5, including
-- Consolidate the two near-identical for loops in dlb2_release_domain_memory().
-- Remove an unnecessary "port = NULL" initialization
-- Consistently use curly braces on the *_LIST_FOR macros
when the for-loop contents spans multiple lines.
-- Add a comment to the definition of DLB2FS_MAGIC
-- Remove always true if statemnets
-- Move the get_cos_bw mutex unlock call earlier to shorten the critical
section.
- Address all of Dan's feedbacks, including
-- Replace the unions for register bits access with bitmask and shifts
-- Centralize the "to/from" user memory copies for ioctl functions.
-- Review ioctl design against Documentation/process/botching-up-ioctls.rst
-- Remove wraper functions for memory barriers.
-- Use ilog() to simplify a switch code block.
-- Add base-commit to cover letter.
v5 (Intel internal version):
- Reduce the scope of the initial patch set (drop the last 8 patches)
- Further decompose some of the remaining patches into multiple patches.
- Address all of Pierre-Louis' feedback, including:
-- Move kerneldoc to *.c files
-- Fix SPDX comment style
-- Add BAR macros
-- Improve/clarify struct dlb2_dev and struct device variable naming
-- Add const where missing
-- Clarify existing comments and add new ones in various places
-- Remove unnecessary memsets and zero-initialization
-- Remove PM abstraction, fix missing pm_runtime_allow(), and don't
update PM refcnt when port files are opened and closed.
-- Convert certain ternary operations into if-statements
-- Out-line the CQ depth valid check
-- De-duplicate the logic in dlb2_release_device_memory()
-- Limit use of devm functions to allocating/freeing struct dlb2
- Address Ira's comments on dlb2.rst and correct commit messages that
don't use the imperative voice.
v4:
- Move PCI device ID definitions into dlb2_hw_types.h, drop the VF definition
- Remove dlb2_dev_list
- Remove open/close functions and fops structure (unused)
- Remove "(char *)" cast from PCI driver name
- Unwind init failures properly
- Remove ID alloc helper functions and call IDA interfaces directly instead
v3:
- Remove DLB2_PCI_REG_READ/WRITE macros
v2:
- Change driver license to GPLv2 only
- Expand Kconfig help text and remove unnecessary (R)s
- Remove unnecessary prints
- Add a new entry in ioctl-number.rst
- Convert the ioctl handler into a switch statement
- Correct some instances of IOWR that should have been IOR
- Align macro blocks
- Don't break ioctl ABI when introducing new commands
- Remove indirect pointers from ioctl data structures
- Remove the get-sched-domain-fd ioctl command
Mike Ximing Chen (20):
dlb: add skeleton for DLB driver
dlb: initialize device
dlb: add resource and device initialization
dlb: add device ioctl layer and first three ioctls
dlb: add scheduling domain configuration
dlb: add domain software reset
dlb: add low-level register reset operations
dlb: add runtime power-management support
dlb: add queue create, reset, get-depth ioctls
dlb: add register operations for queue management
dlb: add ioctl to configure ports and query poll mode
dlb: add register operations for port management
dlb: add port mmap support
dlb: add start domain ioctl
dlb: add queue map, unmap, and pending unmap operations
dlb: add port map/unmap state machine
dlb: add static queue map register operations
dlb: add dynamic queue map register operations
dlb: add queue unmap register operations
dlb: queue map/unmap workqueue
Documentation/misc-devices/dlb.rst | 259 +
Documentation/misc-devices/index.rst | 1 +
.../userspace-api/ioctl/ioctl-number.rst | 1 +
MAINTAINERS | 8 +
drivers/misc/Kconfig | 1 +
drivers/misc/Makefile | 1 +
drivers/misc/dlb/Kconfig | 18 +
drivers/misc/dlb/Makefile | 11 +
drivers/misc/dlb/dlb_bitmap.h | 210 +
drivers/misc/dlb/dlb_file.c | 149 +
drivers/misc/dlb/dlb_hw_types.h | 311 +
drivers/misc/dlb/dlb_ioctl.c | 498 ++
drivers/misc/dlb/dlb_main.c | 614 ++
drivers/misc/dlb/dlb_main.h | 178 +
drivers/misc/dlb/dlb_pf_ops.c | 277 +
drivers/misc/dlb/dlb_regs.h | 3640 +++++++++++
drivers/misc/dlb/dlb_resource.c | 5469 +++++++++++++++++
drivers/misc/dlb/dlb_resource.h | 94 +
include/uapi/linux/dlb.h | 602 ++
19 files changed, 12342 insertions(+)
create mode 100644 Documentation/misc-devices/dlb.rst
create mode 100644 drivers/misc/dlb/Kconfig
create mode 100644 drivers/misc/dlb/Makefile
create mode 100644 drivers/misc/dlb/dlb_bitmap.h
create mode 100644 drivers/misc/dlb/dlb_file.c
create mode 100644 drivers/misc/dlb/dlb_hw_types.h
create mode 100644 drivers/misc/dlb/dlb_ioctl.c
create mode 100644 drivers/misc/dlb/dlb_main.c
create mode 100644 drivers/misc/dlb/dlb_main.h
create mode 100644 drivers/misc/dlb/dlb_pf_ops.c
create mode 100644 drivers/misc/dlb/dlb_regs.h
create mode 100644 drivers/misc/dlb/dlb_resource.c
create mode 100644 drivers/misc/dlb/dlb_resource.h
create mode 100644 include/uapi/linux/dlb.h
base-commit: e71ba9452f0b5b2e8dc8aa5445198cd9214a6a62
--
2.17.1
Add the hardware resource data structures, functions for
their initialization/teardown, and a function for device power-on. In
subsequent commits, dlb_resource.c will be expanded to hold the dlb
resource-management and configuration logic (using the data structures
defined in dlb_hw_types.h).
Introduce dlb_bitmap_* functions, a thin convenience layer wrapping the
Linux bitmap interfaces, used by the bitmaps in the dlb hardware types.
Signed-off-by: Gage Eads <[email protected]>
Signed-off-by: Mike Ximing Chen <[email protected]>
Reviewed-by: Magnus Karlsson <[email protected]>
Reviewed-by: Dan Williams <[email protected]>
---
drivers/misc/dlb/Makefile | 2 +-
drivers/misc/dlb/dlb_bitmap.h | 76 +++++++++++
drivers/misc/dlb/dlb_hw_types.h | 177 +++++++++++++++++++++++++
drivers/misc/dlb/dlb_main.c | 47 +++++++
drivers/misc/dlb/dlb_main.h | 8 ++
drivers/misc/dlb/dlb_pf_ops.c | 66 +++++++++-
drivers/misc/dlb/dlb_regs.h | 119 +++++++++++++++++
drivers/misc/dlb/dlb_resource.c | 220 ++++++++++++++++++++++++++++++++
drivers/misc/dlb/dlb_resource.h | 17 +++
9 files changed, 727 insertions(+), 5 deletions(-)
create mode 100644 drivers/misc/dlb/dlb_bitmap.h
create mode 100644 drivers/misc/dlb/dlb_regs.h
create mode 100644 drivers/misc/dlb/dlb_resource.c
create mode 100644 drivers/misc/dlb/dlb_resource.h
diff --git a/drivers/misc/dlb/Makefile b/drivers/misc/dlb/Makefile
index a33bf774e6a8..8a49ea5fd752 100644
--- a/drivers/misc/dlb/Makefile
+++ b/drivers/misc/dlb/Makefile
@@ -7,4 +7,4 @@
obj-$(CONFIG_INTEL_DLB) := dlb.o
dlb-objs := dlb_main.o
-dlb-objs += dlb_pf_ops.o
+dlb-objs += dlb_pf_ops.o dlb_resource.o
diff --git a/drivers/misc/dlb/dlb_bitmap.h b/drivers/misc/dlb/dlb_bitmap.h
new file mode 100644
index 000000000000..fb3ef52a306d
--- /dev/null
+++ b/drivers/misc/dlb/dlb_bitmap.h
@@ -0,0 +1,76 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#ifndef __DLB_OSDEP_BITMAP_H
+#define __DLB_OSDEP_BITMAP_H
+
+#include <linux/bitmap.h>
+#include <linux/slab.h>
+
+#include "dlb_main.h"
+
+/*************************/
+/*** Bitmap operations ***/
+/*************************/
+struct dlb_bitmap {
+ unsigned long *map;
+ unsigned int len;
+};
+
+/**
+ * dlb_bitmap_alloc() - alloc a bitmap data structure
+ * @bitmap: pointer to dlb_bitmap structure pointer.
+ * @len: number of entries in the bitmap.
+ *
+ * This function allocates a bitmap and initializes it with length @len. All
+ * entries are initially zero.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or len is 0.
+ * ENOMEM - could not allocate memory for the bitmap data structure.
+ */
+static inline int dlb_bitmap_alloc(struct dlb_bitmap **bitmap,
+ unsigned int len)
+{
+ struct dlb_bitmap *bm;
+
+ if (!bitmap || len == 0)
+ return -EINVAL;
+
+ bm = kzalloc(sizeof(*bm), GFP_KERNEL);
+ if (!bm)
+ return -ENOMEM;
+
+ bm->map = bitmap_zalloc(len, GFP_KERNEL);
+ if (!bm->map) {
+ kfree(bm);
+ return -ENOMEM;
+ }
+
+ bm->len = len;
+
+ *bitmap = bm;
+
+ return 0;
+}
+
+/**
+ * dlb_bitmap_free() - free a previously allocated bitmap data structure
+ * @bitmap: pointer to dlb_bitmap structure.
+ *
+ * This function frees a bitmap that was allocated with dlb_bitmap_alloc().
+ */
+static inline void dlb_bitmap_free(struct dlb_bitmap *bitmap)
+{
+ if (!bitmap)
+ return;
+
+ bitmap_free(bitmap->map);
+
+ kfree(bitmap);
+}
+
+#endif /* __DLB_OSDEP_BITMAP_H */
diff --git a/drivers/misc/dlb/dlb_hw_types.h b/drivers/misc/dlb/dlb_hw_types.h
index a4ce28c157de..3e03b061d5ff 100644
--- a/drivers/misc/dlb/dlb_hw_types.h
+++ b/drivers/misc/dlb/dlb_hw_types.h
@@ -5,6 +5,13 @@
#define __DLB_HW_TYPES_H
#include <linux/io.h>
+#include <linux/types.h>
+
+#include "dlb_bitmap.h"
+
+#define BITS_SET(x, val, mask) (x = ((x) & ~(mask)) \
+ | (((val) << (mask##_LOC)) & (mask)))
+#define BITS_GET(x, mask) (((x) & (mask)) >> (mask##_LOC))
/* Read/write register 'reg' in the CSR BAR space */
#define DLB_CSR_REG_ADDR(a, reg) ((a)->csr_kva + (reg))
@@ -43,6 +50,169 @@
#define PCI_DEVICE_ID_INTEL_DLB_PF 0x2710
+struct dlb_resource_id {
+ u32 phys_id;
+ u32 virt_id;
+ u8 vdev_owned;
+ u8 vdev_id;
+};
+
+struct dlb_freelist {
+ u32 base;
+ u32 bound;
+ u32 offset;
+};
+
+static inline u32 dlb_freelist_count(struct dlb_freelist *list)
+{
+ return list->bound - list->base - list->offset;
+}
+
+struct dlb_ldb_queue {
+ struct list_head domain_list;
+ struct list_head func_list;
+ struct dlb_resource_id id;
+ struct dlb_resource_id domain_id;
+ u32 num_qid_inflights;
+ u32 aqed_limit;
+ u32 sn_group; /* sn == sequence number */
+ u32 sn_slot;
+ u32 num_mappings;
+ u8 sn_cfg_valid;
+ u8 num_pending_additions;
+ u8 owned;
+ u8 configured;
+};
+
+/*
+ * Directed ports and queues are paired by nature, so the driver tracks them
+ * with a single data structure.
+ */
+struct dlb_dir_pq_pair {
+ struct list_head domain_list;
+ struct list_head func_list;
+ struct dlb_resource_id id;
+ struct dlb_resource_id domain_id;
+ u32 ref_cnt;
+ u8 init_tkn_cnt;
+ u8 queue_configured;
+ u8 port_configured;
+ u8 owned;
+ u8 enabled;
+};
+
+enum dlb_qid_map_state {
+ /* The slot doesn't contain a valid queue mapping */
+ DLB_QUEUE_UNMAPPED,
+ /* The slot contains a valid queue mapping */
+ DLB_QUEUE_MAPPED,
+ /* The driver is mapping a queue into this slot */
+ DLB_QUEUE_MAP_IN_PROG,
+ /* The driver is unmapping a queue from this slot */
+ DLB_QUEUE_UNMAP_IN_PROG,
+ /*
+ * The driver is unmapping a queue from this slot, and once complete
+ * will replace it with another mapping.
+ */
+ DLB_QUEUE_UNMAP_IN_PROG_PENDING_MAP,
+};
+
+struct dlb_ldb_port_qid_map {
+ enum dlb_qid_map_state state;
+ u16 qid;
+ u16 pending_qid;
+ u8 priority;
+ u8 pending_priority;
+};
+
+struct dlb_ldb_port {
+ struct list_head domain_list;
+ struct list_head func_list;
+ struct dlb_resource_id id;
+ struct dlb_resource_id domain_id;
+ /* The qid_map represents the hardware QID mapping state. */
+ struct dlb_ldb_port_qid_map qid_map[DLB_MAX_NUM_QIDS_PER_LDB_CQ];
+ u32 hist_list_entry_base;
+ u32 hist_list_entry_limit;
+ u32 ref_cnt;
+ u8 init_tkn_cnt;
+ u8 num_pending_removals;
+ u8 num_mappings;
+ u8 owned;
+ u8 enabled;
+ u8 configured;
+};
+
+struct dlb_sn_group {
+ u32 mode;
+ u32 sequence_numbers_per_queue;
+ u32 slot_use_bitmap;
+ u32 id;
+};
+
+struct dlb_hw_domain {
+ struct dlb_function_resources *parent_func;
+ struct list_head func_list;
+ struct list_head used_ldb_queues;
+ struct list_head used_ldb_ports[DLB_NUM_COS_DOMAINS];
+ struct list_head used_dir_pq_pairs;
+ struct list_head avail_ldb_queues;
+ struct list_head avail_ldb_ports[DLB_NUM_COS_DOMAINS];
+ struct list_head avail_dir_pq_pairs;
+ u32 total_hist_list_entries;
+ u32 avail_hist_list_entries;
+ u32 hist_list_entry_base;
+ u32 hist_list_entry_offset;
+ u32 num_ldb_credits;
+ u32 num_dir_credits;
+ u32 num_avail_aqed_entries;
+ u32 num_used_aqed_entries;
+ struct dlb_resource_id id;
+ int num_pending_removals;
+ int num_pending_additions;
+ u8 configured;
+ u8 started;
+};
+
+struct dlb_function_resources {
+ struct list_head avail_domains;
+ struct list_head used_domains;
+ struct list_head avail_ldb_queues;
+ struct list_head avail_ldb_ports[DLB_NUM_COS_DOMAINS];
+ struct list_head avail_dir_pq_pairs;
+ struct dlb_bitmap *avail_hist_list_entries;
+ u32 num_avail_domains;
+ u32 num_avail_ldb_queues;
+ u32 num_avail_ldb_ports[DLB_NUM_COS_DOMAINS];
+ u32 num_avail_dir_pq_pairs;
+ u32 num_avail_qed_entries;
+ u32 num_avail_dqed_entries;
+ u32 num_avail_aqed_entries;
+ u8 locked; /* (VDEV only) */
+};
+
+/*
+ * After initialization, each resource in dlb_hw_resources is located in one
+ * of the following lists:
+ * -- The PF's available resources list. These are unconfigured resources owned
+ * by the PF and not allocated to a dlb scheduling domain.
+ * -- A VDEV's available resources list. These are VDEV-owned unconfigured
+ * resources not allocated to a dlb scheduling domain.
+ * -- A domain's available resources list. These are domain-owned unconfigured
+ * resources.
+ * -- A domain's used resources list. These are domain-owned configured
+ * resources.
+ *
+ * A resource moves to a new list when a VDEV or domain is created or destroyed,
+ * or when the resource is configured.
+ */
+struct dlb_hw_resources {
+ struct dlb_ldb_queue ldb_queues[DLB_MAX_NUM_LDB_QUEUES];
+ struct dlb_ldb_port ldb_ports[DLB_MAX_NUM_LDB_PORTS];
+ struct dlb_dir_pq_pair dir_pq_pairs[DLB_MAX_NUM_DIR_PORTS];
+ struct dlb_sn_group sn_groups[DLB_MAX_NUM_SEQUENCE_NUMBER_GROUPS];
+};
+
struct dlb_hw {
/* BAR 0 address */
void __iomem *csr_kva;
@@ -50,6 +220,13 @@ struct dlb_hw {
/* BAR 2 address */
void __iomem *func_kva;
unsigned long func_phys_addr;
+
+ /* Resource tracking */
+ struct dlb_hw_resources rsrcs;
+ struct dlb_function_resources pf;
+ struct dlb_function_resources vdev[DLB_MAX_NUM_VDEVS];
+ struct dlb_hw_domain domains[DLB_MAX_NUM_DOMAINS];
+ u8 cos_reservation[DLB_NUM_COS_DOMAINS];
};
#endif /* __DLB_HW_TYPES_H */
diff --git a/drivers/misc/dlb/dlb_main.c b/drivers/misc/dlb/dlb_main.c
index 7fb6e9c360c8..12707b23ab3e 100644
--- a/drivers/misc/dlb/dlb_main.c
+++ b/drivers/misc/dlb/dlb_main.c
@@ -11,6 +11,7 @@
#include <linux/uaccess.h>
#include "dlb_main.h"
+#include "dlb_resource.h"
MODULE_LICENSE("GPL v2");
MODULE_DESCRIPTION("Intel(R) Dynamic Load Balancer (DLB) Driver");
@@ -21,6 +22,23 @@ static dev_t dlb_devt;
static DEFINE_IDR(dlb_ids);
static DEFINE_SPINLOCK(dlb_ids_lock);
+static int dlb_reset_device(struct pci_dev *pdev)
+{
+ int ret;
+
+ ret = pci_save_state(pdev);
+ if (ret)
+ return ret;
+
+ ret = __pci_reset_function_locked(pdev);
+ if (ret)
+ return ret;
+
+ pci_restore_state(pdev);
+
+ return 0;
+}
+
static int dlb_device_create(struct dlb *dlb, struct pci_dev *pdev)
{
/*
@@ -113,8 +131,35 @@ static int dlb_probe(struct pci_dev *pdev, const struct pci_device_id *pdev_id)
if (ret)
goto dma_set_mask_fail;
+ /*
+ * PM enable must be done before any other MMIO accesses, and this
+ * setting is persistent across device reset.
+ */
+ dlb->ops->enable_pm(dlb);
+
+ ret = dlb->ops->wait_for_device_ready(dlb, pdev);
+ if (ret)
+ goto wait_for_device_ready_fail;
+
+ ret = dlb_reset_device(pdev);
+ if (ret)
+ goto dlb_reset_fail;
+
+ ret = dlb_resource_init(&dlb->hw);
+ if (ret)
+ goto resource_init_fail;
+
+ ret = dlb->ops->init_driver_state(dlb);
+ if (ret)
+ goto init_driver_state_fail;
+
return 0;
+init_driver_state_fail:
+ dlb_resource_free(&dlb->hw);
+resource_init_fail:
+dlb_reset_fail:
+wait_for_device_ready_fail:
dma_set_mask_fail:
device_destroy(dlb_class, dlb->dev_number);
map_pci_bar_fail:
@@ -131,6 +176,8 @@ static void dlb_remove(struct pci_dev *pdev)
{
struct dlb *dlb = pci_get_drvdata(pdev);
+ dlb_resource_free(&dlb->hw);
+
device_destroy(dlb_class, dlb->dev_number);
pci_disable_pcie_error_reporting(pdev);
diff --git a/drivers/misc/dlb/dlb_main.h b/drivers/misc/dlb/dlb_main.h
index 21570b206419..ec5eb7bd8f54 100644
--- a/drivers/misc/dlb/dlb_main.h
+++ b/drivers/misc/dlb/dlb_main.h
@@ -34,6 +34,9 @@ struct dlb;
struct dlb_device_ops {
int (*map_pci_bar_space)(struct dlb *dlb, struct pci_dev *pdev);
void (*unmap_pci_bar_space)(struct dlb *dlb, struct pci_dev *pdev);
+ int (*init_driver_state)(struct dlb *dlb);
+ void (*enable_pm)(struct dlb *dlb);
+ int (*wait_for_device_ready)(struct dlb *dlb, struct pci_dev *pdev);
};
extern struct dlb_device_ops dlb_pf_ops;
@@ -43,6 +46,11 @@ struct dlb {
struct dlb_hw hw;
struct dlb_device_ops *ops;
struct device *dev;
+ /*
+ * The resource mutex serializes access to driver data structures and
+ * hardware registers.
+ */
+ struct mutex resource_mutex;
enum dlb_device_type type;
int id;
dev_t dev_number;
diff --git a/drivers/misc/dlb/dlb_pf_ops.c b/drivers/misc/dlb/dlb_pf_ops.c
index 0951c99f6183..124b4fee8564 100644
--- a/drivers/misc/dlb/dlb_pf_ops.c
+++ b/drivers/misc/dlb/dlb_pf_ops.c
@@ -1,21 +1,23 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+#include <linux/delay.h>
+
#include "dlb_main.h"
+#include "dlb_regs.h"
+#include "dlb_resource.h"
/********************************/
/****** PCI BAR management ******/
/********************************/
-static void
-dlb_pf_unmap_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev)
+static void dlb_pf_unmap_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev)
{
pcim_iounmap(pdev, dlb->hw.csr_kva);
pcim_iounmap(pdev, dlb->hw.func_kva);
}
-static int
-dlb_pf_map_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev)
+static int dlb_pf_map_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev)
{
dlb->hw.func_kva = pcim_iomap_table(pdev)[DLB_FUNC_BAR];
dlb->hw.func_phys_addr = pci_resource_start(pdev, DLB_FUNC_BAR);
@@ -40,6 +42,59 @@ dlb_pf_map_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev)
return 0;
}
+/*******************************/
+/****** Driver management ******/
+/*******************************/
+
+static int dlb_pf_init_driver_state(struct dlb *dlb)
+{
+ mutex_init(&dlb->resource_mutex);
+
+ return 0;
+}
+
+static void dlb_pf_enable_pm(struct dlb *dlb)
+{
+ /*
+ * Clear the power-management-disable register to power on the bulk of
+ * the device's hardware.
+ */
+ dlb_clr_pmcsr_disable(&dlb->hw);
+}
+
+#define DLB_READY_RETRY_LIMIT 1000
+static int dlb_pf_wait_for_device_ready(struct dlb *dlb, struct pci_dev *pdev)
+{
+ u32 retries = DLB_READY_RETRY_LIMIT;
+
+ /* Allow at least 1s for the device to become active after power-on */
+ do {
+ u32 idle, pm_st, addr;
+
+ addr = CM_CFG_PM_STATUS;
+
+ pm_st = DLB_CSR_RD(&dlb->hw, addr);
+
+ addr = CM_CFG_DIAGNOSTIC_IDLE_STATUS;
+
+ idle = DLB_CSR_RD(&dlb->hw, addr);
+
+ if (BITS_GET(pm_st, CM_CFG_PM_STATUS_PMSM) == 1 &&
+ BITS_GET(idle, CM_CFG_DIAGNOSTIC_IDLE_STATUS_DLB_FUNC_IDLE)
+ == 1)
+ break;
+
+ usleep_range(1000, 2000);
+ } while (--retries);
+
+ if (!retries) {
+ dev_err(&pdev->dev, "Device idle test failed\n");
+ return -EIO;
+ }
+
+ return 0;
+}
+
/********************************/
/****** DLB PF Device Ops ******/
/********************************/
@@ -47,4 +102,7 @@ dlb_pf_map_pci_bar_space(struct dlb *dlb, struct pci_dev *pdev)
struct dlb_device_ops dlb_pf_ops = {
.map_pci_bar_space = dlb_pf_map_pci_bar_space,
.unmap_pci_bar_space = dlb_pf_unmap_pci_bar_space,
+ .init_driver_state = dlb_pf_init_driver_state,
+ .enable_pm = dlb_pf_enable_pm,
+ .wait_for_device_ready = dlb_pf_wait_for_device_ready,
};
diff --git a/drivers/misc/dlb/dlb_regs.h b/drivers/misc/dlb/dlb_regs.h
new file mode 100644
index 000000000000..72f3cb22b933
--- /dev/null
+++ b/drivers/misc/dlb/dlb_regs.h
@@ -0,0 +1,119 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#ifndef __DLB_REGS_H
+#define __DLB_REGS_H
+
+#include <linux/types.h>
+
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS 0xb4000004
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_RST 0x9d0fffff
+
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_CHP_PIPEIDLE 0x00000001
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_ROP_PIPEIDLE 0x00000002
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_LSP_PIPEIDLE 0x00000004
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_NALB_PIPEIDLE 0x00000008
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_AP_PIPEIDLE 0x00000010
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DP_PIPEIDLE 0x00000020
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_QED_PIPEIDLE 0x00000040
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DQED_PIPEIDLE 0x00000080
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_AQED_PIPEIDLE 0x00000100
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_SYS_PIPEIDLE 0x00000200
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_CHP_UNIT_IDLE 0x00000400
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_ROP_UNIT_IDLE 0x00000800
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_LSP_UNIT_IDLE 0x00001000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_NALB_UNIT_IDLE 0x00002000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_AP_UNIT_IDLE 0x00004000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DP_UNIT_IDLE 0x00008000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_QED_UNIT_IDLE 0x00010000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DQED_UNIT_IDLE 0x00020000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_AQED_UNIT_IDLE 0x00040000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_SYS_UNIT_IDLE 0x00080000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_RSVD1 0x00F00000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_CFG_RING_IDLE 0x01000000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_CFG_MSTR_IDLE 0x02000000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_FLR_CLKREQ_B 0x04000000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_PROC_IDLE 0x08000000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_PROC_IDLE_MASKED 0x10000000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_RSVD0 0x60000000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DLB_FUNC_IDLE 0x80000000
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_CHP_PIPEIDLE_LOC 0
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_ROP_PIPEIDLE_LOC 1
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_LSP_PIPEIDLE_LOC 2
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_NALB_PIPEIDLE_LOC 3
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_AP_PIPEIDLE_LOC 4
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DP_PIPEIDLE_LOC 5
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_QED_PIPEIDLE_LOC 6
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DQED_PIPEIDLE_LOC 7
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_AQED_PIPEIDLE_LOC 8
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_SYS_PIPEIDLE_LOC 9
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_CHP_UNIT_IDLE_LOC 10
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_ROP_UNIT_IDLE_LOC 11
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_LSP_UNIT_IDLE_LOC 12
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_NALB_UNIT_IDLE_LOC 13
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_AP_UNIT_IDLE_LOC 14
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DP_UNIT_IDLE_LOC 15
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_QED_UNIT_IDLE_LOC 16
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DQED_UNIT_IDLE_LOC 17
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_AQED_UNIT_IDLE_LOC 18
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_SYS_UNIT_IDLE_LOC 19
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_RSVD1_LOC 20
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_CFG_RING_IDLE_LOC 24
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_CFG_MSTR_IDLE_LOC 25
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_FLR_CLKREQ_B_LOC 26
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_PROC_IDLE_LOC 27
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_MSTR_PROC_IDLE_MASKED_LOC 28
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_RSVD0_LOC 29
+#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_DLB_FUNC_IDLE_LOC 31
+
+#define CM_CFG_PM_STATUS 0xb4000014
+#define CM_CFG_PM_STATUS_RST 0x100403e
+
+#define CM_CFG_PM_STATUS_PROCHOT 0x00000001
+#define CM_CFG_PM_STATUS_PGCB_DLB_IDLE 0x00000002
+#define CM_CFG_PM_STATUS_PGCB_DLB_PG_RDY_ACK_B 0x00000004
+#define CM_CFG_PM_STATUS_PMSM_PGCB_REQ_B 0x00000008
+#define CM_CFG_PM_STATUS_PGBC_PMC_PG_REQ_B 0x00000010
+#define CM_CFG_PM_STATUS_PMC_PGCB_PG_ACK_B 0x00000020
+#define CM_CFG_PM_STATUS_PMC_PGCB_FET_EN_B 0x00000040
+#define CM_CFG_PM_STATUS_PGCB_FET_EN_B 0x00000080
+#define CM_CFG_PM_STATUS_RSVZ0 0x00000100
+#define CM_CFG_PM_STATUS_RSVZ1 0x00000200
+#define CM_CFG_PM_STATUS_FUSE_FORCE_ON 0x00000400
+#define CM_CFG_PM_STATUS_FUSE_PROC_DISABLE 0x00000800
+#define CM_CFG_PM_STATUS_RSVZ2 0x00001000
+#define CM_CFG_PM_STATUS_RSVZ3 0x00002000
+#define CM_CFG_PM_STATUS_PM_FSM_D0TOD3_OK 0x00004000
+#define CM_CFG_PM_STATUS_PM_FSM_D3TOD0_OK 0x00008000
+#define CM_CFG_PM_STATUS_DLB_IN_D3 0x00010000
+#define CM_CFG_PM_STATUS_RSVZ4 0x00FE0000
+#define CM_CFG_PM_STATUS_PMSM 0xFF000000
+#define CM_CFG_PM_STATUS_PROCHOT_LOC 0
+#define CM_CFG_PM_STATUS_PGCB_DLB_IDLE_LOC 1
+#define CM_CFG_PM_STATUS_PGCB_DLB_PG_RDY_ACK_B_LOC 2
+#define CM_CFG_PM_STATUS_PMSM_PGCB_REQ_B_LOC 3
+#define CM_CFG_PM_STATUS_PGBC_PMC_PG_REQ_B_LOC 4
+#define CM_CFG_PM_STATUS_PMC_PGCB_PG_ACK_B_LOC 5
+#define CM_CFG_PM_STATUS_PMC_PGCB_FET_EN_B_LOC 6
+#define CM_CFG_PM_STATUS_PGCB_FET_EN_B_LOC 7
+#define CM_CFG_PM_STATUS_RSVZ0_LOC 8
+#define CM_CFG_PM_STATUS_RSVZ1_LOC 9
+#define CM_CFG_PM_STATUS_FUSE_FORCE_ON_LOC 10
+#define CM_CFG_PM_STATUS_FUSE_PROC_DISABLE_LOC 11
+#define CM_CFG_PM_STATUS_RSVZ2_LOC 12
+#define CM_CFG_PM_STATUS_RSVZ3_LOC 13
+#define CM_CFG_PM_STATUS_PM_FSM_D0TOD3_OK_LOC 14
+#define CM_CFG_PM_STATUS_PM_FSM_D3TOD0_OK_LOC 15
+#define CM_CFG_PM_STATUS_DLB_IN_D3_LOC 16
+#define CM_CFG_PM_STATUS_RSVZ4_LOC 17
+#define CM_CFG_PM_STATUS_PMSM_LOC 24
+
+#define CM_CFG_PM_PMCSR_DISABLE 0xb4000018
+#define CM_CFG_PM_PMCSR_DISABLE_RST 0x1
+
+#define CM_CFG_PM_PMCSR_DISABLE_DISABLE 0x00000001
+#define CM_CFG_PM_PMCSR_DISABLE_RSVZ0 0xFFFFFFFE
+#define CM_CFG_PM_PMCSR_DISABLE_DISABLE_LOC 0
+#define CM_CFG_PM_PMCSR_DISABLE_RSVZ0_LOC 1
+
+#endif /* __DLB_REGS_H */
diff --git a/drivers/misc/dlb/dlb_resource.c b/drivers/misc/dlb/dlb_resource.c
new file mode 100644
index 000000000000..fca444c46aca
--- /dev/null
+++ b/drivers/misc/dlb/dlb_resource.c
@@ -0,0 +1,220 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#include "dlb_bitmap.h"
+#include "dlb_hw_types.h"
+#include "dlb_regs.h"
+#include "dlb_resource.h"
+
+static void dlb_init_fn_rsrc_lists(struct dlb_function_resources *rsrc)
+{
+ int i;
+
+ INIT_LIST_HEAD(&rsrc->avail_domains);
+ INIT_LIST_HEAD(&rsrc->used_domains);
+ INIT_LIST_HEAD(&rsrc->avail_ldb_queues);
+ INIT_LIST_HEAD(&rsrc->avail_dir_pq_pairs);
+
+ for (i = 0; i < DLB_NUM_COS_DOMAINS; i++)
+ INIT_LIST_HEAD(&rsrc->avail_ldb_ports[i]);
+}
+
+static void dlb_init_domain_rsrc_lists(struct dlb_hw_domain *domain)
+{
+ int i;
+
+ INIT_LIST_HEAD(&domain->used_ldb_queues);
+ INIT_LIST_HEAD(&domain->used_dir_pq_pairs);
+ INIT_LIST_HEAD(&domain->avail_ldb_queues);
+ INIT_LIST_HEAD(&domain->avail_dir_pq_pairs);
+
+ for (i = 0; i < DLB_NUM_COS_DOMAINS; i++)
+ INIT_LIST_HEAD(&domain->used_ldb_ports[i]);
+ for (i = 0; i < DLB_NUM_COS_DOMAINS; i++)
+ INIT_LIST_HEAD(&domain->avail_ldb_ports[i]);
+}
+
+/**
+ * dlb_resource_free() - free device state memory
+ * @hw: dlb_hw handle for a particular device.
+ *
+ * This function frees software state pointed to by dlb_hw. This function
+ * should be called when resetting the device or unloading the driver.
+ */
+void dlb_resource_free(struct dlb_hw *hw)
+{
+ int i;
+
+ if (hw->pf.avail_hist_list_entries)
+ dlb_bitmap_free(hw->pf.avail_hist_list_entries);
+
+ for (i = 0; i < DLB_MAX_NUM_VDEVS; i++) {
+ if (hw->vdev[i].avail_hist_list_entries)
+ dlb_bitmap_free(hw->vdev[i].avail_hist_list_entries);
+ }
+}
+
+/**
+ * dlb_resource_init() - initialize the device
+ * @hw: pointer to struct dlb_hw.
+ *
+ * This function initializes the device's software state (pointed to by the hw
+ * argument) and programs global scheduling QoS registers. This function should
+ * be called during driver initialization, and the dlb_hw structure should
+ * be zero-initialized before calling the function.
+ *
+ * The dlb_hw struct must be unique per DLB 2.0 device and persist until the
+ * device is reset.
+ *
+ * Return:
+ * Returns 0 upon success, <0 otherwise.
+ */
+int dlb_resource_init(struct dlb_hw *hw)
+{
+ struct dlb_bitmap *map;
+ struct list_head *list;
+ unsigned int i;
+ int ret;
+
+ /*
+ * For optimal load-balancing, ports that map to one or more QIDs in
+ * common should not be in numerical sequence. The port->QID mapping is
+ * application dependent, but the driver interleaves port IDs as much
+ * as possible to reduce the likelihood of sequential ports mapping to
+ * the same QID(s). This initial allocation of port IDs maximizes the
+ * average distance between an ID and its immediate neighbors (i.e.
+ * the distance from 1 to 0 and to 2, the distance from 2 to 1 and to
+ * 3, etc.).
+ */
+ const u8 init_ldb_port_allocation[DLB_MAX_NUM_LDB_PORTS] = {
+ 0, 7, 14, 5, 12, 3, 10, 1, 8, 15, 6, 13, 4, 11, 2, 9,
+ 16, 23, 30, 21, 28, 19, 26, 17, 24, 31, 22, 29, 20, 27, 18, 25,
+ 32, 39, 46, 37, 44, 35, 42, 33, 40, 47, 38, 45, 36, 43, 34, 41,
+ 48, 55, 62, 53, 60, 51, 58, 49, 56, 63, 54, 61, 52, 59, 50, 57,
+ };
+
+ dlb_init_fn_rsrc_lists(&hw->pf);
+
+ for (i = 0; i < DLB_MAX_NUM_VDEVS; i++)
+ dlb_init_fn_rsrc_lists(&hw->vdev[i]);
+
+ for (i = 0; i < DLB_MAX_NUM_DOMAINS; i++) {
+ dlb_init_domain_rsrc_lists(&hw->domains[i]);
+ hw->domains[i].parent_func = &hw->pf;
+ }
+
+ /* Give all resources to the PF driver */
+ hw->pf.num_avail_domains = DLB_MAX_NUM_DOMAINS;
+ for (i = 0; i < hw->pf.num_avail_domains; i++) {
+ list = &hw->domains[i].func_list;
+
+ list_add(list, &hw->pf.avail_domains);
+ }
+
+ hw->pf.num_avail_ldb_queues = DLB_MAX_NUM_LDB_QUEUES;
+ for (i = 0; i < hw->pf.num_avail_ldb_queues; i++) {
+ list = &hw->rsrcs.ldb_queues[i].func_list;
+
+ list_add(list, &hw->pf.avail_ldb_queues);
+ }
+
+ for (i = 0; i < DLB_NUM_COS_DOMAINS; i++)
+ hw->pf.num_avail_ldb_ports[i] =
+ DLB_MAX_NUM_LDB_PORTS / DLB_NUM_COS_DOMAINS;
+
+ for (i = 0; i < DLB_MAX_NUM_LDB_PORTS; i++) {
+ int cos_id = i >> DLB_NUM_COS_DOMAINS;
+ struct dlb_ldb_port *port;
+
+ port = &hw->rsrcs.ldb_ports[init_ldb_port_allocation[i]];
+
+ list_add(&port->func_list, &hw->pf.avail_ldb_ports[cos_id]);
+ }
+
+ hw->pf.num_avail_dir_pq_pairs = DLB_MAX_NUM_DIR_PORTS;
+ for (i = 0; i < hw->pf.num_avail_dir_pq_pairs; i++) {
+ list = &hw->rsrcs.dir_pq_pairs[i].func_list;
+
+ list_add(list, &hw->pf.avail_dir_pq_pairs);
+ }
+
+ hw->pf.num_avail_qed_entries = DLB_MAX_NUM_LDB_CREDITS;
+ hw->pf.num_avail_dqed_entries = DLB_MAX_NUM_DIR_CREDITS;
+ hw->pf.num_avail_aqed_entries = DLB_MAX_NUM_AQED_ENTRIES;
+
+ ret = dlb_bitmap_alloc(&hw->pf.avail_hist_list_entries,
+ DLB_MAX_NUM_HIST_LIST_ENTRIES);
+ if (ret)
+ goto unwind;
+
+ map = hw->pf.avail_hist_list_entries;
+ bitmap_fill(map->map, map->len);
+
+ for (i = 0; i < DLB_MAX_NUM_VDEVS; i++) {
+ ret = dlb_bitmap_alloc(&hw->vdev[i].avail_hist_list_entries,
+ DLB_MAX_NUM_HIST_LIST_ENTRIES);
+ if (ret)
+ goto unwind;
+
+ map = hw->vdev[i].avail_hist_list_entries;
+ bitmap_zero(map->map, map->len);
+ }
+
+ /* Initialize the hardware resource IDs */
+ for (i = 0; i < DLB_MAX_NUM_DOMAINS; i++) {
+ hw->domains[i].id.phys_id = i;
+ hw->domains[i].id.vdev_owned = false;
+ }
+
+ for (i = 0; i < DLB_MAX_NUM_LDB_QUEUES; i++) {
+ hw->rsrcs.ldb_queues[i].id.phys_id = i;
+ hw->rsrcs.ldb_queues[i].id.vdev_owned = false;
+ }
+
+ for (i = 0; i < DLB_MAX_NUM_LDB_PORTS; i++) {
+ hw->rsrcs.ldb_ports[i].id.phys_id = i;
+ hw->rsrcs.ldb_ports[i].id.vdev_owned = false;
+ }
+
+ for (i = 0; i < DLB_MAX_NUM_DIR_PORTS; i++) {
+ hw->rsrcs.dir_pq_pairs[i].id.phys_id = i;
+ hw->rsrcs.dir_pq_pairs[i].id.vdev_owned = false;
+ }
+
+ for (i = 0; i < DLB_MAX_NUM_SEQUENCE_NUMBER_GROUPS; i++) {
+ hw->rsrcs.sn_groups[i].id = i;
+ /* Default mode (0) is 64 sequence numbers per queue */
+ hw->rsrcs.sn_groups[i].mode = 0;
+ hw->rsrcs.sn_groups[i].sequence_numbers_per_queue = 64;
+ hw->rsrcs.sn_groups[i].slot_use_bitmap = 0;
+ }
+
+ for (i = 0; i < DLB_NUM_COS_DOMAINS; i++)
+ hw->cos_reservation[i] = 100 / DLB_NUM_COS_DOMAINS;
+
+ return 0;
+
+unwind:
+ dlb_resource_free(hw);
+
+ return ret;
+}
+
+/**
+ * dlb_clr_pmcsr_disable() - power on bulk of DLB 2.0 logic
+ * @hw: dlb_hw handle for a particular device.
+ *
+ * Clearing the PMCSR must be done at initialization to make the device fully
+ * operational.
+ */
+void dlb_clr_pmcsr_disable(struct dlb_hw *hw)
+{
+ u32 pmcsr_dis;
+
+ pmcsr_dis = DLB_CSR_RD(hw, CM_CFG_PM_PMCSR_DISABLE);
+
+ /* Clear register bits */
+ pmcsr_dis &= ~CM_CFG_PM_PMCSR_DISABLE_DISABLE;
+
+ DLB_CSR_WR(hw, CM_CFG_PM_PMCSR_DISABLE, pmcsr_dis);
+}
diff --git a/drivers/misc/dlb/dlb_resource.h b/drivers/misc/dlb/dlb_resource.h
new file mode 100644
index 000000000000..2229813d9c45
--- /dev/null
+++ b/drivers/misc/dlb/dlb_resource.h
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#ifndef __DLB_RESOURCE_H
+#define __DLB_RESOURCE_H
+
+#include <linux/types.h>
+
+#include "dlb_hw_types.h"
+
+int dlb_resource_init(struct dlb_hw *hw);
+
+void dlb_resource_free(struct dlb_hw *hw);
+
+void dlb_clr_pmcsr_disable(struct dlb_hw *hw);
+
+#endif /* __DLB_RESOURCE_H */
--
2.17.1
Add support for the port map/unmap state machine. Each load-balanced port
has eight "slots", one for each queue it is linked to, and each slot can be
in one of five states:
1. Queue unmapped
2. Queue mapped
3. Queue unmap in progress
4. Queue map in progress
5. Queue unmap in progress, with a map pending when the unmap completes.
These states exist because the map and unmap operations can be asynchronous
(with respect to the ioctl command). If the domain is already started, the
map operation must (temporarily) disable the queue and wait for it to
quiesce. Similarly, the unmap operation must (temporarily) disable the port
and wait for it to quiesce. 'Quiesce' here means the user processes any
in-flight QEs.
The queue map/unmap in this commit refers to link/unlink between DLB's
load-balanced queues (internal) and consumer ports. See Documentation/
misc-devices/dlb.rst for details.
It's possible that the thread that requires the map/unmap is the same one
which is responsible for doing the processing that would quiesce the
queue/port, in which case the driver may have to complete the operation
asynchronously.
To support this asynchronous operation while also providing a reasonably
usable user-interface, the driver maintains two views of the queue map
(slot) state:
- The hardware view: the actual state in the device
- The user/software view: the state as though the operations were
synchronous.
While a map/unmap operation is inflight, these two views are out-of-sync.
When the user requests a new map/unmap operation, the driver verifies the
request against the software view, so any errors are synchronous from the
user’s perspective, then adds the request to the queue of in-progress
operations. When possible -- for example if the user requests to map a
queue and then immediately requests to unmap it -- the driver will coalesce
or cancel outstanding operations.
Signed-off-by: Gage Eads <[email protected]>
Signed-off-by: Mike Ximing Chen <[email protected]>
Reviewed-by: Björn Töpel <[email protected]>
Reviewed-by: Dan Williams <[email protected]>
---
drivers/misc/dlb/dlb_resource.c | 433 +++++++++++++++++++++++++++++++-
1 file changed, 431 insertions(+), 2 deletions(-)
diff --git a/drivers/misc/dlb/dlb_resource.c b/drivers/misc/dlb/dlb_resource.c
index f39853fc664f..a830b547dadf 100644
--- a/drivers/misc/dlb/dlb_resource.c
+++ b/drivers/misc/dlb/dlb_resource.c
@@ -1335,6 +1335,225 @@ static int dlb_verify_map_qid_args(struct dlb_hw *hw, u32 domain_id,
return 0;
}
+static bool dlb_port_find_slot(struct dlb_ldb_port *port,
+ enum dlb_qid_map_state state, int *slot)
+{
+ int i;
+
+ for (i = 0; i < DLB_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+ if (port->qid_map[i].state == state)
+ break;
+ }
+
+ *slot = i;
+
+ return (i < DLB_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static bool dlb_port_find_slot_queue(struct dlb_ldb_port *port,
+ enum dlb_qid_map_state state,
+ struct dlb_ldb_queue *queue, int *slot)
+{
+ int i;
+
+ for (i = 0; i < DLB_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+ if (port->qid_map[i].state == state &&
+ port->qid_map[i].qid == queue->id.phys_id)
+ break;
+ }
+
+ *slot = i;
+
+ return (i < DLB_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static bool
+dlb_port_find_slot_with_pending_map_queue(struct dlb_ldb_port *port,
+ struct dlb_ldb_queue *queue, int *slot)
+{
+ int i;
+
+ for (i = 0; i < DLB_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+ struct dlb_ldb_port_qid_map *map = &port->qid_map[i];
+
+ if (map->state == DLB_QUEUE_UNMAP_IN_PROG_PENDING_MAP &&
+ map->pending_qid == queue->id.phys_id)
+ break;
+ }
+
+ *slot = i;
+
+ return (i < DLB_MAX_NUM_QIDS_PER_LDB_CQ);
+}
+
+static int dlb_port_slot_state_transition(struct dlb_hw *hw,
+ struct dlb_ldb_port *port,
+ struct dlb_ldb_queue *queue, int slot,
+ enum dlb_qid_map_state new_state)
+{
+ enum dlb_qid_map_state curr_state = port->qid_map[slot].state;
+ struct dlb_hw_domain *domain;
+ int domain_id;
+
+ domain_id = port->domain_id.phys_id;
+
+ domain = dlb_get_domain_from_id(hw, domain_id, false, 0);
+ if (!domain) {
+ DLB_HW_ERR(hw,
+ "[%s()] Internal error: unable to find domain %d\n",
+ __func__, domain_id);
+ return -EINVAL;
+ }
+
+ switch (curr_state) {
+ case DLB_QUEUE_UNMAPPED:
+ switch (new_state) {
+ case DLB_QUEUE_MAPPED:
+ queue->num_mappings++;
+ port->num_mappings++;
+ break;
+ case DLB_QUEUE_MAP_IN_PROG:
+ queue->num_pending_additions++;
+ domain->num_pending_additions++;
+ break;
+ default:
+ goto error;
+ }
+ break;
+ case DLB_QUEUE_MAPPED:
+ switch (new_state) {
+ case DLB_QUEUE_UNMAPPED:
+ queue->num_mappings--;
+ port->num_mappings--;
+ break;
+ case DLB_QUEUE_UNMAP_IN_PROG:
+ port->num_pending_removals++;
+ domain->num_pending_removals++;
+ break;
+ case DLB_QUEUE_MAPPED:
+ /* Priority change, nothing to update */
+ break;
+ default:
+ goto error;
+ }
+ break;
+ case DLB_QUEUE_MAP_IN_PROG:
+ switch (new_state) {
+ case DLB_QUEUE_UNMAPPED:
+ queue->num_pending_additions--;
+ domain->num_pending_additions--;
+ break;
+ case DLB_QUEUE_MAPPED:
+ queue->num_mappings++;
+ port->num_mappings++;
+ queue->num_pending_additions--;
+ domain->num_pending_additions--;
+ break;
+ default:
+ goto error;
+ }
+ break;
+ case DLB_QUEUE_UNMAP_IN_PROG:
+ switch (new_state) {
+ case DLB_QUEUE_UNMAPPED:
+ port->num_pending_removals--;
+ domain->num_pending_removals--;
+ queue->num_mappings--;
+ port->num_mappings--;
+ break;
+ case DLB_QUEUE_MAPPED:
+ port->num_pending_removals--;
+ domain->num_pending_removals--;
+ break;
+ case DLB_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+ /* Nothing to update */
+ break;
+ default:
+ goto error;
+ }
+ break;
+ case DLB_QUEUE_UNMAP_IN_PROG_PENDING_MAP:
+ switch (new_state) {
+ case DLB_QUEUE_UNMAP_IN_PROG:
+ /* Nothing to update */
+ break;
+ case DLB_QUEUE_UNMAPPED:
+ /*
+ * An UNMAP_IN_PROG_PENDING_MAP slot briefly
+ * becomes UNMAPPED before it transitions to
+ * MAP_IN_PROG.
+ */
+ queue->num_mappings--;
+ port->num_mappings--;
+ port->num_pending_removals--;
+ domain->num_pending_removals--;
+ break;
+ default:
+ goto error;
+ }
+ break;
+ default:
+ goto error;
+ }
+
+ port->qid_map[slot].state = new_state;
+
+ DLB_HW_DBG(hw,
+ "[%s()] queue %d -> port %d state transition (%d -> %d)\n",
+ __func__, queue->id.phys_id, port->id.phys_id,
+ curr_state, new_state);
+ return 0;
+
+error:
+ DLB_HW_ERR(hw,
+ "[%s()] Internal error: invalid queue %d -> port %d state transition (%d -> %d)\n",
+ __func__, queue->id.phys_id, port->id.phys_id,
+ curr_state, new_state);
+ return -EFAULT;
+}
+
+static int dlb_verify_map_qid_slot_available(struct dlb_ldb_port *port,
+ struct dlb_ldb_queue *queue,
+ struct dlb_cmd_response *resp)
+{
+ enum dlb_qid_map_state state;
+ int i;
+
+ /* Unused slot available? */
+ if (port->num_mappings < DLB_MAX_NUM_QIDS_PER_LDB_CQ)
+ return 0;
+
+ /*
+ * If the queue is already mapped (from the application's perspective),
+ * this is simply a priority update.
+ */
+ state = DLB_QUEUE_MAPPED;
+ if (dlb_port_find_slot_queue(port, state, queue, &i))
+ return 0;
+
+ state = DLB_QUEUE_MAP_IN_PROG;
+ if (dlb_port_find_slot_queue(port, state, queue, &i))
+ return 0;
+
+ if (dlb_port_find_slot_with_pending_map_queue(port, queue, &i))
+ return 0;
+
+ /*
+ * If the slot contains an unmap in progress, it's considered
+ * available.
+ */
+ state = DLB_QUEUE_UNMAP_IN_PROG;
+ if (dlb_port_find_slot(port, state, &i))
+ return 0;
+
+ state = DLB_QUEUE_UNMAPPED;
+ if (dlb_port_find_slot(port, state, &i))
+ return 0;
+
+ resp->status = DLB_ST_NO_QID_SLOTS_AVAILABLE;
+ return -EINVAL;
+}
+
static int dlb_verify_unmap_qid_args(struct dlb_hw *hw, u32 domain_id,
struct dlb_unmap_qid_args *args,
struct dlb_cmd_response *resp,
@@ -1343,9 +1562,11 @@ static int dlb_verify_unmap_qid_args(struct dlb_hw *hw, u32 domain_id,
struct dlb_ldb_port **out_port,
struct dlb_ldb_queue **out_queue)
{
+ enum dlb_qid_map_state state;
struct dlb_hw_domain *domain;
struct dlb_ldb_queue *queue;
struct dlb_ldb_port *port;
+ int slot;
int id;
domain = dlb_get_domain_from_id(hw, domain_id, vdev_req, vdev_id);
@@ -1383,6 +1604,26 @@ static int dlb_verify_unmap_qid_args(struct dlb_hw *hw, u32 domain_id,
return -EINVAL;
}
+ /*
+ * Verify that the port has the queue mapped. From the application's
+ * perspective a queue is mapped if it is actually mapped, the map is
+ * in progress, or the map is blocked pending an unmap.
+ */
+ state = DLB_QUEUE_MAPPED;
+ if (dlb_port_find_slot_queue(port, state, queue, &slot))
+ goto done;
+
+ state = DLB_QUEUE_MAP_IN_PROG;
+ if (dlb_port_find_slot_queue(port, state, queue, &slot))
+ goto done;
+
+ if (dlb_port_find_slot_with_pending_map_queue(port, queue, &slot))
+ goto done;
+
+ resp->status = DLB_ST_INVALID_QID;
+ return -EINVAL;
+
+done:
*out_domain = domain;
*out_port = port;
*out_queue = queue;
@@ -1881,6 +2122,21 @@ static int dlb_configure_dir_port(struct dlb_hw *hw, struct dlb_hw_domain *domai
return 0;
}
+static void dlb_ldb_port_change_qid_priority(struct dlb_hw *hw,
+ struct dlb_ldb_port *port, int slot,
+ struct dlb_map_qid_args *args)
+{
+ /* Placeholder */
+}
+
+static int dlb_ldb_port_map_qid(struct dlb_hw *hw, struct dlb_hw_domain *domain,
+ struct dlb_ldb_port *port,
+ struct dlb_ldb_queue *queue, u8 prio)
+{
+ /* Placeholder */
+ return 0;
+}
+
static void
dlb_log_create_sched_domain_args(struct dlb_hw *hw,
struct dlb_create_sched_domain_args *args,
@@ -2400,8 +2656,10 @@ int dlb_hw_map_qid(struct dlb_hw *hw, u32 domain_id,
{
struct dlb_hw_domain *domain;
struct dlb_ldb_queue *queue;
+ enum dlb_qid_map_state st;
struct dlb_ldb_port *port;
- int ret;
+ int ret, i;
+ u8 prio;
dlb_log_map_qid(hw, domain_id, args, vdev_req, vdev_id);
@@ -2414,6 +2672,124 @@ int dlb_hw_map_qid(struct dlb_hw *hw, u32 domain_id,
if (ret)
return ret;
+ prio = args->priority;
+
+ /*
+ * If there are any outstanding detach operations for this port,
+ * attempt to complete them. This may be necessary to free up a QID
+ * slot for this requested mapping.
+ */
+ ret = dlb_verify_map_qid_slot_available(port, queue, resp);
+ if (ret)
+ return ret;
+
+ /* Hardware requires disabling the CQ before mapping QIDs. */
+ if (port->enabled)
+ dlb_ldb_port_cq_disable(hw, port);
+
+ /*
+ * If this is only a priority change, don't perform the full QID->CQ
+ * mapping procedure
+ */
+ st = DLB_QUEUE_MAPPED;
+ if (dlb_port_find_slot_queue(port, st, queue, &i)) {
+ if (prio != port->qid_map[i].priority) {
+ dlb_ldb_port_change_qid_priority(hw, port, i, args);
+ DLB_HW_DBG(hw, "DLB map: priority change\n");
+ }
+
+ st = DLB_QUEUE_MAPPED;
+ ret = dlb_port_slot_state_transition(hw, port, queue, i, st);
+ if (ret)
+ return ret;
+
+ goto map_qid_done;
+ }
+
+ st = DLB_QUEUE_UNMAP_IN_PROG;
+ if (dlb_port_find_slot_queue(port, st, queue, &i)) {
+ if (prio != port->qid_map[i].priority) {
+ dlb_ldb_port_change_qid_priority(hw, port, i, args);
+ DLB_HW_DBG(hw, "DLB map: priority change\n");
+ }
+
+ st = DLB_QUEUE_MAPPED;
+ ret = dlb_port_slot_state_transition(hw, port, queue, i, st);
+ if (ret)
+ return ret;
+
+ goto map_qid_done;
+ }
+
+ /*
+ * If this is a priority change on an in-progress mapping, don't
+ * perform the full QID->CQ mapping procedure.
+ */
+ st = DLB_QUEUE_MAP_IN_PROG;
+ if (dlb_port_find_slot_queue(port, st, queue, &i)) {
+ port->qid_map[i].priority = prio;
+
+ DLB_HW_DBG(hw, "DLB map: priority change only\n");
+
+ goto map_qid_done;
+ }
+
+ /*
+ * If this is a priority change on a pending mapping, update the
+ * pending priority
+ */
+ if (dlb_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+ port->qid_map[i].pending_priority = prio;
+
+ DLB_HW_DBG(hw, "DLB map: priority change only\n");
+
+ goto map_qid_done;
+ }
+
+ /*
+ * If all the CQ's slots are in use, then there's an unmap in progress
+ * (guaranteed by dlb_verify_map_qid_slot_available()), so add this
+ * mapping to pending_map and return. When the removal is completed for
+ * the slot's current occupant, this mapping will be performed.
+ */
+ if (!dlb_port_find_slot(port, DLB_QUEUE_UNMAPPED, &i)) {
+ if (dlb_port_find_slot(port, DLB_QUEUE_UNMAP_IN_PROG, &i)) {
+ enum dlb_qid_map_state new_st;
+
+ port->qid_map[i].pending_qid = queue->id.phys_id;
+ port->qid_map[i].pending_priority = prio;
+
+ new_st = DLB_QUEUE_UNMAP_IN_PROG_PENDING_MAP;
+
+ ret = dlb_port_slot_state_transition(hw, port, queue,
+ i, new_st);
+ if (ret)
+ return ret;
+
+ DLB_HW_DBG(hw, "DLB map: map pending removal\n");
+
+ goto map_qid_done;
+ }
+ }
+
+ /*
+ * If the domain has started, a special "dynamic" CQ->queue mapping
+ * procedure is required in order to safely update the CQ<->QID tables.
+ * The "static" procedure cannot be used when traffic is flowing,
+ * because the CQ<->QID tables cannot be updated atomically and the
+ * scheduler won't see the new mapping unless the queue's if_status
+ * changes, which isn't guaranteed.
+ */
+ ret = dlb_ldb_port_map_qid(hw, domain, port, queue, prio);
+
+ /* If ret is less than zero, it's due to an internal error */
+ if (ret < 0)
+ return ret;
+
+map_qid_done:
+ if (port->enabled)
+ dlb_ldb_port_cq_enable(hw, port);
+
resp->status = 0;
return 0;
@@ -2475,8 +2851,9 @@ int dlb_hw_unmap_qid(struct dlb_hw *hw, u32 domain_id,
{
struct dlb_hw_domain *domain;
struct dlb_ldb_queue *queue;
+ enum dlb_qid_map_state st;
struct dlb_ldb_port *port;
- int ret;
+ int i, ret;
dlb_log_unmap_qid(hw, domain_id, args, vdev_req, vdev_id);
@@ -2489,6 +2866,58 @@ int dlb_hw_unmap_qid(struct dlb_hw *hw, u32 domain_id,
if (ret)
return ret;
+ /*
+ * If the queue hasn't been mapped yet, we need to update the slot's
+ * state and re-enable the queue's inflights.
+ */
+ st = DLB_QUEUE_MAP_IN_PROG;
+ if (dlb_port_find_slot_queue(port, st, queue, &i)) {
+ st = DLB_QUEUE_UNMAPPED;
+ ret = dlb_port_slot_state_transition(hw, port, queue, i, st);
+ if (ret)
+ return ret;
+
+ goto unmap_qid_done;
+ }
+
+ /*
+ * If the queue mapping is on hold pending an unmap, we simply need to
+ * update the slot's state.
+ */
+ if (dlb_port_find_slot_with_pending_map_queue(port, queue, &i)) {
+ st = DLB_QUEUE_UNMAP_IN_PROG;
+ ret = dlb_port_slot_state_transition(hw, port, queue, i, st);
+ if (ret)
+ return ret;
+
+ goto unmap_qid_done;
+ }
+
+ st = DLB_QUEUE_MAPPED;
+ if (!dlb_port_find_slot_queue(port, st, queue, &i)) {
+ DLB_HW_ERR(hw,
+ "[%s()] Internal error: no available CQ slots\n",
+ __func__);
+ return -EFAULT;
+ }
+
+ /*
+ * QID->CQ mapping removal is an asychronous procedure. It requires
+ * stopping the DLB from scheduling this CQ, draining all inflights
+ * from the CQ, then unmapping the queue from the CQ. This function
+ * simply marks the port as needing the queue unmapped, and (if
+ * necessary) starts the unmapping worker thread.
+ */
+ dlb_ldb_port_cq_disable(hw, port);
+
+ st = DLB_QUEUE_UNMAP_IN_PROG;
+ ret = dlb_port_slot_state_transition(hw, port, queue, i, st);
+ if (ret)
+ return ret;
+
+unmap_qid_done:
+ resp->status = 0;
+
return 0;
}
--
2.17.1
Introduce the dlb device ioctl layer and the first three ioctls: query
device version, query available resources, and create a scheduling domain.
Also introduce the user-space interface file dlb_user.h.
The device version query is designed to allow each DLB device version/type
to have its own unique ioctl API through the /dev/dlb%d node. Each such API
would share in common the device version command as its first command, and
all subsequent commands can be unique to the particular device.
The hardware operation for scheduling domain creation will be added in a
subsequent commit.
Signed-off-by: Gage Eads <[email protected]>
Signed-off-by: Mike Ximing Chen <[email protected]>
Reviewed-by: Magnus Karlsson <[email protected]>
Reviewed-by: Dan Williams <[email protected]>
---
.../userspace-api/ioctl/ioctl-number.rst | 1 +
drivers/misc/dlb/Makefile | 2 +-
drivers/misc/dlb/dlb_bitmap.h | 32 ++++
drivers/misc/dlb/dlb_ioctl.c | 103 +++++++++++
drivers/misc/dlb/dlb_main.c | 1 +
drivers/misc/dlb/dlb_main.h | 10 ++
drivers/misc/dlb/dlb_pf_ops.c | 22 +++
drivers/misc/dlb/dlb_resource.c | 62 +++++++
drivers/misc/dlb/dlb_resource.h | 4 +
include/uapi/linux/dlb.h | 167 ++++++++++++++++++
10 files changed, 403 insertions(+), 1 deletion(-)
create mode 100644 drivers/misc/dlb/dlb_ioctl.c
create mode 100644 include/uapi/linux/dlb.h
diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst
index a4c75a28c839..747b48b141c8 100644
--- a/Documentation/userspace-api/ioctl/ioctl-number.rst
+++ b/Documentation/userspace-api/ioctl/ioctl-number.rst
@@ -300,6 +300,7 @@ Code Seq# Include File Comments
'z' 10-4F drivers/s390/crypto/zcrypt_api.h conflict!
'|' 00-7F linux/media.h
0x80 00-1F linux/fb.h
+0x81 00-1F uapi/linux/dlb.h
0x89 00-06 arch/x86/include/asm/sockios.h
0x89 0B-DF linux/sockios.h
0x89 E0-EF linux/sockios.h SIOCPROTOPRIVATE range
diff --git a/drivers/misc/dlb/Makefile b/drivers/misc/dlb/Makefile
index 8a49ea5fd752..aaafb3086d8d 100644
--- a/drivers/misc/dlb/Makefile
+++ b/drivers/misc/dlb/Makefile
@@ -7,4 +7,4 @@
obj-$(CONFIG_INTEL_DLB) := dlb.o
dlb-objs := dlb_main.o
-dlb-objs += dlb_pf_ops.o dlb_resource.o
+dlb-objs += dlb_pf_ops.o dlb_resource.o dlb_ioctl.o
diff --git a/drivers/misc/dlb/dlb_bitmap.h b/drivers/misc/dlb/dlb_bitmap.h
index fb3ef52a306d..3ea78b42c79f 100644
--- a/drivers/misc/dlb/dlb_bitmap.h
+++ b/drivers/misc/dlb/dlb_bitmap.h
@@ -73,4 +73,36 @@ static inline void dlb_bitmap_free(struct dlb_bitmap *bitmap)
kfree(bitmap);
}
+/**
+ * dlb_bitmap_longest_set_range() - returns longest contiguous range of set
+ * bits
+ * @bitmap: pointer to dlb_bitmap structure.
+ *
+ * Return:
+ * Returns the bitmap's longest contiguous range of set bits upon success,
+ * <0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized.
+ */
+static inline int dlb_bitmap_longest_set_range(struct dlb_bitmap *bitmap)
+{
+ int max_len, len;
+ int start, end;
+
+ if (!bitmap || !bitmap->map)
+ return -EINVAL;
+
+ if (bitmap_weight(bitmap->map, bitmap->len) == 0)
+ return 0;
+
+ max_len = 0;
+ bitmap_for_each_set_region(bitmap->map, start, end, 0, bitmap->len) {
+ len = end - start;
+ if (max_len < len)
+ max_len = len;
+ }
+ return max_len;
+}
+
#endif /* __DLB_OSDEP_BITMAP_H */
diff --git a/drivers/misc/dlb/dlb_ioctl.c b/drivers/misc/dlb/dlb_ioctl.c
new file mode 100644
index 000000000000..47d6cab773d4
--- /dev/null
+++ b/drivers/misc/dlb/dlb_ioctl.c
@@ -0,0 +1,103 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#include <linux/uaccess.h>
+
+#include <uapi/linux/dlb.h>
+
+#include "dlb_main.h"
+
+/* [7:0]: device revision, [15:8]: device version */
+#define DLB_SET_DEVICE_VERSION(ver, rev) (((ver) << 8) | (rev))
+
+static int dlb_ioctl_get_device_version(unsigned long user_arg)
+{
+ struct dlb_get_device_version_args arg;
+ u8 revision;
+
+ switch (boot_cpu_data.x86_stepping) {
+ case 0:
+ revision = DLB_REV_A0;
+ break;
+ case 1:
+ revision = DLB_REV_A1;
+ break;
+ case 2:
+ revision = DLB_REV_A2;
+ break;
+ default:
+ /* Treat all revisions >= 3 as B0 */
+ revision = DLB_REV_B0;
+ break;
+ }
+
+ arg.response.status = 0;
+ arg.response.id = DLB_SET_DEVICE_VERSION(2, revision);
+
+ if (copy_to_user((void __user *)user_arg, &arg, sizeof(arg)))
+ return -EFAULT;
+
+ return 0;
+}
+
+static int dlb_ioctl_create_sched_domain(struct dlb *dlb, unsigned long user_arg)
+{
+ struct dlb_create_sched_domain_args __user *uarg;
+ struct dlb_create_sched_domain_args arg;
+ struct dlb_cmd_response response = {0};
+ int ret;
+
+ uarg = (void __user *)user_arg;
+ if (copy_from_user(&arg, uarg, sizeof(arg)))
+ return -EFAULT;
+
+ mutex_lock(&dlb->resource_mutex);
+
+ ret = dlb->ops->create_sched_domain(&dlb->hw, &arg, &response);
+
+ mutex_unlock(&dlb->resource_mutex);
+
+ response.status = ret;
+ BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
+
+ if (copy_to_user((void __user *)&uarg->response, &response, sizeof(response)))
+ return -EFAULT;
+
+ return ret;
+}
+
+static int dlb_ioctl_get_num_resources(struct dlb *dlb, unsigned long user_arg)
+{
+ struct dlb_get_num_resources_args arg = {0};
+ int ret;
+
+ mutex_lock(&dlb->resource_mutex);
+
+ ret = dlb->ops->get_num_resources(&dlb->hw, &arg);
+
+ mutex_unlock(&dlb->resource_mutex);
+
+ BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
+ arg.response.status = ret;
+
+ if (copy_to_user((void __user *)user_arg, &arg, sizeof(arg)))
+ return -EFAULT;
+
+ return ret;
+}
+
+long dlb_ioctl(struct file *f, unsigned int cmd, unsigned long arg)
+{
+ struct dlb *dlb = f->private_data;
+
+ switch (cmd) {
+ case DLB_IOC_GET_DEVICE_VERSION:
+ return dlb_ioctl_get_device_version(arg);
+ case DLB_IOC_CREATE_SCHED_DOMAIN:
+ return dlb_ioctl_create_sched_domain(dlb, arg);
+ case DLB_IOC_GET_NUM_RESOURCES:
+ return dlb_ioctl_get_num_resources(dlb, arg);
+ default:
+ return -ENOTTY;
+ }
+}
diff --git a/drivers/misc/dlb/dlb_main.c b/drivers/misc/dlb/dlb_main.c
index 12707b23ab3e..d92956b1643d 100644
--- a/drivers/misc/dlb/dlb_main.c
+++ b/drivers/misc/dlb/dlb_main.c
@@ -64,6 +64,7 @@ static int dlb_device_create(struct dlb *dlb, struct pci_dev *pdev)
static const struct file_operations dlb_fops = {
.owner = THIS_MODULE,
+ .unlocked_ioctl = dlb_ioctl,
};
/**********************************/
diff --git a/drivers/misc/dlb/dlb_main.h b/drivers/misc/dlb/dlb_main.h
index ec5eb7bd8f54..3089a66a3560 100644
--- a/drivers/misc/dlb/dlb_main.h
+++ b/drivers/misc/dlb/dlb_main.h
@@ -12,6 +12,8 @@
#include <linux/pci.h>
#include <linux/types.h>
+#include <uapi/linux/dlb.h>
+
#include "dlb_hw_types.h"
/*
@@ -37,6 +39,11 @@ struct dlb_device_ops {
int (*init_driver_state)(struct dlb *dlb);
void (*enable_pm)(struct dlb *dlb);
int (*wait_for_device_ready)(struct dlb *dlb, struct pci_dev *pdev);
+ int (*create_sched_domain)(struct dlb_hw *hw,
+ struct dlb_create_sched_domain_args *args,
+ struct dlb_cmd_response *resp);
+ int (*get_num_resources)(struct dlb_hw *hw,
+ struct dlb_get_num_resources_args *args);
};
extern struct dlb_device_ops dlb_pf_ops;
@@ -56,4 +63,7 @@ struct dlb {
dev_t dev_number;
};
+/* Prototypes for dlb_ioctl.c */
+long dlb_ioctl(struct file *f, unsigned int cmd, unsigned long arg);
+
#endif /* __DLB_MAIN_H */
diff --git a/drivers/misc/dlb/dlb_pf_ops.c b/drivers/misc/dlb/dlb_pf_ops.c
index 124b4fee8564..125ef6fe6c70 100644
--- a/drivers/misc/dlb/dlb_pf_ops.c
+++ b/drivers/misc/dlb/dlb_pf_ops.c
@@ -95,6 +95,26 @@ static int dlb_pf_wait_for_device_ready(struct dlb *dlb, struct pci_dev *pdev)
return 0;
}
+/*****************************/
+/****** IOCTL callbacks ******/
+/*****************************/
+
+static int dlb_pf_create_sched_domain(struct dlb_hw *hw,
+ struct dlb_create_sched_domain_args *args,
+ struct dlb_cmd_response *resp)
+{
+ resp->id = 0;
+ resp->status = 0;
+
+ return 0;
+}
+
+static int dlb_pf_get_num_resources(struct dlb_hw *hw,
+ struct dlb_get_num_resources_args *args)
+{
+ return dlb_hw_get_num_resources(hw, args, false, 0);
+}
+
/********************************/
/****** DLB PF Device Ops ******/
/********************************/
@@ -105,4 +125,6 @@ struct dlb_device_ops dlb_pf_ops = {
.init_driver_state = dlb_pf_init_driver_state,
.enable_pm = dlb_pf_enable_pm,
.wait_for_device_ready = dlb_pf_wait_for_device_ready,
+ .create_sched_domain = dlb_pf_create_sched_domain,
+ .get_num_resources = dlb_pf_get_num_resources,
};
diff --git a/drivers/misc/dlb/dlb_resource.c b/drivers/misc/dlb/dlb_resource.c
index fca444c46aca..9d75b12eb793 100644
--- a/drivers/misc/dlb/dlb_resource.c
+++ b/drivers/misc/dlb/dlb_resource.c
@@ -200,6 +200,68 @@ int dlb_resource_init(struct dlb_hw *hw)
return ret;
}
+/**
+ * dlb_hw_get_num_resources() - query the PCI function's available resources
+ * @hw: dlb_hw handle for a particular device.
+ * @arg: pointer to resource counts.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_req is true, this contains the vdev's ID.
+ *
+ * This function returns the number of available resources for the PF or for a
+ * VF.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, -EINVAL if vdev_req is true and vdev_id is
+ * invalid.
+ */
+int dlb_hw_get_num_resources(struct dlb_hw *hw,
+ struct dlb_get_num_resources_args *arg,
+ bool vdev_req, unsigned int vdev_id)
+{
+ struct dlb_function_resources *rsrcs;
+ struct dlb_bitmap *map;
+ int i;
+
+ if (vdev_req && vdev_id >= DLB_MAX_NUM_VDEVS)
+ return -EINVAL;
+
+ if (vdev_req)
+ rsrcs = &hw->vdev[vdev_id];
+ else
+ rsrcs = &hw->pf;
+
+ arg->num_sched_domains = rsrcs->num_avail_domains;
+
+ arg->num_ldb_queues = rsrcs->num_avail_ldb_queues;
+
+ arg->num_ldb_ports = 0;
+ for (i = 0; i < DLB_NUM_COS_DOMAINS; i++)
+ arg->num_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+ for (i = 0; i < DLB_NUM_COS_DOMAINS; i++)
+ arg->num_cos_ldb_ports[i] = rsrcs->num_avail_ldb_ports[i];
+
+ arg->num_dir_ports = rsrcs->num_avail_dir_pq_pairs;
+
+ arg->num_atomic_inflights = rsrcs->num_avail_aqed_entries;
+
+ map = rsrcs->avail_hist_list_entries;
+
+ arg->num_hist_list_entries = bitmap_weight(map->map, map->len);
+
+ arg->max_contiguous_hist_list_entries =
+ dlb_bitmap_longest_set_range(map);
+
+ arg->num_ldb_credits = rsrcs->num_avail_qed_entries;
+
+ arg->num_dir_credits = rsrcs->num_avail_dqed_entries;
+
+ return 0;
+}
+
/**
* dlb_clr_pmcsr_disable() - power on bulk of DLB 2.0 logic
* @hw: dlb_hw handle for a particular device.
diff --git a/drivers/misc/dlb/dlb_resource.h b/drivers/misc/dlb/dlb_resource.h
index 2229813d9c45..3e6d419796bc 100644
--- a/drivers/misc/dlb/dlb_resource.h
+++ b/drivers/misc/dlb/dlb_resource.h
@@ -12,6 +12,10 @@ int dlb_resource_init(struct dlb_hw *hw);
void dlb_resource_free(struct dlb_hw *hw);
+int dlb_hw_get_num_resources(struct dlb_hw *hw,
+ struct dlb_get_num_resources_args *arg,
+ bool vdev_req, unsigned int vdev_id);
+
void dlb_clr_pmcsr_disable(struct dlb_hw *hw);
#endif /* __DLB_RESOURCE_H */
diff --git a/include/uapi/linux/dlb.h b/include/uapi/linux/dlb.h
new file mode 100644
index 000000000000..b31c67de7fb4
--- /dev/null
+++ b/include/uapi/linux/dlb.h
@@ -0,0 +1,167 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#ifndef __DLB_H
+#define __DLB_H
+
+#include <linux/types.h>
+
+struct dlb_cmd_response {
+ __u32 status; /* Interpret using enum dlb_error */
+ __u32 id;
+};
+
+/********************************/
+/* 'dlb' device file commands */
+/********************************/
+
+#define DLB_DEVICE_VERSION(x) (((x) >> 8) & 0xFF)
+#define DLB_DEVICE_REVISION(x) ((x) & 0xFF)
+
+enum dlb_revisions {
+ DLB_REV_A0 = 0,
+ DLB_REV_A1,
+ DLB_REV_A2,
+ DLB_REV_B0,
+};
+
+/*
+ * DLB_CMD_GET_DEVICE_VERSION: Query the DLB device version.
+ *
+ * All DLB device versions have the same ioctl API. Each version may have
+ * different resource and feature set. The device revision is provided
+ * in case of any hardware errata.
+ *
+ * Output parameters:
+ * @response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ * @response.id[7:0]: Device revision.
+ * @response.id[15:8]: Device version.
+ */
+
+struct dlb_get_device_version_args {
+ /* Output parameters */
+ struct dlb_cmd_response response;
+};
+
+/*
+ * DLB_CMD_CREATE_SCHED_DOMAIN: Create a DLB 2.0 scheduling domain and reserve
+ * its hardware resources. This command returns the newly created domain
+ * ID and a file descriptor for accessing the domain.
+ *
+ * Output parameters:
+ * @response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ * @response.id: domain ID.
+ * @domain_fd: file descriptor for performing the domain's ioctl operations
+ * @padding0: Reserved for future use.
+ *
+ * Input parameters:
+ * @num_ldb_queues: Number of load-balanced queues.
+ * @num_ldb_ports: Number of load-balanced ports that can be allocated from
+ * any class-of-service with available ports.
+ * @num_cos_ldb_ports[4]: Number of load-balanced ports from
+ * classes-of-service 0-3.
+ * @num_dir_ports: Number of directed ports. A directed port has one directed
+ * queue, so no num_dir_queues argument is necessary.
+ * @num_atomic_inflights: This specifies the amount of temporary atomic QE
+ * storage for the domain. This storage is divided among the domain's
+ * load-balanced queues that are configured for atomic scheduling.
+ * @num_hist_list_entries: Amount of history list storage. This is divided
+ * among the domain's CQs.
+ * @num_ldb_credits: Amount of load-balanced QE storage (QED). QEs occupy this
+ * space until they are scheduled to a load-balanced CQ. One credit
+ * represents the storage for one QE.
+ * @num_dir_credits: Amount of directed QE storage (DQED). QEs occupy this
+ * space until they are scheduled to a directed CQ. One credit represents
+ * the storage for one QE.
+ * @cos_strict: If set, return an error if there are insufficient ports in
+ * class-of-service N to satisfy the num_ldb_ports_cosN argument. If
+ * unset, attempt to fulfill num_ldb_ports_cosN arguments from other
+ * classes-of-service if class N does not contain enough free ports.
+ * @padding1: Reserved for future use.
+ */
+struct dlb_create_sched_domain_args {
+ /* Output parameters */
+ struct dlb_cmd_response response;
+ __u32 domain_fd;
+ __u32 padding0;
+ /* Input parameters */
+ __u32 num_ldb_queues;
+ __u32 num_ldb_ports;
+ __u32 num_cos_ldb_ports[4];
+ __u32 num_dir_ports;
+ __u32 num_atomic_inflights;
+ __u32 num_hist_list_entries;
+ __u32 num_ldb_credits;
+ __u32 num_dir_credits;
+ __u8 cos_strict;
+ __u8 padding1[3];
+};
+
+/*
+ * DLB_CMD_GET_NUM_RESOURCES: Return the number of available resources
+ * (queues, ports, etc.) that this device owns.
+ *
+ * Output parameters:
+ * @response.status: Detailed error code. In certain cases, such as if the
+ * ioctl request arg is invalid, the driver won't set status.
+ * @num_domains: Number of available scheduling domains.
+ * @num_ldb_queues: Number of available load-balanced queues.
+ * @num_ldb_ports: Total number of available load-balanced ports.
+ * @num_cos_ldb_ports[4]: Number of available load-balanced ports from
+ * classes-of-service 0-3.
+ * @num_dir_ports: Number of available directed ports. There is one directed
+ * queue for every directed port.
+ * @num_atomic_inflights: Amount of available temporary atomic QE storage.
+ * @num_hist_list_entries: Amount of history list storage.
+ * @max_contiguous_hist_list_entries: History list storage is allocated in
+ * a contiguous chunk, and this return value is the longest available
+ * contiguous range of history list entries.
+ * @num_ldb_credits: Amount of available load-balanced QE storage.
+ * @num_dir_credits: Amount of available directed QE storage.
+ */
+struct dlb_get_num_resources_args {
+ /* Output parameters */
+ struct dlb_cmd_response response;
+ __u32 num_sched_domains;
+ __u32 num_ldb_queues;
+ __u32 num_ldb_ports;
+ __u32 num_cos_ldb_ports[4];
+ __u32 num_dir_ports;
+ __u32 num_atomic_inflights;
+ __u32 num_hist_list_entries;
+ __u32 max_contiguous_hist_list_entries;
+ __u32 num_ldb_credits;
+ __u32 num_dir_credits;
+};
+
+enum dlb_user_interface_commands {
+ DLB_CMD_GET_DEVICE_VERSION,
+ DLB_CMD_CREATE_SCHED_DOMAIN,
+ DLB_CMD_GET_NUM_RESOURCES,
+
+ /* NUM_DLB_CMD must be last */
+ NUM_DLB_CMD,
+};
+
+/********************/
+/* dlb ioctl codes */
+/********************/
+
+#define DLB_IOC_MAGIC 0x81
+
+#define DLB_IOC_GET_DEVICE_VERSION \
+ _IOR(DLB_IOC_MAGIC, \
+ DLB_CMD_GET_DEVICE_VERSION, \
+ struct dlb_get_device_version_args)
+#define DLB_IOC_CREATE_SCHED_DOMAIN \
+ _IOWR(DLB_IOC_MAGIC, \
+ DLB_CMD_CREATE_SCHED_DOMAIN, \
+ struct dlb_create_sched_domain_args)
+#define DLB_IOC_GET_NUM_RESOURCES \
+ _IOR(DLB_IOC_MAGIC, \
+ DLB_CMD_GET_NUM_RESOURCES, \
+ struct dlb_get_num_resources_args)
+
+#endif /* __DLB_H */
--
2.17.1
Adds the "dynamic" map procedure and register operations. If a queue map
is requested after the domain is started, the driver must disable the
requested queue and wait for it to quiesce before mapping it to the
requested port.
Signed-off-by: Gage Eads <[email protected]>
Signed-off-by: Mike Ximing Chen <[email protected]>
Reviewed-by: Björn Töpel <[email protected]>
Reviewed-by: Dan Williams <[email protected]>
---
drivers/misc/dlb/dlb_resource.c | 393 +++++++++++++++++++++++++++++++-
1 file changed, 392 insertions(+), 1 deletion(-)
diff --git a/drivers/misc/dlb/dlb_resource.c b/drivers/misc/dlb/dlb_resource.c
index 95ccb7eddb8b..93a3de642024 100644
--- a/drivers/misc/dlb/dlb_resource.c
+++ b/drivers/misc/dlb/dlb_resource.c
@@ -328,6 +328,37 @@ dlb_get_domain_dir_pq(u32 id, bool vdev_req, struct dlb_hw_domain *domain)
return NULL;
}
+static struct dlb_ldb_queue *
+dlb_get_ldb_queue_from_id(struct dlb_hw *hw, u32 id, bool vdev_req,
+ unsigned int vdev_id)
+{
+ struct dlb_function_resources *rsrcs;
+ struct dlb_hw_domain *domain;
+ struct dlb_ldb_queue *queue;
+
+ if (id >= DLB_MAX_NUM_LDB_QUEUES)
+ return NULL;
+
+ rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+ if (!vdev_req)
+ return &hw->rsrcs.ldb_queues[id];
+
+ list_for_each_entry(domain, &rsrcs->used_domains, func_list) {
+ list_for_each_entry(queue, &domain->used_ldb_queues, domain_list) {
+ if (queue->id.virt_id == id)
+ return queue;
+ }
+ }
+
+ list_for_each_entry(queue, &rsrcs->avail_ldb_queues, func_list) {
+ if (queue->id.virt_id == id)
+ return queue;
+ }
+
+ return NULL;
+}
+
static struct dlb_ldb_queue *
dlb_get_domain_ldb_queue(u32 id, bool vdev_req, struct dlb_hw_domain *domain)
{
@@ -2251,6 +2282,75 @@ static void dlb_ldb_port_change_qid_priority(struct dlb_hw *hw,
port->qid_map[slot].priority = args->priority;
}
+static int dlb_ldb_port_set_has_work_bits(struct dlb_hw *hw,
+ struct dlb_ldb_port *port,
+ struct dlb_ldb_queue *queue, int slot)
+{
+ u32 ctrl = 0;
+ u32 active;
+ u32 enq;
+
+ /* Set the atomic scheduling haswork bit */
+ active = DLB_CSR_RD(hw, LSP_QID_AQED_ACTIVE_CNT(queue->id.phys_id));
+
+ BITS_SET(ctrl, port->id.phys_id, LSP_LDB_SCHED_CTRL_CQ);
+ BITS_SET(ctrl, slot, LSP_LDB_SCHED_CTRL_QIDIX);
+ ctrl |= LSP_LDB_SCHED_CTRL_VALUE;
+ BITS_SET(ctrl, (u32)(BITS_GET(active, LSP_QID_AQED_ACTIVE_CNT_COUNT) > 0),
+ LSP_LDB_SCHED_CTRL_RLIST_HASWORK_V);
+
+ /* Set the non-atomic scheduling haswork bit */
+ DLB_CSR_WR(hw, LSP_LDB_SCHED_CTRL, ctrl);
+
+ enq = DLB_CSR_RD(hw,
+ LSP_QID_LDB_ENQUEUE_CNT(queue->id.phys_id));
+
+ memset(&ctrl, 0, sizeof(ctrl));
+
+ BITS_SET(ctrl, port->id.phys_id, LSP_LDB_SCHED_CTRL_CQ);
+ BITS_SET(ctrl, slot, LSP_LDB_SCHED_CTRL_QIDIX);
+ ctrl |= LSP_LDB_SCHED_CTRL_VALUE;
+ BITS_SET(ctrl, (u32)(BITS_GET(enq, LSP_QID_LDB_ENQUEUE_CNT_COUNT) > 0),
+ LSP_LDB_SCHED_CTRL_NALB_HASWORK_V);
+
+ DLB_CSR_WR(hw, LSP_LDB_SCHED_CTRL, ctrl);
+
+ dlb_flush_csr(hw);
+
+ return 0;
+}
+
+static void dlb_ldb_port_clear_queue_if_status(struct dlb_hw *hw,
+ struct dlb_ldb_port *port,
+ int slot)
+{
+ u32 ctrl = 0;
+
+ BITS_SET(ctrl, port->id.phys_id, LSP_LDB_SCHED_CTRL_CQ);
+ BITS_SET(ctrl, slot, LSP_LDB_SCHED_CTRL_QIDIX);
+ ctrl |= LSP_LDB_SCHED_CTRL_INFLIGHT_OK_V;
+
+ DLB_CSR_WR(hw, LSP_LDB_SCHED_CTRL, ctrl);
+
+ dlb_flush_csr(hw);
+}
+
+static void dlb_ldb_port_set_queue_if_status(struct dlb_hw *hw,
+ struct dlb_ldb_port *port,
+ int slot)
+{
+ u32 ctrl = 0;
+
+ BITS_SET(ctrl, port->id.phys_id, LSP_LDB_SCHED_CTRL_CQ);
+ BITS_SET(ctrl, slot, LSP_LDB_SCHED_CTRL_QIDIX);
+ ctrl |= LSP_LDB_SCHED_CTRL_VALUE;
+ ctrl |= LSP_LDB_SCHED_CTRL_INFLIGHT_OK_V;
+
+ DLB_CSR_WR(hw, LSP_LDB_SCHED_CTRL, ctrl);
+
+ dlb_flush_csr(hw);
+}
+
static void dlb_ldb_queue_set_inflight_limit(struct dlb_hw *hw,
struct dlb_ldb_queue *queue)
{
@@ -2261,11 +2361,222 @@ static void dlb_ldb_queue_set_inflight_limit(struct dlb_hw *hw,
DLB_CSR_WR(hw, LSP_QID_LDB_INFL_LIM(queue->id.phys_id), infl_lim);
}
+static void dlb_ldb_queue_clear_inflight_limit(struct dlb_hw *hw,
+ struct dlb_ldb_queue *queue)
+{
+ DLB_CSR_WR(hw,
+ LSP_QID_LDB_INFL_LIM(queue->id.phys_id),
+ LSP_QID_LDB_INFL_LIM_RST);
+}
+
+/*
+ * dlb_ldb_queue_{enable, disable}_mapped_cqs() don't operate exactly as
+ * their function names imply, and should only be called by the dynamic CQ
+ * mapping code.
+ */
+static void dlb_ldb_queue_disable_mapped_cqs(struct dlb_hw *hw,
+ struct dlb_hw_domain *domain,
+ struct dlb_ldb_queue *queue)
+{
+ struct dlb_ldb_port *port;
+ int slot, i;
+
+ for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+ list_for_each_entry(port, &domain->used_ldb_ports[i], domain_list) {
+ enum dlb_qid_map_state state = DLB_QUEUE_MAPPED;
+
+ if (!dlb_port_find_slot_queue(port, state,
+ queue, &slot))
+ continue;
+
+ if (port->enabled)
+ dlb_ldb_port_cq_disable(hw, port);
+ }
+ }
+}
+
+static void dlb_ldb_queue_enable_mapped_cqs(struct dlb_hw *hw,
+ struct dlb_hw_domain *domain,
+ struct dlb_ldb_queue *queue)
+{
+ struct dlb_ldb_port *port;
+ int slot, i;
+
+ for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+ list_for_each_entry(port, &domain->used_ldb_ports[i], domain_list) {
+ enum dlb_qid_map_state state = DLB_QUEUE_MAPPED;
+
+ if (!dlb_port_find_slot_queue(port, state,
+ queue, &slot))
+ continue;
+
+ if (port->enabled)
+ dlb_ldb_port_cq_enable(hw, port);
+ }
+ }
+}
+
+static int dlb_ldb_port_finish_map_qid_dynamic(struct dlb_hw *hw,
+ struct dlb_hw_domain *domain,
+ struct dlb_ldb_port *port,
+ struct dlb_ldb_queue *queue)
+{
+ enum dlb_qid_map_state state;
+ int slot, ret, i;
+ u32 infl_cnt;
+ u8 prio;
+
+ infl_cnt = DLB_CSR_RD(hw, LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+ if (BITS_GET(infl_cnt, LSP_QID_LDB_INFL_CNT_COUNT)) {
+ DLB_HW_ERR(hw,
+ "[%s()] Internal error: non-zero QID inflight count\n",
+ __func__);
+ return -EINVAL;
+ }
+
+ /*
+ * Static map the port and set its corresponding has_work bits.
+ */
+ state = DLB_QUEUE_MAP_IN_PROG;
+ if (!dlb_port_find_slot_queue(port, state, queue, &slot))
+ return -EINVAL;
+
+ prio = port->qid_map[slot].priority;
+
+ /*
+ * Update the CQ2QID, CQ2PRIOV, and QID2CQIDX registers, and
+ * the port's qid_map state.
+ */
+ ret = dlb_ldb_port_map_qid_static(hw, port, queue, prio);
+ if (ret)
+ return ret;
+
+ ret = dlb_ldb_port_set_has_work_bits(hw, port, queue, slot);
+ if (ret)
+ return ret;
+
+ /*
+ * Ensure IF_status(cq,qid) is 0 before enabling the port to
+ * prevent spurious schedules to cause the queue's inflight
+ * count to increase.
+ */
+ dlb_ldb_port_clear_queue_if_status(hw, port, slot);
+
+ /* Reset the queue's inflight status */
+ for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+ list_for_each_entry(port, &domain->used_ldb_ports[i], domain_list) {
+ state = DLB_QUEUE_MAPPED;
+ if (!dlb_port_find_slot_queue(port, state,
+ queue, &slot))
+ continue;
+
+ dlb_ldb_port_set_queue_if_status(hw, port, slot);
+ }
+ }
+
+ dlb_ldb_queue_set_inflight_limit(hw, queue);
+
+ /* Re-enable CQs mapped to this queue */
+ dlb_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+ /* If this queue has other mappings pending, clear its inflight limit */
+ if (queue->num_pending_additions > 0)
+ dlb_ldb_queue_clear_inflight_limit(hw, queue);
+
+ return 0;
+}
+
+/**
+ * dlb_ldb_port_map_qid_dynamic() - perform a "dynamic" QID->CQ mapping
+ * @hw: dlb_hw handle for a particular device.
+ * @port: load-balanced port
+ * @queue: load-balanced queue
+ * @priority: queue servicing priority
+ *
+ * Returns 0 if the queue was mapped, 1 if the mapping is scheduled to occur
+ * at a later point, and <0 if an error occurred.
+ */
+static int dlb_ldb_port_map_qid_dynamic(struct dlb_hw *hw,
+ struct dlb_ldb_port *port,
+ struct dlb_ldb_queue *queue,
+ u8 priority)
+{
+ enum dlb_qid_map_state state;
+ struct dlb_hw_domain *domain;
+ int domain_id, slot, ret;
+ u32 infl_cnt;
+
+ domain_id = port->domain_id.phys_id;
+
+ domain = dlb_get_domain_from_id(hw, domain_id, false, 0);
+ if (!domain) {
+ DLB_HW_ERR(hw,
+ "[%s()] Internal error: unable to find domain %d\n",
+ __func__, port->domain_id.phys_id);
+ return -EINVAL;
+ }
+
+ /*
+ * Set the QID inflight limit to 0 to prevent further scheduling of the
+ * queue.
+ */
+ DLB_CSR_WR(hw, LSP_QID_LDB_INFL_LIM(queue->id.phys_id), 0);
+
+ if (!dlb_port_find_slot(port, DLB_QUEUE_UNMAPPED, &slot)) {
+ DLB_HW_ERR(hw,
+ "Internal error: No available unmapped slots\n");
+ return -EFAULT;
+ }
+
+ port->qid_map[slot].qid = queue->id.phys_id;
+ port->qid_map[slot].priority = priority;
+
+ state = DLB_QUEUE_MAP_IN_PROG;
+ ret = dlb_port_slot_state_transition(hw, port, queue, slot, state);
+ if (ret)
+ return ret;
+
+ infl_cnt = DLB_CSR_RD(hw, LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+ if (BITS_GET(infl_cnt, LSP_QID_LDB_INFL_CNT_COUNT))
+ return 1;
+
+ /*
+ * Disable the affected CQ, and the CQs already mapped to the QID,
+ * before reading the QID's inflight count a second time. There is an
+ * unlikely race in which the QID may schedule one more QE after we
+ * read an inflight count of 0, and disabling the CQs guarantees that
+ * the race will not occur after a re-read of the inflight count
+ * register.
+ */
+ if (port->enabled)
+ dlb_ldb_port_cq_disable(hw, port);
+
+ dlb_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+ infl_cnt = DLB_CSR_RD(hw, LSP_QID_LDB_INFL_CNT(queue->id.phys_id));
+
+ if (BITS_GET(infl_cnt, LSP_QID_LDB_INFL_CNT_COUNT)) {
+ if (port->enabled)
+ dlb_ldb_port_cq_enable(hw, port);
+
+ dlb_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+ return 1;
+ }
+
+ return dlb_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+}
+
static int dlb_ldb_port_map_qid(struct dlb_hw *hw, struct dlb_hw_domain *domain,
struct dlb_ldb_port *port,
struct dlb_ldb_queue *queue, u8 prio)
{
- return dlb_ldb_port_map_qid_static(hw, port, queue, prio);
+ if (domain->started)
+ return dlb_ldb_port_map_qid_dynamic(hw, port, queue, prio);
+ else
+ return dlb_ldb_port_map_qid_static(hw, port, queue, prio);
}
static void
@@ -2722,6 +3033,82 @@ int dlb_hw_create_dir_port(struct dlb_hw *hw, u32 domain_id,
return 0;
}
+static void dlb_domain_finish_map_port(struct dlb_hw *hw,
+ struct dlb_hw_domain *domain,
+ struct dlb_ldb_port *port)
+{
+ int i;
+
+ for (i = 0; i < DLB_MAX_NUM_QIDS_PER_LDB_CQ; i++) {
+ struct dlb_ldb_queue *queue;
+ u32 infl_cnt;
+ int qid;
+
+ if (port->qid_map[i].state != DLB_QUEUE_MAP_IN_PROG)
+ continue;
+
+ qid = port->qid_map[i].qid;
+
+ queue = dlb_get_ldb_queue_from_id(hw, qid, false, 0);
+
+ if (!queue) {
+ DLB_HW_ERR(hw,
+ "[%s()] Internal error: unable to find queue %d\n",
+ __func__, qid);
+ continue;
+ }
+
+ infl_cnt = DLB_CSR_RD(hw, LSP_QID_LDB_INFL_CNT(qid));
+
+ if (BITS_GET(infl_cnt, LSP_QID_LDB_INFL_CNT_COUNT))
+ continue;
+
+ /*
+ * Disable the affected CQ, and the CQs already mapped to the
+ * QID, before reading the QID's inflight count a second time.
+ * There is an unlikely race in which the QID may schedule one
+ * more QE after we read an inflight count of 0, and disabling
+ * the CQs guarantees that the race will not occur after a
+ * re-read of the inflight count register.
+ */
+ if (port->enabled)
+ dlb_ldb_port_cq_disable(hw, port);
+
+ dlb_ldb_queue_disable_mapped_cqs(hw, domain, queue);
+
+ infl_cnt = DLB_CSR_RD(hw, LSP_QID_LDB_INFL_CNT(qid));
+
+ if (BITS_GET(infl_cnt, LSP_QID_LDB_INFL_CNT_COUNT)) {
+ if (port->enabled)
+ dlb_ldb_port_cq_enable(hw, port);
+
+ dlb_ldb_queue_enable_mapped_cqs(hw, domain, queue);
+
+ continue;
+ }
+
+ dlb_ldb_port_finish_map_qid_dynamic(hw, domain, port, queue);
+ }
+}
+
+static unsigned int
+dlb_domain_finish_map_qid_procedures(struct dlb_hw *hw,
+ struct dlb_hw_domain *domain)
+{
+ struct dlb_ldb_port *port;
+ int i;
+
+ if (!domain->configured || domain->num_pending_additions == 0)
+ return 0;
+
+ for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+ list_for_each_entry(port, &domain->used_ldb_ports[i], domain_list)
+ dlb_domain_finish_map_port(hw, domain, port);
+ }
+
+ return domain->num_pending_additions;
+}
+
static void dlb_log_map_qid(struct dlb_hw *hw, u32 domain_id,
struct dlb_map_qid_args *args,
bool vdev_req, unsigned int vdev_id)
@@ -4454,6 +4841,10 @@ int dlb_reset_domain(struct dlb_hw *hw, u32 domain_id, bool vdev_req,
if (ret)
return ret;
+ ret = dlb_domain_finish_map_qid_procedures(hw, domain);
+ if (ret)
+ return ret;
+
/* Re-enable the CQs in order to drain the mapped queues. */
dlb_domain_enable_ldb_cqs(hw, domain);
--
2.17.1
Add the low-level code for configuring a new queue and querying its depth.
When configuring a queue, program the device based on the user-supplied
queue configuration ioctl arguments.
Add low-level code for resetting (draining) a non-empty queue during
scheduling domain reset. Draining a queue is an iterative process of
checking if the queue is empty, and if not then selecting a linked 'victim'
port and dequeueing the queue's events through this port. A port can only
receive a small number of events at a time, usually much fewer than the
queue depth, so draining a queue typically takes multiple iterations. This
process is finite since software cannot enqueue new events to the DLB's
(finite) on-device storage.
Signed-off-by: Gage Eads <[email protected]>
Signed-off-by: Mike Ximing Chen <[email protected]>
Reviewed-by: Magnus Karlsson <[email protected]>
Reviewed-by: Dan Williams <[email protected]>
---
drivers/misc/dlb/dlb_hw_types.h | 46 +++
drivers/misc/dlb/dlb_resource.c | 610 +++++++++++++++++++++++++++++++-
2 files changed, 654 insertions(+), 2 deletions(-)
diff --git a/drivers/misc/dlb/dlb_hw_types.h b/drivers/misc/dlb/dlb_hw_types.h
index d382c414e2b0..c7827defa66a 100644
--- a/drivers/misc/dlb/dlb_hw_types.h
+++ b/drivers/misc/dlb/dlb_hw_types.h
@@ -50,6 +50,29 @@
#define PCI_DEVICE_ID_INTEL_DLB_PF 0x2710
+/*
+ * Hardware-defined base addresses. Those prefixed 'DLB_DRV' are only used by
+ * the PF driver.
+ */
+#define DLB_DRV_LDB_PP_BASE 0x2300000
+#define DLB_DRV_LDB_PP_STRIDE 0x1000
+#define DLB_DRV_LDB_PP_BOUND (DLB_DRV_LDB_PP_BASE + \
+ DLB_DRV_LDB_PP_STRIDE * DLB_MAX_NUM_LDB_PORTS)
+#define DLB_DRV_DIR_PP_BASE 0x2200000
+#define DLB_DRV_DIR_PP_STRIDE 0x1000
+#define DLB_DRV_DIR_PP_BOUND (DLB_DRV_DIR_PP_BASE + \
+ DLB_DRV_DIR_PP_STRIDE * DLB_MAX_NUM_DIR_PORTS)
+#define DLB_LDB_PP_BASE 0x2100000
+#define DLB_LDB_PP_STRIDE 0x1000
+#define DLB_LDB_PP_BOUND (DLB_LDB_PP_BASE + \
+ DLB_LDB_PP_STRIDE * DLB_MAX_NUM_LDB_PORTS)
+#define DLB_LDB_PP_OFFS(id) (DLB_LDB_PP_BASE + (id) * DLB_PP_SIZE)
+#define DLB_DIR_PP_BASE 0x2000000
+#define DLB_DIR_PP_STRIDE 0x1000
+#define DLB_DIR_PP_BOUND (DLB_DIR_PP_BASE + \
+ DLB_DIR_PP_STRIDE * DLB_MAX_NUM_DIR_PORTS)
+#define DLB_DIR_PP_OFFS(id) (DLB_DIR_PP_BASE + (id) * DLB_PP_SIZE)
+
struct dlb_resource_id {
u32 phys_id;
u32 virt_id;
@@ -68,6 +91,29 @@ static inline u32 dlb_freelist_count(struct dlb_freelist *list)
return list->bound - list->base - list->offset;
}
+struct dlb_hcw {
+ u64 data;
+ /* Word 3 */
+ u16 opaque;
+ u8 qid;
+ u8 sched_type:2;
+ u8 priority:3;
+ u8 msg_type:3;
+ /* Word 4 */
+ u16 lock_id;
+ u8 ts_flag:1;
+ u8 rsvd1:2;
+ u8 no_dec:1;
+ u8 cmp_id:4;
+ u8 cq_token:1;
+ u8 qe_comp:1;
+ u8 qe_frag:1;
+ u8 qe_valid:1;
+ u8 int_arm:1;
+ u8 error:1;
+ u8 rsvd:2;
+};
+
struct dlb_ldb_queue {
struct list_head domain_list;
struct list_head func_list;
diff --git a/drivers/misc/dlb/dlb_resource.c b/drivers/misc/dlb/dlb_resource.c
index b36f14a661fa..3c4f4c4af2ac 100644
--- a/drivers/misc/dlb/dlb_resource.c
+++ b/drivers/misc/dlb/dlb_resource.c
@@ -1,12 +1,24 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+#include <linux/log2.h>
#include "dlb_bitmap.h"
#include "dlb_hw_types.h"
#include "dlb_main.h"
#include "dlb_regs.h"
#include "dlb_resource.h"
+/*
+ * The PF driver cannot assume that a register write will affect subsequent HCW
+ * writes. To ensure a write completes, the driver must read back a CSR. This
+ * function only need be called for configuration that can occur after the
+ * domain has started; prior to starting, applications can't send HCWs.
+ */
+static inline void dlb_flush_csr(struct dlb_hw *hw)
+{
+ DLB_CSR_RD(hw, SYS_TOTAL_VAS);
+}
+
static void dlb_init_fn_rsrc_lists(struct dlb_function_resources *rsrc)
{
int i;
@@ -844,6 +856,148 @@ dlb_verify_create_dir_queue_args(struct dlb_hw *hw, u32 domain_id,
return 0;
}
+static void dlb_configure_ldb_queue(struct dlb_hw *hw,
+ struct dlb_hw_domain *domain,
+ struct dlb_ldb_queue *queue,
+ struct dlb_create_ldb_queue_args *args,
+ bool vdev_req, unsigned int vdev_id)
+{
+ struct dlb_sn_group *sn_group;
+ unsigned int offs;
+ u32 reg = 0;
+ u32 alimit;
+ u32 level;
+
+ /* QID write permissions are turned on when the domain is started */
+ offs = domain->id.phys_id * DLB_MAX_NUM_LDB_QUEUES + queue->id.phys_id;
+
+ DLB_CSR_WR(hw, SYS_LDB_VASQID_V(offs), reg);
+
+ /*
+ * Unordered QIDs get 4K inflights, ordered get as many as the number
+ * of sequence numbers.
+ */
+ BITS_SET(reg, args->num_qid_inflights, LSP_QID_LDB_INFL_LIM_LIMIT);
+ DLB_CSR_WR(hw, LSP_QID_LDB_INFL_LIM(queue->id.phys_id), reg);
+
+ alimit = queue->aqed_limit;
+
+ if (alimit > DLB_MAX_NUM_AQED_ENTRIES)
+ alimit = DLB_MAX_NUM_AQED_ENTRIES;
+
+ reg = 0;
+ BITS_SET(reg, alimit, LSP_QID_AQED_ACTIVE_LIM_LIMIT);
+ DLB_CSR_WR(hw, LSP_QID_AQED_ACTIVE_LIM(queue->id.phys_id), reg);
+
+ level = args->lock_id_comp_level;
+ if (level >= 64 && level <= 4096)
+ BITS_SET(reg, ilog2(level) - 5, AQED_QID_HID_WIDTH_COMPRESS_CODE);
+ else
+ reg = 0;
+
+ DLB_CSR_WR(hw, AQED_QID_HID_WIDTH(queue->id.phys_id), reg);
+
+ reg = 0;
+ /* Don't timestamp QEs that pass through this queue */
+ DLB_CSR_WR(hw, SYS_LDB_QID_ITS(queue->id.phys_id), reg);
+
+ BITS_SET(reg, args->depth_threshold, LSP_QID_ATM_DEPTH_THRSH_THRESH);
+ DLB_CSR_WR(hw, LSP_QID_ATM_DEPTH_THRSH(queue->id.phys_id), reg);
+
+ reg = 0;
+ BITS_SET(reg, args->depth_threshold, LSP_QID_NALDB_DEPTH_THRSH_THRESH);
+ DLB_CSR_WR(hw, LSP_QID_NALDB_DEPTH_THRSH(queue->id.phys_id), reg);
+
+ /*
+ * This register limits the number of inflight flows a queue can have
+ * at one time. It has an upper bound of 2048, but can be
+ * over-subscribed. 512 is chosen so that a single queue doesn't use
+ * the entire atomic storage, but can use a substantial portion if
+ * needed.
+ */
+ reg = 0;
+ BITS_SET(reg, 512, AQED_QID_FID_LIM_QID_FID_LIMIT);
+ DLB_CSR_WR(hw, AQED_QID_FID_LIM(queue->id.phys_id), reg);
+
+ /* Configure SNs */
+ reg = 0;
+ sn_group = &hw->rsrcs.sn_groups[queue->sn_group];
+ BITS_SET(reg, sn_group->mode, CHP_ORD_QID_SN_MAP_MODE);
+ BITS_SET(reg, queue->sn_slot, CHP_ORD_QID_SN_MAP_SLOT);
+ BITS_SET(reg, sn_group->id, CHP_ORD_QID_SN_MAP_GRP);
+
+ DLB_CSR_WR(hw, CHP_ORD_QID_SN_MAP(queue->id.phys_id), reg);
+
+ reg = 0;
+ BITS_SET(reg, (u32)(args->num_sequence_numbers != 0),
+ SYS_LDB_QID_CFG_V_SN_CFG_V);
+ BITS_SET(reg, (u32)(args->num_atomic_inflights != 0),
+ SYS_LDB_QID_CFG_V_FID_CFG_V);
+
+ DLB_CSR_WR(hw, SYS_LDB_QID_CFG_V(queue->id.phys_id), reg);
+
+ if (vdev_req) {
+ offs = vdev_id * DLB_MAX_NUM_LDB_QUEUES + queue->id.virt_id;
+
+ reg = 0;
+ reg |= SYS_VF_LDB_VQID_V_VQID_V;
+ DLB_CSR_WR(hw, SYS_VF_LDB_VQID_V(offs), reg);
+
+ reg = 0;
+ BITS_SET(reg, queue->id.phys_id, SYS_VF_LDB_VQID2QID_QID);
+ DLB_CSR_WR(hw, SYS_VF_LDB_VQID2QID(offs), reg);
+
+ reg = 0;
+ BITS_SET(reg, queue->id.virt_id, SYS_LDB_QID2VQID_VQID);
+ DLB_CSR_WR(hw, SYS_LDB_QID2VQID(queue->id.phys_id), reg);
+ }
+
+ reg = 0;
+ reg |= SYS_LDB_QID_V_QID_V;
+ DLB_CSR_WR(hw, SYS_LDB_QID_V(queue->id.phys_id), reg);
+}
+
+static void dlb_configure_dir_queue(struct dlb_hw *hw,
+ struct dlb_hw_domain *domain,
+ struct dlb_dir_pq_pair *queue,
+ struct dlb_create_dir_queue_args *args,
+ bool vdev_req, unsigned int vdev_id)
+{
+ unsigned int offs;
+ u32 reg = 0;
+
+ /* QID write permissions are turned on when the domain is started */
+ offs = domain->id.phys_id * DLB_MAX_NUM_DIR_QUEUES +
+ queue->id.phys_id;
+
+ DLB_CSR_WR(hw, SYS_DIR_VASQID_V(offs), reg);
+
+ /* Don't timestamp QEs that pass through this queue */
+ DLB_CSR_WR(hw, SYS_DIR_QID_ITS(queue->id.phys_id), reg);
+
+ reg = 0;
+ BITS_SET(reg, args->depth_threshold, LSP_QID_DIR_DEPTH_THRSH_THRESH);
+ DLB_CSR_WR(hw, LSP_QID_DIR_DEPTH_THRSH(queue->id.phys_id), reg);
+
+ if (vdev_req) {
+ offs = vdev_id * DLB_MAX_NUM_DIR_QUEUES + queue->id.virt_id;
+
+ reg = 0;
+ reg |= SYS_VF_DIR_VQID_V_VQID_V;
+ DLB_CSR_WR(hw, SYS_VF_DIR_VQID_V(offs), reg);
+
+ reg = 0;
+ BITS_SET(reg, queue->id.phys_id, SYS_VF_DIR_VQID2QID_QID);
+ DLB_CSR_WR(hw, SYS_VF_DIR_VQID2QID(offs), reg);
+ }
+
+ reg = 0;
+ reg |= SYS_DIR_QID_V_QID_V;
+ DLB_CSR_WR(hw, SYS_DIR_QID_V(queue->id.phys_id), reg);
+
+ queue->queue_configured = true;
+}
+
static void dlb_configure_domain_credits(struct dlb_hw *hw,
struct dlb_hw_domain *domain)
{
@@ -971,6 +1125,56 @@ dlb_ldb_queue_attach_resources(struct dlb_hw *hw,
return 0;
}
+static void dlb_ldb_port_cq_enable(struct dlb_hw *hw,
+ struct dlb_ldb_port *port)
+{
+ u32 reg = 0;
+
+ /*
+ * Don't re-enable the port if a removal is pending. The caller should
+ * mark this port as enabled (if it isn't already), and when the
+ * removal completes the port will be enabled.
+ */
+ if (port->num_pending_removals)
+ return;
+
+ DLB_CSR_WR(hw, LSP_CQ_LDB_DSBL(port->id.phys_id), reg);
+
+ dlb_flush_csr(hw);
+}
+
+static void dlb_ldb_port_cq_disable(struct dlb_hw *hw,
+ struct dlb_ldb_port *port)
+{
+ u32 reg = 0;
+
+ reg |= LSP_CQ_LDB_DSBL_DISABLED;
+ DLB_CSR_WR(hw, LSP_CQ_LDB_DSBL(port->id.phys_id), reg);
+
+ dlb_flush_csr(hw);
+}
+
+static void dlb_dir_port_cq_enable(struct dlb_hw *hw,
+ struct dlb_dir_pq_pair *port)
+{
+ u32 reg = 0;
+
+ DLB_CSR_WR(hw, LSP_CQ_DIR_DSBL(port->id.phys_id), reg);
+
+ dlb_flush_csr(hw);
+}
+
+static void dlb_dir_port_cq_disable(struct dlb_hw *hw,
+ struct dlb_dir_pq_pair *port)
+{
+ u32 reg = 0;
+
+ reg |= LSP_CQ_DIR_DSBL_DISABLED;
+ DLB_CSR_WR(hw, LSP_CQ_DIR_DSBL(port->id.phys_id), reg);
+
+ dlb_flush_csr(hw);
+}
+
static void
dlb_log_create_sched_domain_args(struct dlb_hw *hw,
struct dlb_create_sched_domain_args *args,
@@ -1147,6 +1351,8 @@ int dlb_hw_create_ldb_queue(struct dlb_hw *hw, u32 domain_id,
return ret;
}
+ dlb_configure_ldb_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
queue->num_mappings = 0;
queue->configured = true;
@@ -1223,6 +1429,8 @@ int dlb_hw_create_dir_queue(struct dlb_hw *hw, u32 domain_id,
if (ret)
return ret;
+ dlb_configure_dir_queue(hw, domain, queue, args, vdev_req, vdev_id);
+
/*
* Configuration succeeded, so move the resource from the 'avail' to
* the 'used' list (if it's not already there).
@@ -1240,6 +1448,92 @@ int dlb_hw_create_dir_queue(struct dlb_hw *hw, u32 domain_id,
return 0;
}
+static u32 dlb_ldb_cq_inflight_count(struct dlb_hw *hw,
+ struct dlb_ldb_port *port)
+{
+ u32 cnt;
+
+ cnt = DLB_CSR_RD(hw, LSP_CQ_LDB_INFL_CNT(port->id.phys_id));
+
+ return BITS_GET(cnt, LSP_CQ_LDB_INFL_CNT_COUNT);
+}
+
+static u32 dlb_ldb_cq_token_count(struct dlb_hw *hw, struct dlb_ldb_port *port)
+{
+ u32 cnt;
+
+ cnt = DLB_CSR_RD(hw, LSP_CQ_LDB_TKN_CNT(port->id.phys_id));
+
+ /*
+ * Account for the initial token count, which is used in order to
+ * provide a CQ with depth less than 8.
+ */
+
+ return BITS_GET(cnt, LSP_CQ_LDB_TKN_CNT_TOKEN_COUNT) - port->init_tkn_cnt;
+}
+
+static void __iomem *dlb_producer_port_addr(struct dlb_hw *hw, u8 port_id,
+ bool is_ldb)
+{
+ struct dlb *dlb = container_of(hw, struct dlb, hw);
+ uintptr_t address = (uintptr_t)dlb->hw.func_kva;
+ unsigned long size;
+
+ if (is_ldb) {
+ size = DLB_LDB_PP_STRIDE;
+ address += DLB_DRV_LDB_PP_BASE + size * port_id;
+ } else {
+ size = DLB_DIR_PP_STRIDE;
+ address += DLB_DRV_DIR_PP_BASE + size * port_id;
+ }
+
+ return (void __iomem *)address;
+}
+
+static void dlb_drain_ldb_cq(struct dlb_hw *hw, struct dlb_ldb_port *port)
+{
+ u32 infl_cnt, tkn_cnt;
+ unsigned int i;
+
+ infl_cnt = dlb_ldb_cq_inflight_count(hw, port);
+ tkn_cnt = dlb_ldb_cq_token_count(hw, port);
+
+ if (infl_cnt || tkn_cnt) {
+ struct dlb_hcw hcw_mem[8], *hcw;
+ void __iomem *pp_addr;
+
+ pp_addr = dlb_producer_port_addr(hw, port->id.phys_id, true);
+
+ /* Point hcw to a 64B-aligned location */
+ hcw = (struct dlb_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+ /*
+ * Program the first HCW for a completion and token return and
+ * the other HCWs as NOOPS
+ */
+
+ memset(hcw, 0, 4 * sizeof(*hcw));
+ hcw->qe_comp = (infl_cnt > 0);
+ hcw->cq_token = (tkn_cnt > 0);
+ hcw->lock_id = tkn_cnt - 1;
+
+ /* Return tokens in the first HCW */
+ iosubmit_cmds512(pp_addr, hcw, 1);
+
+ hcw->cq_token = 0;
+
+ /* Issue remaining completions (if any) */
+ for (i = 1; i < infl_cnt; i++)
+ iosubmit_cmds512(pp_addr, hcw, 1);
+
+ /*
+ * To ensure outstanding HCWs reach the device before subsequent device
+ * accesses, fence them.
+ */
+ mb();
+ }
+}
+
static int dlb_domain_reset_software_state(struct dlb_hw *hw,
struct dlb_hw_domain *domain)
{
@@ -1385,6 +1679,21 @@ static int dlb_domain_reset_software_state(struct dlb_hw *hw,
return 0;
}
+static u32 dlb_dir_queue_depth(struct dlb_hw *hw, struct dlb_dir_pq_pair *queue)
+{
+ u32 cnt;
+
+ cnt = DLB_CSR_RD(hw, LSP_QID_DIR_ENQUEUE_CNT(queue->id.phys_id));
+
+ return BITS_GET(cnt, LSP_QID_DIR_ENQUEUE_CNT_COUNT);
+}
+
+static bool dlb_dir_queue_is_empty(struct dlb_hw *hw,
+ struct dlb_dir_pq_pair *queue)
+{
+ return dlb_dir_queue_depth(hw, queue) == 0;
+}
+
static void dlb_log_get_dir_queue_depth(struct dlb_hw *hw, u32 domain_id,
u32 queue_id, bool vdev_req,
unsigned int vf_id)
@@ -1446,7 +1755,7 @@ int dlb_hw_get_dir_queue_depth(struct dlb_hw *hw, u32 domain_id,
return -EINVAL;
}
- resp->id = 0;
+ resp->id = dlb_dir_queue_depth(hw, queue);
return 0;
}
@@ -1525,7 +1834,7 @@ int dlb_hw_get_ldb_queue_depth(struct dlb_hw *hw, u32 domain_id,
return -EINVAL;
}
- resp->id = 0;
+ resp->id = dlb_ldb_queue_depth(hw, queue);
return 0;
}
@@ -1894,6 +2203,21 @@ static void dlb_domain_reset_dir_queue_registers(struct dlb_hw *hw,
}
}
+static u32 dlb_dir_cq_token_count(struct dlb_hw *hw,
+ struct dlb_dir_pq_pair *port)
+{
+ u32 cnt;
+
+ cnt = DLB_CSR_RD(hw, LSP_CQ_DIR_TKN_CNT(port->id.phys_id));
+
+ /*
+ * Account for the initial token count, which is used in order to
+ * provide a CQ with depth less than 8.
+ */
+
+ return BITS_GET(cnt, LSP_CQ_DIR_TKN_CNT_COUNT) - port->init_tkn_cnt;
+}
+
static int dlb_domain_verify_reset_success(struct dlb_hw *hw,
struct dlb_hw_domain *domain)
{
@@ -1935,6 +2259,270 @@ static void dlb_domain_reset_registers(struct dlb_hw *hw,
CHP_CFG_DIR_VAS_CRD_RST);
}
+static void dlb_domain_drain_ldb_cqs(struct dlb_hw *hw,
+ struct dlb_hw_domain *domain,
+ bool toggle_port)
+{
+ struct dlb_ldb_port *port;
+ int i;
+
+ /* If the domain hasn't been started, there's no traffic to drain */
+ if (!domain->started)
+ return;
+
+ for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+ list_for_each_entry(port, &domain->used_ldb_ports[i], domain_list) {
+ if (toggle_port)
+ dlb_ldb_port_cq_disable(hw, port);
+
+ dlb_drain_ldb_cq(hw, port);
+
+ if (toggle_port)
+ dlb_ldb_port_cq_enable(hw, port);
+ }
+ }
+}
+
+static bool dlb_domain_mapped_queues_empty(struct dlb_hw *hw,
+ struct dlb_hw_domain *domain)
+{
+ struct dlb_ldb_queue *queue;
+
+ list_for_each_entry(queue, &domain->used_ldb_queues, domain_list) {
+ if (queue->num_mappings == 0)
+ continue;
+
+ if (!dlb_ldb_queue_is_empty(hw, queue))
+ return false;
+ }
+
+ return true;
+}
+
+static int dlb_domain_drain_mapped_queues(struct dlb_hw *hw,
+ struct dlb_hw_domain *domain)
+{
+ int i;
+
+ /* If the domain hasn't been started, there's no traffic to drain */
+ if (!domain->started)
+ return 0;
+
+ if (domain->num_pending_removals > 0) {
+ DLB_HW_ERR(hw,
+ "[%s()] Internal error: failed to unmap domain queues\n",
+ __func__);
+ return -EFAULT;
+ }
+
+ for (i = 0; i < DLB_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+ dlb_domain_drain_ldb_cqs(hw, domain, true);
+
+ if (dlb_domain_mapped_queues_empty(hw, domain))
+ break;
+ }
+
+ if (i == DLB_MAX_QID_EMPTY_CHECK_LOOPS) {
+ DLB_HW_ERR(hw,
+ "[%s()] Internal error: failed to empty queues\n",
+ __func__);
+ return -EFAULT;
+ }
+
+ /*
+ * Drain the CQs one more time. For the queues to go empty, they would
+ * have scheduled one or more QEs.
+ */
+ dlb_domain_drain_ldb_cqs(hw, domain, true);
+
+ return 0;
+}
+
+static int dlb_drain_dir_cq(struct dlb_hw *hw, struct dlb_dir_pq_pair *port)
+{
+ unsigned int port_id = port->id.phys_id;
+ u32 cnt;
+
+ /* Return any outstanding tokens */
+ cnt = dlb_dir_cq_token_count(hw, port);
+
+ if (cnt != 0) {
+ struct dlb_hcw hcw_mem[8], *hcw;
+ void __iomem *pp_addr;
+
+ pp_addr = dlb_producer_port_addr(hw, port_id, false);
+
+ /* Point hcw to a 64B-aligned location */
+ hcw = (struct dlb_hcw *)((uintptr_t)&hcw_mem[4] & ~0x3F);
+
+ /*
+ * Program the first HCW for a batch token return and
+ * the rest as NOOPS
+ */
+ memset(hcw, 0, 4 * sizeof(*hcw));
+ hcw->cq_token = 1;
+ hcw->lock_id = cnt - 1;
+
+ iosubmit_cmds512(pp_addr, hcw, 1);
+
+ /*
+ * To ensure outstanding HCWs reach the device before subsequent device
+ * accesses, fence them.
+ */
+ mb();
+ }
+
+ return 0;
+}
+
+static int dlb_domain_drain_dir_cqs(struct dlb_hw *hw,
+ struct dlb_hw_domain *domain,
+ bool toggle_port)
+{
+ struct dlb_dir_pq_pair *port;
+ int ret;
+
+ list_for_each_entry(port, &domain->used_dir_pq_pairs, domain_list) {
+ /*
+ * Can't drain a port if it's not configured, and there's
+ * nothing to drain if its queue is unconfigured.
+ */
+ if (!port->port_configured || !port->queue_configured)
+ continue;
+
+ if (toggle_port)
+ dlb_dir_port_cq_disable(hw, port);
+
+ ret = dlb_drain_dir_cq(hw, port);
+ if (ret)
+ return ret;
+
+ if (toggle_port)
+ dlb_dir_port_cq_enable(hw, port);
+ }
+
+ return 0;
+}
+
+static bool dlb_domain_dir_queues_empty(struct dlb_hw *hw,
+ struct dlb_hw_domain *domain)
+{
+ struct dlb_dir_pq_pair *queue;
+
+ list_for_each_entry(queue, &domain->used_dir_pq_pairs, domain_list) {
+ if (!dlb_dir_queue_is_empty(hw, queue))
+ return false;
+ }
+
+ return true;
+}
+
+static int dlb_domain_drain_dir_queues(struct dlb_hw *hw,
+ struct dlb_hw_domain *domain)
+{
+ int i, ret;
+
+ /* If the domain hasn't been started, there's no traffic to drain */
+ if (!domain->started)
+ return 0;
+
+ for (i = 0; i < DLB_MAX_QID_EMPTY_CHECK_LOOPS; i++) {
+ ret = dlb_domain_drain_dir_cqs(hw, domain, true);
+ if (ret)
+ return ret;
+
+ if (dlb_domain_dir_queues_empty(hw, domain))
+ break;
+ }
+
+ if (i == DLB_MAX_QID_EMPTY_CHECK_LOOPS) {
+ DLB_HW_ERR(hw,
+ "[%s()] Internal error: failed to empty queues\n",
+ __func__);
+ return -EFAULT;
+ }
+
+ /*
+ * Drain the CQs one more time. For the queues to go empty, they would
+ * have scheduled one or more QEs.
+ */
+ ret = dlb_domain_drain_dir_cqs(hw, domain, true);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static void
+dlb_domain_disable_ldb_queue_write_perms(struct dlb_hw *hw,
+ struct dlb_hw_domain *domain)
+{
+ int domain_offset = domain->id.phys_id * DLB_MAX_NUM_LDB_QUEUES;
+ struct dlb_ldb_queue *queue;
+
+ list_for_each_entry(queue, &domain->used_ldb_queues, domain_list) {
+ int idx = domain_offset + queue->id.phys_id;
+
+ DLB_CSR_WR(hw, SYS_LDB_VASQID_V(idx), 0);
+ }
+}
+
+static void
+dlb_domain_disable_dir_queue_write_perms(struct dlb_hw *hw,
+ struct dlb_hw_domain *domain)
+{
+ int domain_offset = domain->id.phys_id * DLB_MAX_NUM_DIR_PORTS;
+ struct dlb_dir_pq_pair *queue;
+
+ list_for_each_entry(queue, &domain->used_dir_pq_pairs, domain_list) {
+ int idx = domain_offset + queue->id.phys_id;
+
+ DLB_CSR_WR(hw, SYS_DIR_VASQID_V(idx), 0);
+ }
+}
+
+static void dlb_domain_disable_dir_cqs(struct dlb_hw *hw,
+ struct dlb_hw_domain *domain)
+{
+ struct dlb_dir_pq_pair *port;
+
+ list_for_each_entry(port, &domain->used_dir_pq_pairs, domain_list) {
+ port->enabled = false;
+
+ dlb_dir_port_cq_disable(hw, port);
+ }
+}
+
+static void dlb_domain_disable_ldb_cqs(struct dlb_hw *hw,
+ struct dlb_hw_domain *domain)
+{
+ struct dlb_ldb_port *port;
+ int i;
+
+ for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+ list_for_each_entry(port, &domain->used_ldb_ports[i], domain_list) {
+ port->enabled = false;
+
+ dlb_ldb_port_cq_disable(hw, port);
+ }
+ }
+}
+
+static void dlb_domain_enable_ldb_cqs(struct dlb_hw *hw,
+ struct dlb_hw_domain *domain)
+{
+ struct dlb_ldb_port *port;
+ int i;
+
+ for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+ list_for_each_entry(port, &domain->used_ldb_ports[i], domain_list) {
+ port->enabled = true;
+
+ dlb_ldb_port_cq_enable(hw, port);
+ }
+ }
+}
+
static void dlb_log_reset_domain(struct dlb_hw *hw, u32 domain_id,
bool vdev_req, unsigned int vdev_id)
{
@@ -1987,6 +2575,24 @@ int dlb_reset_domain(struct dlb_hw *hw, u32 domain_id, bool vdev_req,
* cause any traffic sent to it to be dropped. Well-behaved software
* should not be sending QEs at this point.
*/
+ dlb_domain_disable_dir_queue_write_perms(hw, domain);
+
+ dlb_domain_disable_ldb_queue_write_perms(hw, domain);
+
+ /* Re-enable the CQs in order to drain the mapped queues. */
+ dlb_domain_enable_ldb_cqs(hw, domain);
+
+ ret = dlb_domain_drain_mapped_queues(hw, domain);
+ if (ret)
+ return ret;
+
+ /* Done draining LDB QEs, so disable the CQs. */
+ dlb_domain_disable_ldb_cqs(hw, domain);
+
+ dlb_domain_drain_dir_queues(hw, domain);
+
+ /* Done draining DIR QEs, so disable the CQs. */
+ dlb_domain_disable_dir_cqs(hw, domain);
ret = dlb_domain_verify_reset_success(hw, domain);
if (ret)
--
2.17.1
Add basic driver functionality (load, unload, probe, and remove callbacks)
for the DLB driver.
Add documentation which describes in detail the hardware, the user
interface, device interrupts, and the driver's power-management strategy.
For more details about the driver see the documentation in the patch.
Add a DLB entry to the MAINTAINERS file.
Signed-off-by: Gage Eads <[email protected]>
Signed-off-by: Mike Ximing Chen <[email protected]>
Reviewed-by: Magnus Karlsson <[email protected]>
Reviewed-by: Dan Williams <[email protected]>
---
Documentation/misc-devices/dlb.rst | 259 +++++++++++++++++++++++++++
Documentation/misc-devices/index.rst | 1 +
MAINTAINERS | 8 +
drivers/misc/Kconfig | 1 +
drivers/misc/Makefile | 1 +
drivers/misc/dlb/Kconfig | 18 ++
drivers/misc/dlb/Makefile | 9 +
drivers/misc/dlb/dlb_hw_types.h | 32 ++++
drivers/misc/dlb/dlb_main.c | 156 ++++++++++++++++
drivers/misc/dlb/dlb_main.h | 37 ++++
10 files changed, 522 insertions(+)
create mode 100644 Documentation/misc-devices/dlb.rst
create mode 100644 drivers/misc/dlb/Kconfig
create mode 100644 drivers/misc/dlb/Makefile
create mode 100644 drivers/misc/dlb/dlb_hw_types.h
create mode 100644 drivers/misc/dlb/dlb_main.c
create mode 100644 drivers/misc/dlb/dlb_main.h
diff --git a/Documentation/misc-devices/dlb.rst b/Documentation/misc-devices/dlb.rst
new file mode 100644
index 000000000000..aa79be07ee49
--- /dev/null
+++ b/Documentation/misc-devices/dlb.rst
@@ -0,0 +1,259 @@
+.. SPDX-License-Identifier: GPL-2.0-only
+
+===========================================
+Intel(R) Dynamic Load Balancer Overview
+===========================================
+
+:Authors: Gage Eads and Mike Ximing Chen
+
+Contents
+========
+
+- Introduction
+- Scheduling
+- Queue Entry
+- Port
+- Queue
+- Credits
+- Scheduling Domain
+- Interrupts
+- Power Management
+- User Interface
+- Reset
+
+Introduction
+============
+
+The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device that
+provides load-balanced, prioritized scheduling of core-to-core communication.
+
+Intel DLB is an accelerator for the event-driven programming model of
+DPDK's Event Device Library[2]. The library is used in packet processing
+pipelines that arrange for multi-core scalability, dynamic load-balancing, and
+variety of packet distribution and synchronization schemes.
+
+Intel DLB device consists of queues and arbiters that connect producer
+cores and consumer cores. The device implements load-balanced queueing features
+including:
+- Lock-free multi-producer/multi-consumer operation.
+- Multiple priority levels for varying traffic types.
+- 'Direct' traffic (i.e. multi-producer/single-consumer)
+- Simple unordered load-balanced distribution.
+- Atomic lock free load balancing across multiple consumers.
+- Queue element reordering feature allowing ordered load-balanced distribution.
+
+Note: this document uses 'DLB' when discussing the device hardware and 'dlb' when
+discussing the driver implementation.
+
+Following diagram illustrates the functional blocks of an Intel DLB device.
+
+ +----+
+ | |
+ +----------+ | | +-------+
+ /| IQ |---|----|--/| |
+ / +----------+ | | / | CP |
+ / | |/ +-------+
+ +--------+ / | |
+ | | / +----------+ | /| +-------+
+ | PP |------| IQ |---|----|---| |
+ +--------+ \ +----------+ | / | | CP |
+ \ |/ | +-------+
+ ... \ ... | |
+ +--------+ \ /| | +-------+
+ | | \+----------+ / | | | |
+ | PP |------| IQ |/--|----|---| CP |
+ +--------+ +----------+ | | +-------+
+ | |
+ +----+ ...
+PP: Producer Port |
+CP: Consumer Port |
+IQ: Internal Queue DLB Scheduler
+
+
+Scheduling Types
+================
+
+Intel DLB supports four types of scheduling of 'events' (using DPDK
+terminology), where an event can represent any type of data (e.g. a network
+packet). The first, ``directed``, is multi-producer/single-consumer style
+core-to-core communication. The remaining three are
+multi-producer/multi-consumer, and support load-balancing across the consumers.
+
+- ``Directed``: events are scheduled to a single consumer.
+
+- ``Unordered``: events are load-balanced across consumers without any ordering
+ guarantees.
+
+- ``Ordered``: events are load-balanced across consumers, and the consumer can
+ re-enqueue its events so the device re-orders them into the
+ original order. This scheduling type allows software to
+ parallelize ordered event processing without the synchronization
+ cost of re-ordering packets.
+
+- ``Atomic``: events are load-balanced across consumers, with the guarantee that
+ events from a particular 'flow' are only scheduled to a single
+ consumer at a time (but can migrate over time). This allows, for
+ example, packet processing applications to parallelize while
+ avoiding locks on per-flow data and maintaining ordering within a
+ flow.
+
+Intel DLB provides hierarchical priority scheduling, with eight priority
+levels within each. Each consumer selects up to eight queues to receive events
+from, and assigns a priority to each of these 'connected' queues. To schedule
+an event to a consumer, the device selects the highest priority non-empty queue
+of the (up to) eight connected queues. Within that queue, the device selects
+the highest priority event available (selecting a lower priority event for
+starvation avoidance 1% of the time, by default).
+
+The device also supports four load-balanced scheduler classes of service. Each
+class of service receives a (user-configurable) guaranteed percentage of the
+scheduler bandwidth, and any unreserved bandwidth is divided evenly among the
+four classes.
+
+Queue Entry
+===========
+
+Each event is contained in a queue entry (QE), the fundamental unit of
+communication through the device, which consists of 8B of data and 8B of
+metadata, as depicted below.
+
+QE structure format
+::
+ data :64
+ opaque :16
+ qid :8
+ sched :2
+ priority :3
+ msg_type :3
+ lock_id :16
+ rsvd :8
+ cmd :8
+
+The ``data`` field can be any type that fits within 8B (pointer, integer,
+etc.); DLB merely copies this field from producer to consumer. The
+``opaque`` and ``msg_type`` fields behave the same way.
+
+``qid`` is set by the producer to specify to which DLB 2.0 queue it wishes to
+enqueue this QE. The ID spaces for load-balanced and directed queues are both
+zero-based.
+
+``sched`` controls the scheduling type: atomic, unordered, ordered, or
+directed. The first three scheduling types are only valid for load-balanced
+queues, and the directed scheduling type is only valid for directed queues.
+This field distinguishes whether ``qid`` is load-balanced or directed, since
+their ID spaces overlap.
+
+``priority`` is the priority with which this QE should be scheduled.
+
+``lock_id``, used for atomic scheduling and ignored for ordered and unordered
+scheduling, identifies the atomic flow to which the QE belongs. When sending a
+directed event, ``lock_id`` is simply copied like the ``data``, ``opaque``, and
+``msg_type`` fields.
+
+``cmd`` specifies the operation, such as:
+- Enqueue a new QE
+- Forward a QE that was dequeued
+- Complete/terminate a QE that was dequeued
+- Return one or more consumer queue tokens.
+- Arm the port's consumer queue interrupt.
+
+Port
+====
+
+A core's interface to the DLB is called a "port," and consists of an MMIO
+region (producer port) through which the core enqueues a queue entry, and an
+in-memory queue (the "consumer queue" or cosumer port) to which the device
+schedules QEs. A core enqueues a QE to a device queue, then the device
+schedules the event to a port. Software specifies the connection of queues
+and ports; i.e. for each queue, to which ports the device is allowed to
+schedule its events. The device uses a credit scheme to prevent overflow of
+the on-device queue storage.
+
+Applications interface directly with the device by mapping the port's memory
+and MMIO regions into the application's address space for enqueue and dequeue
+operations, but call into the kernel driver for configuration operations. An
+application can be polling- or interrupt-driven; DLB supports both modes
+of operation.
+
+Internal Queue
+==============
+
+A DLB device supports an implementation specific and runtime discoverable
+number of load-balanced (i.e. capable of atomic, ordered, and unordered
+scheduling) and directed queues. Each internal queue supports a set of
+priority levels.
+
+A load-balanced queue is capable of scheduling its events to any combination of
+load-balanced ports, whereas each directed queue has one-to-one mapping with a
+directed port. There is no restriction on port or queue types when a port
+enqueues an event to a queue; that is, a load-balanced port can enqueue to a
+directed queue and vice versa.
+
+Credits
+=======
+
+The Intel DLB uses a credit scheme to prevent overflow of the on-device
+queue storage, with separate credits for load-balanced and directed queues. A
+port spends one credit when it enqueues a QE, and one credit is replenished
+when a QE is scheduled to a consumer queue. Each scheduling domain has one pool
+of load-balanced credits and one pool of directed credits; software is
+responsible for managing the allocation and replenishment of these credits among
+the scheduling domain's ports.
+
+Scheduling Domain
+=================
+
+Device resources -- including ports, queues, and credits -- are contained
+within a scheduling domain. Scheduling domains are isolated from one another; a
+port can only enqueue to and dequeue from queues within its scheduling domain.
+
+The scheduling domain creation ioctl returns a domain file descriptor, through
+which the domain's resources are configured. For a multi-process scenario, the
+owner of this descriptor must share it with the other processes (e.g. inherited
+through fork() or shared over a unix domain socket).
+
+Consumer Queue Interrupts
+=========================
+
+Each port has its own interrupt which fires, if armed, when the consumer queue
+depth becomes non-zero. Software arms an interrupt by enqueueing a special
+'interrupt arm' command to the device through the port's MMIO window.
+
+Power Management
+================
+
+The kernel driver keeps the device in D3Hot when not in use. The driver
+transitions the device to D0 when the first device file is opened, and keeps it
+there until there are no open device files or memory mappings.
+
+User Interface
+==============
+
+The dlb driver uses ioctls as its primary interface. It provides two types of
+files: the dlb device file and the scheduling domain file.
+
+The two types support different ioctl interfaces; the dlb device file is used
+for device-wide operations (including scheduling domain creation), and the
+scheduling domain device file supports operations on the scheduling domain's
+resources such as port and queue configuration.
+
+The dlb device file is created during driver probe and is located at
+/dev/dlb<N>, where N is the zero-based device ID. The scheduling domain fd is
+an anonymous inode created by a dlb device ioctl.
+
+The driver also exports an mmap interface through port files, which are
+acquired through scheduling domain ioctls. This mmap interface is used to map
+a port's memory and MMIO window into the process's address space. Once the
+ports are mapped, applications may use 64-byte direct-store instructions such
+movdir64b or enqcmd to enqueue the events for better performance.
+
+Reset
+=====
+
+The dlb driver currently supports scheduling domain reset.
+
+Scheduling domain reset occurs when an application stops using its domain.
+Specifically, when no more file references or memory mappings exist. At this
+time, the driver resets all the domain's resources (flushes its queues and
+ports) and puts them in their respective available-resource lists for later
+use.
diff --git a/Documentation/misc-devices/index.rst b/Documentation/misc-devices/index.rst
index 64420b3314fe..c9954e8b72a3 100644
--- a/Documentation/misc-devices/index.rst
+++ b/Documentation/misc-devices/index.rst
@@ -17,6 +17,7 @@ fit into other categories.
ad525x_dpot
apds990x
bh1770glc
+ dlb
eeprom
c2port
ibmvmc
diff --git a/MAINTAINERS b/MAINTAINERS
index 6eff4f720c72..b408c1ec8fdf 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -8896,6 +8896,14 @@ L: [email protected]
S: Supported
F: arch/x86/include/asm/intel-family.h
+INTEL DYNAMIC LOAD BALANCER DRIVER
+M: Mike Ximing Chen <[email protected]>
+M: Gage Eads <[email protected]>
+S: Maintained
+F: Documentation/ABI/testing/sysfs-driver-dlb
+F: drivers/misc/dlb/
+F: include/uapi/linux/dlb_user.h
+
INTEL DRM DRIVERS (excluding Poulsbo, Moorestown and derivative chipsets)
M: Jani Nikula <[email protected]>
M: Joonas Lahtinen <[email protected]>
diff --git a/drivers/misc/Kconfig b/drivers/misc/Kconfig
index fafa8b0d8099..fef26819eb1e 100644
--- a/drivers/misc/Kconfig
+++ b/drivers/misc/Kconfig
@@ -481,4 +481,5 @@ source "drivers/misc/ocxl/Kconfig"
source "drivers/misc/cardreader/Kconfig"
source "drivers/misc/habanalabs/Kconfig"
source "drivers/misc/uacce/Kconfig"
+source "drivers/misc/dlb/Kconfig"
endmenu
diff --git a/drivers/misc/Makefile b/drivers/misc/Makefile
index d23231e73330..a0bafe336277 100644
--- a/drivers/misc/Makefile
+++ b/drivers/misc/Makefile
@@ -57,3 +57,4 @@ obj-$(CONFIG_HABANA_AI) += habanalabs/
obj-$(CONFIG_UACCE) += uacce/
obj-$(CONFIG_XILINX_SDFEC) += xilinx_sdfec.o
obj-$(CONFIG_HISI_HIKEY_USB) += hisi_hikey_usb.o
+obj-$(CONFIG_INTEL_DLB) += dlb/
diff --git a/drivers/misc/dlb/Kconfig b/drivers/misc/dlb/Kconfig
new file mode 100644
index 000000000000..cfa978c705bd
--- /dev/null
+++ b/drivers/misc/dlb/Kconfig
@@ -0,0 +1,18 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+config INTEL_DLB
+ tristate "Intel Dynamic Load Balancer Driver"
+ depends on 64BIT && PCI && X86
+ help
+ This driver supports the Intel Dynamic Load Balancer (DLB), a
+ PCIe device (PCI ID 8086:27xx) that provides load-balanced,
+ prioritized scheduling of core-to-core communication and improves
+ DPDK Event Device library performance.
+
+ The user-space interface is described in
+ include/uapi/linux/dlb_user.h
+
+ To compile this driver as a module, choose M here. The module
+ will be called dlb.
+
+ If unsure, select N.
diff --git a/drivers/misc/dlb/Makefile b/drivers/misc/dlb/Makefile
new file mode 100644
index 000000000000..8911375effd2
--- /dev/null
+++ b/drivers/misc/dlb/Makefile
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+#
+# Makefile for the Intel(R) Dynamic Load Balancer (dlb.ko) driver
+#
+
+obj-$(CONFIG_INTEL_DLB) := dlb.o
+
+dlb-objs := dlb_main.o
diff --git a/drivers/misc/dlb/dlb_hw_types.h b/drivers/misc/dlb/dlb_hw_types.h
new file mode 100644
index 000000000000..778ec8665ea0
--- /dev/null
+++ b/drivers/misc/dlb/dlb_hw_types.h
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#ifndef __DLB_HW_TYPES_H
+#define __DLB_HW_TYPES_H
+
+#define DLB_MAX_NUM_VDEVS 16
+#define DLB_MAX_NUM_DOMAINS 32
+#define DLB_MAX_NUM_LDB_QUEUES 32 /* LDB == load-balanced */
+#define DLB_MAX_NUM_DIR_QUEUES 64 /* DIR == directed */
+#define DLB_MAX_NUM_LDB_PORTS 64
+#define DLB_MAX_NUM_DIR_PORTS DLB_MAX_NUM_DIR_QUEUES
+#define DLB_MAX_NUM_LDB_CREDITS 8192
+#define DLB_MAX_NUM_DIR_CREDITS 2048
+#define DLB_MAX_NUM_HIST_LIST_ENTRIES 2048
+#define DLB_MAX_NUM_AQED_ENTRIES 2048
+#define DLB_MAX_NUM_QIDS_PER_LDB_CQ 8
+#define DLB_MAX_NUM_SEQUENCE_NUMBER_GROUPS 2
+#define DLB_MAX_NUM_SEQUENCE_NUMBER_MODES 5
+#define DLB_QID_PRIORITIES 8
+#define DLB_NUM_ARB_WEIGHTS 8
+#define DLB_MAX_WEIGHT 255
+#define DLB_NUM_COS_DOMAINS 4
+#define DLB_MAX_CQ_COMP_CHECK_LOOPS 409600
+#define DLB_MAX_QID_EMPTY_CHECK_LOOPS (32 * 64 * 1024 * (800 / 30))
+#define DLB_HZ 800000000
+#define DLB_FUNC_BAR 0
+#define DLB_CSR_BAR 2
+
+#define PCI_DEVICE_ID_INTEL_DLB_PF 0x2710
+
+#endif /* __DLB_HW_TYPES_H */
diff --git a/drivers/misc/dlb/dlb_main.c b/drivers/misc/dlb/dlb_main.c
new file mode 100644
index 000000000000..fbd77203b398
--- /dev/null
+++ b/drivers/misc/dlb/dlb_main.c
@@ -0,0 +1,156 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#include <linux/aer.h>
+#include <linux/cdev.h>
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/init.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/uaccess.h>
+
+#include "dlb_main.h"
+
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("Intel(R) Dynamic Load Balancer (DLB) Driver");
+
+static struct class *dlb_class;
+static dev_t dlb_devt;
+static DEFINE_IDR(dlb_ids);
+static DEFINE_SPINLOCK(dlb_ids_lock);
+
+/**********************************/
+/****** PCI driver callbacks ******/
+/**********************************/
+
+static int dlb_probe(struct pci_dev *pdev, const struct pci_device_id *pdev_id)
+{
+ struct dlb *dlb;
+ int ret;
+
+ dlb = devm_kzalloc(&pdev->dev, sizeof(*dlb), GFP_KERNEL);
+ if (!dlb)
+ return -ENOMEM;
+
+ pci_set_drvdata(pdev, dlb);
+
+ dlb->pdev = pdev;
+
+ spin_lock(&dlb_ids_lock);
+ dlb->id = idr_alloc(&dlb_ids, (void *)dlb, 0, DLB_MAX_NUM_DEVICES - 1,
+ GFP_KERNEL);
+ spin_unlock(&dlb_ids_lock);
+
+ if (dlb->id < 0) {
+ dev_err(&pdev->dev, "probe: device ID allocation failed\n");
+
+ ret = dlb->id;
+ goto alloc_id_fail;
+ }
+
+ ret = pcim_enable_device(pdev);
+ if (ret != 0) {
+ dev_err(&pdev->dev, "pcim_enable_device() returned %d\n", ret);
+
+ goto pci_enable_device_fail;
+ }
+
+ ret = pcim_iomap_regions(pdev,
+ (1U << DLB_CSR_BAR) | (1U << DLB_FUNC_BAR),
+ "dlb");
+ if (ret != 0) {
+ dev_err(&pdev->dev, "pcim_iomap_regions(): returned %d\n", ret);
+
+ goto pci_enable_device_fail;
+ }
+
+ pci_set_master(pdev);
+
+ ret = pci_enable_pcie_error_reporting(pdev);
+ if (ret != 0)
+ dev_info(&pdev->dev, "Failed to enable AER %d\n", ret);
+
+ return 0;
+
+pci_enable_device_fail:
+ spin_lock(&dlb_ids_lock);
+ idr_remove(&dlb_ids, dlb->id);
+ spin_unlock(&dlb_ids_lock);
+alloc_id_fail:
+ return ret;
+}
+
+static void dlb_remove(struct pci_dev *pdev)
+{
+ struct dlb *dlb = pci_get_drvdata(pdev);
+
+ pci_disable_pcie_error_reporting(pdev);
+
+ spin_lock(&dlb_ids_lock);
+ idr_remove(&dlb_ids, dlb->id);
+ spin_unlock(&dlb_ids_lock);
+}
+
+static struct pci_device_id dlb_id_table[] = {
+ { PCI_DEVICE_DATA(INTEL, DLB_PF, DLB_PF) },
+ { 0 }
+};
+MODULE_DEVICE_TABLE(pci, dlb_id_table);
+
+static struct pci_driver dlb_pci_driver = {
+ .name = "dlb",
+ .id_table = dlb_id_table,
+ .probe = dlb_probe,
+ .remove = dlb_remove,
+};
+
+static int __init dlb_init_module(void)
+{
+ int err;
+
+ dlb_class = class_create(THIS_MODULE, "dlb");
+
+ if (IS_ERR(dlb_class)) {
+ pr_err("dlb: class_create() returned %ld\n",
+ PTR_ERR(dlb_class));
+
+ return PTR_ERR(dlb_class);
+ }
+
+ err = alloc_chrdev_region(&dlb_devt, 0, DLB_MAX_NUM_DEVICES, "dlb");
+
+ if (err < 0) {
+ pr_err("dlb: alloc_chrdev_region() returned %d\n", err);
+
+ goto alloc_chrdev_fail;
+ }
+
+ err = pci_register_driver(&dlb_pci_driver);
+ if (err < 0) {
+ pr_err("dlb: pci_register_driver() returned %d\n", err);
+
+ goto pci_register_fail;
+ }
+
+ return 0;
+
+pci_register_fail:
+ unregister_chrdev_region(dlb_devt, DLB_MAX_NUM_DEVICES);
+alloc_chrdev_fail:
+ class_destroy(dlb_class);
+
+ return err;
+}
+
+static void __exit dlb_exit_module(void)
+{
+ pci_unregister_driver(&dlb_pci_driver);
+
+ unregister_chrdev_region(dlb_devt, DLB_MAX_NUM_DEVICES);
+
+ class_destroy(dlb_class);
+}
+
+module_init(dlb_init_module);
+module_exit(dlb_exit_module);
diff --git a/drivers/misc/dlb/dlb_main.h b/drivers/misc/dlb/dlb_main.h
new file mode 100644
index 000000000000..ab2a2014bafd
--- /dev/null
+++ b/drivers/misc/dlb/dlb_main.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#ifndef __DLB_MAIN_H
+#define __DLB_MAIN_H
+
+#include <linux/cdev.h>
+#include <linux/device.h>
+#include <linux/ktime.h>
+#include <linux/list.h>
+#include <linux/mutex.h>
+#include <linux/pci.h>
+#include <linux/types.h>
+
+#include "dlb_hw_types.h"
+
+/*
+ * The dlb driver uses a different minor number for each device file, of which
+ * there are:
+ * - 33 per device (PF or VF/VDEV): 1 for the device, 32 for scheduling domains
+ * - Up to 17 devices per PF: 1 PF and up to 16 VFs/VDEVs
+ * - Up to 16 PFs per system
+ */
+#define DLB_MAX_NUM_PFS 16
+#define DLB_NUM_FUNCS_PER_DEVICE (1 + DLB_MAX_NUM_VDEVS)
+#define DLB_MAX_NUM_DEVICES (DLB_MAX_NUM_PFS * DLB_NUM_FUNCS_PER_DEVICE)
+
+enum dlb_device_type {
+ DLB_PF,
+};
+
+struct dlb {
+ struct pci_dev *pdev;
+ int id;
+};
+
+#endif /* __DLB_MAIN_H */
--
2.17.1
Add the register accesses that implement the static queue map operation and
handle an unmap request when a queue map operation is in progress.
If a queue map operation is requested before the domain is started, it is a
synchronous procedure on "static"/unchanging hardware. (The "dynamic"
operation, when traffic is flowing in the device, will be added in a later
commit.)
Signed-off-by: Gage Eads <[email protected]>
Signed-off-by: Mike Ximing Chen <[email protected]>
Reviewed-by: Björn Töpel <[email protected]>
Reviewed-by: Dan Williams <[email protected]>
---
drivers/misc/dlb/dlb_resource.c | 144 +++++++++++++++++++++++++++++++-
1 file changed, 141 insertions(+), 3 deletions(-)
diff --git a/drivers/misc/dlb/dlb_resource.c b/drivers/misc/dlb/dlb_resource.c
index a830b547dadf..95ccb7eddb8b 100644
--- a/drivers/misc/dlb/dlb_resource.c
+++ b/drivers/misc/dlb/dlb_resource.c
@@ -2122,19 +2122,150 @@ static int dlb_configure_dir_port(struct dlb_hw *hw, struct dlb_hw_domain *domai
return 0;
}
+static int dlb_ldb_port_map_qid_static(struct dlb_hw *hw, struct dlb_ldb_port *p,
+ struct dlb_ldb_queue *q, u8 priority)
+{
+ enum dlb_qid_map_state state;
+ u32 lsp_qid2cq2;
+ u32 lsp_qid2cq;
+ u32 atm_qid2cq;
+ u32 cq2priov;
+ u32 cq2qid;
+ int i;
+
+ /* Look for a pending or already mapped slot, else an unused slot */
+ if (!dlb_port_find_slot_queue(p, DLB_QUEUE_MAP_IN_PROG, q, &i) &&
+ !dlb_port_find_slot_queue(p, DLB_QUEUE_MAPPED, q, &i) &&
+ !dlb_port_find_slot(p, DLB_QUEUE_UNMAPPED, &i)) {
+ DLB_HW_ERR(hw,
+ "[%s():%d] Internal error: CQ has no available QID mapping slots\n",
+ __func__, __LINE__);
+ return -EFAULT;
+ }
+
+ /* Read-modify-write the priority and valid bit register */
+ cq2priov = DLB_CSR_RD(hw, LSP_CQ2PRIOV(p->id.phys_id));
+
+ cq2priov |= (1U << (i + LSP_CQ2PRIOV_V_LOC)) & LSP_CQ2PRIOV_V;
+ cq2priov |= ((priority & 0x7) << (i + LSP_CQ2PRIOV_PRIO_LOC) * 3)
+ & LSP_CQ2PRIOV_PRIO;
+
+ DLB_CSR_WR(hw, LSP_CQ2PRIOV(p->id.phys_id), cq2priov);
+
+ /* Read-modify-write the QID map register */
+ if (i < 4)
+ cq2qid = DLB_CSR_RD(hw, LSP_CQ2QID0(p->id.phys_id));
+ else
+ cq2qid = DLB_CSR_RD(hw, LSP_CQ2QID1(p->id.phys_id));
+
+ if (i == 0 || i == 4)
+ BITS_SET(cq2qid, q->id.phys_id, LSP_CQ2QID0_QID_P0);
+ if (i == 1 || i == 5)
+ BITS_SET(cq2qid, q->id.phys_id, LSP_CQ2QID0_QID_P1);
+ if (i == 2 || i == 6)
+ BITS_SET(cq2qid, q->id.phys_id, LSP_CQ2QID0_QID_P2);
+ if (i == 3 || i == 7)
+ BITS_SET(cq2qid, q->id.phys_id, LSP_CQ2QID0_QID_P3);
+
+ if (i < 4)
+ DLB_CSR_WR(hw, LSP_CQ2QID0(p->id.phys_id), cq2qid);
+ else
+ DLB_CSR_WR(hw, LSP_CQ2QID1(p->id.phys_id), cq2qid);
+
+ atm_qid2cq = DLB_CSR_RD(hw,
+ ATM_QID2CQIDIX(q->id.phys_id,
+ p->id.phys_id / 4));
+
+ lsp_qid2cq = DLB_CSR_RD(hw,
+ LSP_QID2CQIDIX(q->id.phys_id,
+ p->id.phys_id / 4));
+
+ lsp_qid2cq2 = DLB_CSR_RD(hw,
+ LSP_QID2CQIDIX2(q->id.phys_id,
+ p->id.phys_id / 4));
+
+ switch (p->id.phys_id % 4) {
+ case 0:
+ atm_qid2cq |= (1 << (i + ATM_QID2CQIDIX_00_CQ_P0_LOC));
+ lsp_qid2cq |= (1 << (i + LSP_QID2CQIDIX_00_CQ_P0_LOC));
+ lsp_qid2cq2 |= (1 << (i + LSP_QID2CQIDIX2_00_CQ_P0_LOC));
+ break;
+
+ case 1:
+ atm_qid2cq |= (1 << (i + ATM_QID2CQIDIX_00_CQ_P1_LOC));
+ lsp_qid2cq |= (1 << (i + LSP_QID2CQIDIX_00_CQ_P1_LOC));
+ lsp_qid2cq2 |= (1 << (i + LSP_QID2CQIDIX2_00_CQ_P1_LOC));
+ break;
+
+ case 2:
+ atm_qid2cq |= (1 << (i + ATM_QID2CQIDIX_00_CQ_P2_LOC));
+ lsp_qid2cq |= (1 << (i + LSP_QID2CQIDIX_00_CQ_P2_LOC));
+ lsp_qid2cq2 |= (1 << (i + LSP_QID2CQIDIX2_00_CQ_P2_LOC));
+ break;
+
+ case 3:
+ atm_qid2cq |= (1 << (i + ATM_QID2CQIDIX_00_CQ_P3_LOC));
+ lsp_qid2cq |= (1 << (i + LSP_QID2CQIDIX_00_CQ_P3_LOC));
+ lsp_qid2cq2 |= (1 << (i + LSP_QID2CQIDIX2_00_CQ_P3_LOC));
+ break;
+ }
+
+ DLB_CSR_WR(hw,
+ ATM_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+ atm_qid2cq);
+
+ DLB_CSR_WR(hw,
+ LSP_QID2CQIDIX(q->id.phys_id, p->id.phys_id / 4),
+ lsp_qid2cq);
+
+ DLB_CSR_WR(hw,
+ LSP_QID2CQIDIX2(q->id.phys_id, p->id.phys_id / 4),
+ lsp_qid2cq2);
+
+ dlb_flush_csr(hw);
+
+ p->qid_map[i].qid = q->id.phys_id;
+ p->qid_map[i].priority = priority;
+
+ state = DLB_QUEUE_MAPPED;
+
+ return dlb_port_slot_state_transition(hw, p, q, i, state);
+}
+
static void dlb_ldb_port_change_qid_priority(struct dlb_hw *hw,
struct dlb_ldb_port *port, int slot,
struct dlb_map_qid_args *args)
{
- /* Placeholder */
+ u32 cq2priov;
+
+ /* Read-modify-write the priority and valid bit register */
+ cq2priov = DLB_CSR_RD(hw, LSP_CQ2PRIOV(port->id.phys_id));
+
+ cq2priov |= (1 << (slot + LSP_CQ2PRIOV_V_LOC)) & LSP_CQ2PRIOV_V;
+ cq2priov |= ((args->priority & 0x7) << slot * 3) & LSP_CQ2PRIOV_PRIO;
+
+ DLB_CSR_WR(hw, LSP_CQ2PRIOV(port->id.phys_id), cq2priov);
+
+ dlb_flush_csr(hw);
+
+ port->qid_map[slot].priority = args->priority;
+}
+
+static void dlb_ldb_queue_set_inflight_limit(struct dlb_hw *hw,
+ struct dlb_ldb_queue *queue)
+{
+ u32 infl_lim = 0;
+
+ BITS_SET(infl_lim, queue->num_qid_inflights, LSP_QID_LDB_INFL_LIM_LIMIT);
+
+ DLB_CSR_WR(hw, LSP_QID_LDB_INFL_LIM(queue->id.phys_id), infl_lim);
}
static int dlb_ldb_port_map_qid(struct dlb_hw *hw, struct dlb_hw_domain *domain,
struct dlb_ldb_port *port,
struct dlb_ldb_queue *queue, u8 prio)
{
- /* Placeholder */
- return 0;
+ return dlb_ldb_port_map_qid_static(hw, port, queue, prio);
}
static void
@@ -2872,6 +3003,13 @@ int dlb_hw_unmap_qid(struct dlb_hw *hw, u32 domain_id,
*/
st = DLB_QUEUE_MAP_IN_PROG;
if (dlb_port_find_slot_queue(port, st, queue, &i)) {
+ /*
+ * Since the in-progress map was aborted, re-enable the QID's
+ * inflights.
+ */
+ if (queue->num_pending_additions == 0)
+ dlb_ldb_queue_set_inflight_limit(hw, queue);
+
st = DLB_QUEUE_UNMAPPED;
ret = dlb_port_slot_state_transition(hw, port, queue, i, st);
if (ret)
--
2.17.1
Add support for configuring a scheduling domain, creating the domain fd,
and reserving the domain's resources.
A scheduling domain serves as a container of DLB resources -- e.g. ports,
queues, and credits -- with the property that a port can only enqueue
to and dequeue from queues within its domain. A scheduling domain is
created on-demand by a user-space application, whose request includes
the DLB resource allocation.
When a user requests to create a scheduling domain, the requested resources
are validated against the number currently available, and then reserved for
the scheduling domain. Finally, the ioctl handler allocates an anonymous
file descriptor for the domain and installs this in the calling process's
file descriptor table.
Once created, user-space can use this file descriptor to configure the
scheduling domain's resources (to be added in a subsequent commit). For
multiprocess applications, this descriptor can be shared over a unix
domain socket.
The driver maintains a reference count for each scheduling domain,
incrementing it each time user-space requests a file descriptor and
decrementing it in the file's release callback.
When the reference count transitions from 1->0 the driver automatically
resets the scheduling domain's resources and makes them available for use
by future applications. This ensures that applications that crash without
explicitly cleaning up do not orphan device resources. The code to perform
the domain reset will be added in subsequent commits.
Signed-off-by: Gage Eads <[email protected]>
Signed-off-by: Mike Ximing Chen <[email protected]>
Reviewed-by: Magnus Karlsson <[email protected]>
Reviewed-by: Dan Williams <[email protected]>
---
drivers/misc/dlb/dlb_bitmap.h | 74 ++++
drivers/misc/dlb/dlb_ioctl.c | 36 +-
drivers/misc/dlb/dlb_main.c | 68 ++++
drivers/misc/dlb/dlb_main.h | 24 ++
drivers/misc/dlb/dlb_pf_ops.c | 5 +-
drivers/misc/dlb/dlb_regs.h | 18 +
drivers/misc/dlb/dlb_resource.c | 605 ++++++++++++++++++++++++++++++++
drivers/misc/dlb/dlb_resource.h | 7 +
include/uapi/linux/dlb.h | 14 +
9 files changed, 845 insertions(+), 6 deletions(-)
diff --git a/drivers/misc/dlb/dlb_bitmap.h b/drivers/misc/dlb/dlb_bitmap.h
index 3ea78b42c79f..5cebf833fab4 100644
--- a/drivers/misc/dlb/dlb_bitmap.h
+++ b/drivers/misc/dlb/dlb_bitmap.h
@@ -73,6 +73,80 @@ static inline void dlb_bitmap_free(struct dlb_bitmap *bitmap)
kfree(bitmap);
}
+/**
+ * dlb_bitmap_clear_range() - clear a range of bitmap entries
+ * @bitmap: pointer to dlb_bitmap structure.
+ * @bit: starting bit index.
+ * @len: length of the range.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise.
+ *
+ * Errors:
+ * EINVAL - bitmap is NULL or is uninitialized, or the range exceeds the bitmap
+ * length.
+ */
+static inline int dlb_bitmap_clear_range(struct dlb_bitmap *bitmap,
+ unsigned int bit,
+ unsigned int len)
+{
+ if (!bitmap || !bitmap->map)
+ return -EINVAL;
+
+ if (bitmap->len <= bit)
+ return -EINVAL;
+
+ bitmap_clear(bitmap->map, bit, len);
+
+ return 0;
+}
+
+/**
+ * dlb_bitmap_find_set_bit_range() - find an range of set bits
+ * @bitmap: pointer to dlb_bitmap structure.
+ * @len: length of the range.
+ *
+ * This function looks for a range of set bits of length @len.
+ *
+ * Return:
+ * Returns the base bit index upon success, < 0 otherwise.
+ *
+ * Errors:
+ * ENOENT - unable to find a length *len* range of set bits.
+ * EINVAL - bitmap is NULL or is uninitialized, or len is invalid.
+ */
+static inline int dlb_bitmap_find_set_bit_range(struct dlb_bitmap *bitmap,
+ unsigned int len)
+{
+ struct dlb_bitmap *complement_mask = NULL;
+ int ret;
+
+ if (!bitmap || !bitmap->map || len == 0)
+ return -EINVAL;
+
+ if (bitmap->len < len)
+ return -ENOENT;
+
+ ret = dlb_bitmap_alloc(&complement_mask, bitmap->len);
+ if (ret)
+ return ret;
+
+ bitmap_zero(complement_mask->map, complement_mask->len);
+
+ bitmap_complement(complement_mask->map, bitmap->map, bitmap->len);
+
+ ret = bitmap_find_next_zero_area(complement_mask->map,
+ complement_mask->len,
+ 0,
+ len,
+ 0);
+
+ dlb_bitmap_free(complement_mask);
+
+ /* No set bit range of length len? */
+ return (ret >= (int)bitmap->len) ? -ENOENT : ret;
+}
+
/**
* dlb_bitmap_longest_set_range() - returns longest contiguous range of set
* bits
diff --git a/drivers/misc/dlb/dlb_ioctl.c b/drivers/misc/dlb/dlb_ioctl.c
index 47d6cab773d4..7871d0cea118 100644
--- a/drivers/misc/dlb/dlb_ioctl.c
+++ b/drivers/misc/dlb/dlb_ioctl.c
@@ -1,6 +1,7 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+#include <linux/anon_inodes.h>
#include <linux/uaccess.h>
#include <uapi/linux/dlb.h>
@@ -45,7 +46,10 @@ static int dlb_ioctl_create_sched_domain(struct dlb *dlb, unsigned long user_arg
struct dlb_create_sched_domain_args __user *uarg;
struct dlb_create_sched_domain_args arg;
struct dlb_cmd_response response = {0};
- int ret;
+ struct dlb_domain *domain;
+ u32 flags = O_RDONLY;
+ size_t offset;
+ int ret, fd;
uarg = (void __user *)user_arg;
if (copy_from_user(&arg, uarg, sizeof(arg)))
@@ -54,10 +58,38 @@ static int dlb_ioctl_create_sched_domain(struct dlb *dlb, unsigned long user_arg
mutex_lock(&dlb->resource_mutex);
ret = dlb->ops->create_sched_domain(&dlb->hw, &arg, &response);
+ if (ret)
+ goto unlock;
+
+ ret = dlb_init_domain(dlb, response.id);
+ if (ret)
+ goto unlock;
+
+ domain = dlb->sched_domains[response.id];
+
+ if (dlb->f->f_mode & FMODE_WRITE)
+ flags = O_RDWR;
+
+ fd = anon_inode_getfd("[dlbdomain]", &dlb_domain_fops,
+ domain, flags);
+
+ if (fd < 0) {
+ dev_err(dlb->dev, "Failed to get anon fd.\n");
+ kref_put(&domain->refcnt, dlb_free_domain);
+ ret = fd;
+ goto unlock;
+ }
+
+ offset = offsetof(struct dlb_create_sched_domain_args, domain_fd);
+
+ if (copy_to_user((void __user *)(user_arg + offset), &fd, sizeof(fd))) {
+ mutex_unlock(&dlb->resource_mutex);
+ return -EFAULT;
+ }
+unlock:
mutex_unlock(&dlb->resource_mutex);
- response.status = ret;
BUILD_BUG_ON(offsetof(typeof(arg), response) != 0);
if (copy_to_user((void __user *)&uarg->response, &response, sizeof(response)))
diff --git a/drivers/misc/dlb/dlb_main.c b/drivers/misc/dlb/dlb_main.c
index d92956b1643d..a4ed413eee2f 100644
--- a/drivers/misc/dlb/dlb_main.c
+++ b/drivers/misc/dlb/dlb_main.c
@@ -62,11 +62,79 @@ static int dlb_device_create(struct dlb *dlb, struct pci_dev *pdev)
/****** Char dev callbacks ******/
/********************************/
+static int dlb_open(struct inode *i, struct file *f)
+{
+ struct dlb *dlb;
+
+ spin_lock(&dlb_ids_lock);
+ dlb = idr_find(&dlb_ids, iminor(i));
+ spin_unlock(&dlb_ids_lock);
+
+ f->private_data = dlb;
+ dlb->f = f;
+
+ return 0;
+}
+
static const struct file_operations dlb_fops = {
.owner = THIS_MODULE,
+ .open = dlb_open,
.unlocked_ioctl = dlb_ioctl,
};
+int dlb_init_domain(struct dlb *dlb, u32 domain_id)
+{
+ struct dlb_domain *domain;
+
+ domain = kzalloc(sizeof(*domain), GFP_KERNEL);
+ if (!domain)
+ return -ENOMEM;
+
+ domain->id = domain_id;
+
+ kref_init(&domain->refcnt);
+ domain->dlb = dlb;
+
+ dlb->sched_domains[domain_id] = domain;
+
+ return 0;
+}
+
+static int __dlb_free_domain(struct dlb_domain *domain)
+{
+ struct dlb *dlb = domain->dlb;
+
+ dlb->sched_domains[domain->id] = NULL;
+
+ kfree(domain);
+
+ return 0;
+}
+
+void dlb_free_domain(struct kref *kref)
+{
+ __dlb_free_domain(container_of(kref, struct dlb_domain, refcnt));
+}
+
+static int dlb_domain_close(struct inode *i, struct file *f)
+{
+ struct dlb_domain *domain = f->private_data;
+ struct dlb *dlb = domain->dlb;
+
+ mutex_lock(&dlb->resource_mutex);
+
+ kref_put(&domain->refcnt, dlb_free_domain);
+
+ mutex_unlock(&dlb->resource_mutex);
+
+ return 0;
+}
+
+const struct file_operations dlb_domain_fops = {
+ .owner = THIS_MODULE,
+ .release = dlb_domain_close,
+};
+
/**********************************/
/****** PCI driver callbacks ******/
/**********************************/
diff --git a/drivers/misc/dlb/dlb_main.h b/drivers/misc/dlb/dlb_main.h
index 3089a66a3560..824416e6cdcf 100644
--- a/drivers/misc/dlb/dlb_main.h
+++ b/drivers/misc/dlb/dlb_main.h
@@ -47,12 +47,21 @@ struct dlb_device_ops {
};
extern struct dlb_device_ops dlb_pf_ops;
+extern const struct file_operations dlb_domain_fops;
+
+struct dlb_domain {
+ struct dlb *dlb;
+ struct kref refcnt;
+ u8 id;
+};
struct dlb {
struct pci_dev *pdev;
struct dlb_hw hw;
struct dlb_device_ops *ops;
struct device *dev;
+ struct dlb_domain *sched_domains[DLB_MAX_NUM_DOMAINS];
+ struct file *f;
/*
* The resource mutex serializes access to driver data structures and
* hardware registers.
@@ -66,4 +75,19 @@ struct dlb {
/* Prototypes for dlb_ioctl.c */
long dlb_ioctl(struct file *f, unsigned int cmd, unsigned long arg);
+int dlb_init_domain(struct dlb *dlb, u32 domain_id);
+void dlb_free_domain(struct kref *kref);
+
+#define DLB_HW_ERR(hw, ...) do { \
+ struct dlb *dlb; \
+ dlb = container_of(hw, struct dlb, hw); \
+ dev_err(dlb->dev, __VA_ARGS__); \
+} while (0)
+
+#define DLB_HW_DBG(hw, ...) do { \
+ struct dlb *dlb; \
+ dlb = container_of(hw, struct dlb, hw); \
+ dev_dbg(dlb->dev, __VA_ARGS__); \
+} while (0)
+
#endif /* __DLB_MAIN_H */
diff --git a/drivers/misc/dlb/dlb_pf_ops.c b/drivers/misc/dlb/dlb_pf_ops.c
index 125ef6fe6c70..b59e9eaa600d 100644
--- a/drivers/misc/dlb/dlb_pf_ops.c
+++ b/drivers/misc/dlb/dlb_pf_ops.c
@@ -103,10 +103,7 @@ static int dlb_pf_create_sched_domain(struct dlb_hw *hw,
struct dlb_create_sched_domain_args *args,
struct dlb_cmd_response *resp)
{
- resp->id = 0;
- resp->status = 0;
-
- return 0;
+ return dlb_hw_create_sched_domain(hw, args, resp, false, 0);
}
static int dlb_pf_get_num_resources(struct dlb_hw *hw,
diff --git a/drivers/misc/dlb/dlb_regs.h b/drivers/misc/dlb/dlb_regs.h
index 72f3cb22b933..0fd499f384de 100644
--- a/drivers/misc/dlb/dlb_regs.h
+++ b/drivers/misc/dlb/dlb_regs.h
@@ -6,6 +6,24 @@
#include <linux/types.h>
+#define CHP_CFG_DIR_VAS_CRD(x) \
+ (0x40000000 + (x) * 0x1000)
+#define CHP_CFG_DIR_VAS_CRD_RST 0x0
+
+#define CHP_CFG_DIR_VAS_CRD_COUNT 0x00003FFF
+#define CHP_CFG_DIR_VAS_CRD_RSVD0 0xFFFFC000
+#define CHP_CFG_DIR_VAS_CRD_COUNT_LOC 0
+#define CHP_CFG_DIR_VAS_CRD_RSVD0_LOC 14
+
+#define CHP_CFG_LDB_VAS_CRD(x) \
+ (0x40080000 + (x) * 0x1000)
+#define CHP_CFG_LDB_VAS_CRD_RST 0x0
+
+#define CHP_CFG_LDB_VAS_CRD_COUNT 0x00007FFF
+#define CHP_CFG_LDB_VAS_CRD_RSVD0 0xFFFF8000
+#define CHP_CFG_LDB_VAS_CRD_COUNT_LOC 0
+#define CHP_CFG_LDB_VAS_CRD_RSVD0_LOC 15
+
#define CM_CFG_DIAGNOSTIC_IDLE_STATUS 0xb4000004
#define CM_CFG_DIAGNOSTIC_IDLE_STATUS_RST 0x9d0fffff
diff --git a/drivers/misc/dlb/dlb_resource.c b/drivers/misc/dlb/dlb_resource.c
index 9d75b12eb793..b7df23c6a158 100644
--- a/drivers/misc/dlb/dlb_resource.c
+++ b/drivers/misc/dlb/dlb_resource.c
@@ -3,6 +3,7 @@
#include "dlb_bitmap.h"
#include "dlb_hw_types.h"
+#include "dlb_main.h"
#include "dlb_regs.h"
#include "dlb_resource.h"
@@ -200,6 +201,610 @@ int dlb_resource_init(struct dlb_hw *hw)
return ret;
}
+static int dlb_attach_ldb_queues(struct dlb_hw *hw,
+ struct dlb_function_resources *rsrcs,
+ struct dlb_hw_domain *domain, u32 num_queues,
+ struct dlb_cmd_response *resp)
+{
+ unsigned int i;
+
+ if (rsrcs->num_avail_ldb_queues < num_queues) {
+ resp->status = DLB_ST_LDB_QUEUES_UNAVAILABLE;
+ DLB_HW_DBG(hw, "[%s()] Internal error: %d\n", __func__,
+ resp->status);
+ return -EINVAL;
+ }
+
+ for (i = 0; i < num_queues; i++) {
+ struct dlb_ldb_queue *queue;
+
+ queue = list_first_entry_or_null(&rsrcs->avail_ldb_queues,
+ typeof(*queue), func_list);
+ if (!queue) {
+ DLB_HW_ERR(hw,
+ "[%s()] Internal error: domain validation failed\n",
+ __func__);
+ return -EFAULT;
+ }
+
+ list_del(&queue->func_list);
+
+ queue->domain_id = domain->id;
+ queue->owned = true;
+
+ list_add(&queue->domain_list, &domain->avail_ldb_queues);
+ }
+
+ rsrcs->num_avail_ldb_queues -= num_queues;
+
+ return 0;
+}
+
+static struct dlb_ldb_port *
+dlb_get_next_ldb_port(struct dlb_hw *hw, struct dlb_function_resources *rsrcs,
+ u32 domain_id, u32 cos_id)
+{
+ struct dlb_ldb_port *port;
+
+ /*
+ * To reduce the odds of consecutive load-balanced ports mapping to the
+ * same queue(s), the driver attempts to allocate ports whose neighbors
+ * are owned by a different domain.
+ */
+ list_for_each_entry(port, &rsrcs->avail_ldb_ports[cos_id], func_list) {
+ u32 next, prev;
+ u32 phys_id;
+
+ phys_id = port->id.phys_id;
+ next = phys_id + 1;
+ prev = phys_id - 1;
+
+ if (phys_id == DLB_MAX_NUM_LDB_PORTS - 1)
+ next = 0;
+ if (phys_id == 0)
+ prev = DLB_MAX_NUM_LDB_PORTS - 1;
+
+ if (!hw->rsrcs.ldb_ports[next].owned ||
+ hw->rsrcs.ldb_ports[next].domain_id.phys_id == domain_id)
+ continue;
+
+ if (!hw->rsrcs.ldb_ports[prev].owned ||
+ hw->rsrcs.ldb_ports[prev].domain_id.phys_id == domain_id)
+ continue;
+
+ return port;
+ }
+
+ /*
+ * Failing that, the driver looks for a port with one neighbor owned by
+ * a different domain and the other unallocated.
+ */
+ list_for_each_entry(port, &rsrcs->avail_ldb_ports[cos_id], func_list) {
+ u32 next, prev;
+ u32 phys_id;
+
+ phys_id = port->id.phys_id;
+ next = phys_id + 1;
+ prev = phys_id - 1;
+
+ if (phys_id == DLB_MAX_NUM_LDB_PORTS - 1)
+ next = 0;
+ if (phys_id == 0)
+ prev = DLB_MAX_NUM_LDB_PORTS - 1;
+
+ if (!hw->rsrcs.ldb_ports[prev].owned &&
+ hw->rsrcs.ldb_ports[next].owned &&
+ hw->rsrcs.ldb_ports[next].domain_id.phys_id != domain_id)
+ return port;
+
+ if (!hw->rsrcs.ldb_ports[next].owned &&
+ hw->rsrcs.ldb_ports[prev].owned &&
+ hw->rsrcs.ldb_ports[prev].domain_id.phys_id != domain_id)
+ return port;
+ }
+
+ /*
+ * Failing that, the driver looks for a port with both neighbors
+ * unallocated.
+ */
+ list_for_each_entry(port, &rsrcs->avail_ldb_ports[cos_id], func_list) {
+ u32 next, prev;
+ u32 phys_id;
+
+ phys_id = port->id.phys_id;
+ next = phys_id + 1;
+ prev = phys_id - 1;
+
+ if (phys_id == DLB_MAX_NUM_LDB_PORTS - 1)
+ next = 0;
+ if (phys_id == 0)
+ prev = DLB_MAX_NUM_LDB_PORTS - 1;
+
+ if (!hw->rsrcs.ldb_ports[prev].owned &&
+ !hw->rsrcs.ldb_ports[next].owned)
+ return port;
+ }
+
+ /* If all else fails, the driver returns the next available port. */
+ return list_first_entry_or_null(&rsrcs->avail_ldb_ports[cos_id],
+ typeof(*port), func_list);
+}
+
+static int __dlb_attach_ldb_ports(struct dlb_hw *hw,
+ struct dlb_function_resources *rsrcs,
+ struct dlb_hw_domain *domain, u32 num_ports,
+ u32 cos_id, struct dlb_cmd_response *resp)
+{
+ unsigned int i;
+
+ if (rsrcs->num_avail_ldb_ports[cos_id] < num_ports) {
+ resp->status = DLB_ST_LDB_PORTS_UNAVAILABLE;
+ DLB_HW_DBG(hw,
+ "[%s()] Internal error: %d\n",
+ __func__,
+ resp->status);
+ return -EINVAL;
+ }
+
+ for (i = 0; i < num_ports; i++) {
+ struct dlb_ldb_port *port;
+
+ port = dlb_get_next_ldb_port(hw, rsrcs,
+ domain->id.phys_id, cos_id);
+ if (!port) {
+ DLB_HW_ERR(hw,
+ "[%s()] Internal error: domain validation failed\n",
+ __func__);
+ return -EFAULT;
+ }
+
+ list_del(&port->func_list);
+
+ port->domain_id = domain->id;
+ port->owned = true;
+
+ list_add(&port->domain_list,
+ &domain->avail_ldb_ports[cos_id]);
+ }
+
+ rsrcs->num_avail_ldb_ports[cos_id] -= num_ports;
+
+ return 0;
+}
+
+static int dlb_attach_ldb_ports(struct dlb_hw *hw,
+ struct dlb_function_resources *rsrcs,
+ struct dlb_hw_domain *domain,
+ struct dlb_create_sched_domain_args *args,
+ struct dlb_cmd_response *resp)
+{
+ unsigned int i, j;
+ int ret;
+
+ if (args->cos_strict) {
+ for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+ u32 num = args->num_cos_ldb_ports[i];
+
+ /* Allocate ports from specific classes-of-service */
+ ret = __dlb_attach_ldb_ports(hw, rsrcs, domain, num,
+ i, resp);
+ if (ret)
+ return ret;
+ }
+ } else {
+ unsigned int k;
+ u32 cos_id;
+
+ /*
+ * Attempt to allocate from specific class-of-service, but
+ * fallback to the other classes if that fails.
+ */
+ for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+ for (j = 0; j < args->num_cos_ldb_ports[i]; j++) {
+ for (k = 0; k < DLB_NUM_COS_DOMAINS; k++) {
+ cos_id = (i + k) % DLB_NUM_COS_DOMAINS;
+
+ ret = __dlb_attach_ldb_ports(hw, rsrcs, domain,
+ 1, cos_id, resp);
+ if (ret == 0)
+ break;
+ }
+
+ if (ret)
+ return ret;
+ }
+ }
+ }
+
+ /* Allocate num_ldb_ports from any class-of-service */
+ for (i = 0; i < args->num_ldb_ports; i++) {
+ for (j = 0; j < DLB_NUM_COS_DOMAINS; j++) {
+ ret = __dlb_attach_ldb_ports(hw, rsrcs, domain, 1, j, resp);
+ if (ret == 0)
+ break;
+ }
+
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+static int dlb_attach_dir_ports(struct dlb_hw *hw,
+ struct dlb_function_resources *rsrcs,
+ struct dlb_hw_domain *domain, u32 num_ports,
+ struct dlb_cmd_response *resp)
+{
+ unsigned int i;
+
+ if (rsrcs->num_avail_dir_pq_pairs < num_ports) {
+ resp->status = DLB_ST_DIR_PORTS_UNAVAILABLE;
+ DLB_HW_DBG(hw,
+ "[%s()] Internal error: %d\n",
+ __func__,
+ resp->status);
+ return -EINVAL;
+ }
+
+ for (i = 0; i < num_ports; i++) {
+ struct dlb_dir_pq_pair *port;
+
+ port = list_first_entry_or_null(&rsrcs->avail_dir_pq_pairs,
+ typeof(*port), func_list);
+ if (!port) {
+ DLB_HW_ERR(hw,
+ "[%s()] Internal error: domain validation failed\n",
+ __func__);
+ return -EFAULT;
+ }
+
+ list_del(&port->func_list);
+
+ port->domain_id = domain->id;
+ port->owned = true;
+
+ list_add(&port->domain_list, &domain->avail_dir_pq_pairs);
+ }
+
+ rsrcs->num_avail_dir_pq_pairs -= num_ports;
+
+ return 0;
+}
+
+static int dlb_attach_ldb_credits(struct dlb_function_resources *rsrcs,
+ struct dlb_hw_domain *domain, u32 num_credits,
+ struct dlb_cmd_response *resp)
+{
+ if (rsrcs->num_avail_qed_entries < num_credits) {
+ resp->status = DLB_ST_LDB_CREDITS_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ rsrcs->num_avail_qed_entries -= num_credits;
+ domain->num_ldb_credits += num_credits;
+ return 0;
+}
+
+static int dlb_attach_dir_credits(struct dlb_function_resources *rsrcs,
+ struct dlb_hw_domain *domain, u32 num_credits,
+ struct dlb_cmd_response *resp)
+{
+ if (rsrcs->num_avail_dqed_entries < num_credits) {
+ resp->status = DLB_ST_DIR_CREDITS_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ rsrcs->num_avail_dqed_entries -= num_credits;
+ domain->num_dir_credits += num_credits;
+ return 0;
+}
+
+static int dlb_attach_atomic_inflights(struct dlb_function_resources *rsrcs,
+ struct dlb_hw_domain *domain,
+ u32 num_atomic_inflights,
+ struct dlb_cmd_response *resp)
+{
+ if (rsrcs->num_avail_aqed_entries < num_atomic_inflights) {
+ resp->status = DLB_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ rsrcs->num_avail_aqed_entries -= num_atomic_inflights;
+ domain->num_avail_aqed_entries += num_atomic_inflights;
+ return 0;
+}
+
+static int
+dlb_attach_domain_hist_list_entries(struct dlb_function_resources *rsrcs,
+ struct dlb_hw_domain *domain,
+ u32 num_hist_list_entries,
+ struct dlb_cmd_response *resp)
+{
+ struct dlb_bitmap *bitmap;
+ int base;
+
+ if (num_hist_list_entries) {
+ bitmap = rsrcs->avail_hist_list_entries;
+
+ base = dlb_bitmap_find_set_bit_range(bitmap,
+ num_hist_list_entries);
+ if (base < 0)
+ goto error;
+
+ domain->total_hist_list_entries = num_hist_list_entries;
+ domain->avail_hist_list_entries = num_hist_list_entries;
+ domain->hist_list_entry_base = base;
+ domain->hist_list_entry_offset = 0;
+
+ dlb_bitmap_clear_range(bitmap, base, num_hist_list_entries);
+ }
+ return 0;
+
+error:
+ resp->status = DLB_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+ return -EINVAL;
+}
+
+static int
+dlb_verify_create_sched_dom_args(struct dlb_function_resources *rsrcs,
+ struct dlb_create_sched_domain_args *args,
+ struct dlb_cmd_response *resp,
+ struct dlb_hw_domain **out_domain)
+{
+ u32 num_avail_ldb_ports, req_ldb_ports;
+ struct dlb_bitmap *avail_hl_entries;
+ unsigned int max_contig_hl_range;
+ struct dlb_hw_domain *domain;
+ int i;
+
+ avail_hl_entries = rsrcs->avail_hist_list_entries;
+
+ max_contig_hl_range = dlb_bitmap_longest_set_range(avail_hl_entries);
+
+ num_avail_ldb_ports = 0;
+ req_ldb_ports = 0;
+ for (i = 0; i < DLB_NUM_COS_DOMAINS; i++) {
+ num_avail_ldb_ports += rsrcs->num_avail_ldb_ports[i];
+
+ req_ldb_ports += args->num_cos_ldb_ports[i];
+ }
+
+ req_ldb_ports += args->num_ldb_ports;
+
+ if (rsrcs->num_avail_domains < 1) {
+ resp->status = DLB_ST_DOMAIN_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ domain = list_first_entry_or_null(&rsrcs->avail_domains,
+ typeof(*domain), func_list);
+ if (!domain) {
+ resp->status = DLB_ST_DOMAIN_UNAVAILABLE;
+ return -EFAULT;
+ }
+
+ if (rsrcs->num_avail_ldb_queues < args->num_ldb_queues) {
+ resp->status = DLB_ST_LDB_QUEUES_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ if (req_ldb_ports > num_avail_ldb_ports) {
+ resp->status = DLB_ST_LDB_PORTS_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ for (i = 0; args->cos_strict && i < DLB_NUM_COS_DOMAINS; i++) {
+ if (args->num_cos_ldb_ports[i] >
+ rsrcs->num_avail_ldb_ports[i]) {
+ resp->status = DLB_ST_LDB_PORTS_UNAVAILABLE;
+ return -EINVAL;
+ }
+ }
+
+ if (args->num_ldb_queues > 0 && req_ldb_ports == 0) {
+ resp->status = DLB_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES;
+ return -EINVAL;
+ }
+
+ if (rsrcs->num_avail_dir_pq_pairs < args->num_dir_ports) {
+ resp->status = DLB_ST_DIR_PORTS_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ if (rsrcs->num_avail_qed_entries < args->num_ldb_credits) {
+ resp->status = DLB_ST_LDB_CREDITS_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ if (rsrcs->num_avail_dqed_entries < args->num_dir_credits) {
+ resp->status = DLB_ST_DIR_CREDITS_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ if (rsrcs->num_avail_aqed_entries < args->num_atomic_inflights) {
+ resp->status = DLB_ST_ATOMIC_INFLIGHTS_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ if (max_contig_hl_range < args->num_hist_list_entries) {
+ resp->status = DLB_ST_HIST_LIST_ENTRIES_UNAVAILABLE;
+ return -EINVAL;
+ }
+
+ *out_domain = domain;
+
+ return 0;
+}
+
+static void dlb_configure_domain_credits(struct dlb_hw *hw,
+ struct dlb_hw_domain *domain)
+{
+ u32 reg = 0;
+
+ BITS_SET(reg, domain->num_ldb_credits, CHP_CFG_LDB_VAS_CRD_COUNT);
+ DLB_CSR_WR(hw, CHP_CFG_LDB_VAS_CRD(domain->id.phys_id), reg);
+
+ reg = 0;
+ BITS_SET(reg, domain->num_dir_credits, CHP_CFG_DIR_VAS_CRD_COUNT);
+ DLB_CSR_WR(hw, CHP_CFG_DIR_VAS_CRD(domain->id.phys_id), reg);
+}
+
+static int
+dlb_domain_attach_resources(struct dlb_hw *hw,
+ struct dlb_function_resources *rsrcs,
+ struct dlb_hw_domain *domain,
+ struct dlb_create_sched_domain_args *args,
+ struct dlb_cmd_response *resp)
+{
+ int ret;
+
+ ret = dlb_attach_ldb_queues(hw, rsrcs, domain, args->num_ldb_queues, resp);
+ if (ret)
+ return ret;
+
+ ret = dlb_attach_ldb_ports(hw, rsrcs, domain, args, resp);
+ if (ret)
+ return ret;
+
+ ret = dlb_attach_dir_ports(hw, rsrcs, domain, args->num_dir_ports, resp);
+ if (ret)
+ return ret;
+
+ ret = dlb_attach_ldb_credits(rsrcs, domain,
+ args->num_ldb_credits, resp);
+ if (ret)
+ return ret;
+
+ ret = dlb_attach_dir_credits(rsrcs, domain, args->num_dir_credits, resp);
+ if (ret)
+ return ret;
+
+ ret = dlb_attach_domain_hist_list_entries(rsrcs, domain,
+ args->num_hist_list_entries,
+ resp);
+ if (ret)
+ return ret;
+
+ ret = dlb_attach_atomic_inflights(rsrcs, domain,
+ args->num_atomic_inflights, resp);
+ if (ret)
+ return ret;
+
+ dlb_configure_domain_credits(hw, domain);
+
+ domain->configured = true;
+
+ domain->started = false;
+
+ rsrcs->num_avail_domains--;
+
+ return 0;
+}
+
+static void
+dlb_log_create_sched_domain_args(struct dlb_hw *hw,
+ struct dlb_create_sched_domain_args *args,
+ bool vdev_req, unsigned int vdev_id)
+{
+ DLB_HW_DBG(hw, "DLB create sched domain arguments:\n");
+ if (vdev_req)
+ DLB_HW_DBG(hw, "(Request from vdev %d)\n", vdev_id);
+ DLB_HW_DBG(hw, "\tNumber of LDB queues: %d\n",
+ args->num_ldb_queues);
+ DLB_HW_DBG(hw, "\tNumber of LDB ports (any CoS): %d\n",
+ args->num_ldb_ports);
+ DLB_HW_DBG(hw, "\tNumber of LDB ports (CoS 0): %d\n",
+ args->num_cos_ldb_ports[0]);
+ DLB_HW_DBG(hw, "\tNumber of LDB ports (CoS 1): %d\n",
+ args->num_cos_ldb_ports[1]);
+ DLB_HW_DBG(hw, "\tNumber of LDB ports (CoS 2): %d\n",
+ args->num_cos_ldb_ports[2]);
+ DLB_HW_DBG(hw, "\tNumber of LDB ports (CoS 3): %d\n",
+ args->num_cos_ldb_ports[3]);
+ DLB_HW_DBG(hw, "\tStrict CoS allocation: %d\n",
+ args->cos_strict);
+ DLB_HW_DBG(hw, "\tNumber of DIR ports: %d\n",
+ args->num_dir_ports);
+ DLB_HW_DBG(hw, "\tNumber of ATM inflights: %d\n",
+ args->num_atomic_inflights);
+ DLB_HW_DBG(hw, "\tNumber of hist list entries: %d\n",
+ args->num_hist_list_entries);
+ DLB_HW_DBG(hw, "\tNumber of LDB credits: %d\n",
+ args->num_ldb_credits);
+ DLB_HW_DBG(hw, "\tNumber of DIR credits: %d\n",
+ args->num_dir_credits);
+}
+
+/**
+ * dlb_hw_create_sched_domain() - create a scheduling domain
+ * @hw: dlb_hw handle for a particular device.
+ * @args: scheduling domain creation arguments.
+ * @resp: response structure.
+ * @vdev_req: indicates whether this request came from a vdev.
+ * @vdev_id: If vdev_req is true, this contains the vdev's ID.
+ *
+ * This function creates a scheduling domain containing the resources specified
+ * in args. The individual resources (queues, ports, credits) can be configured
+ * after creating a scheduling domain.
+ *
+ * A vdev can be either an SR-IOV virtual function or a Scalable IOV virtual
+ * device.
+ *
+ * Return:
+ * Returns 0 upon success, < 0 otherwise. If an error occurs, resp->status is
+ * assigned a detailed error code from enum dlb_error. If successful, resp->id
+ * contains the domain ID.
+ *
+ * resp->id contains a virtual ID if vdev_req is true.
+ *
+ * Errors:
+ * EINVAL - A requested resource is unavailable, or the requested domain name
+ * is already in use.
+ * EFAULT - Internal error (resp->status not set).
+ */
+int dlb_hw_create_sched_domain(struct dlb_hw *hw,
+ struct dlb_create_sched_domain_args *args,
+ struct dlb_cmd_response *resp,
+ bool vdev_req, unsigned int vdev_id)
+{
+ struct dlb_function_resources *rsrcs;
+ struct dlb_hw_domain *domain;
+ int ret;
+
+ rsrcs = (vdev_req) ? &hw->vdev[vdev_id] : &hw->pf;
+
+ dlb_log_create_sched_domain_args(hw, args, vdev_req, vdev_id);
+
+ /*
+ * Verify that hardware resources are available before attempting to
+ * satisfy the request. This simplifies the error unwinding code.
+ */
+ ret = dlb_verify_create_sched_dom_args(rsrcs, args, resp, &domain);
+ if (ret)
+ return ret;
+
+ dlb_init_domain_rsrc_lists(domain);
+
+ ret = dlb_domain_attach_resources(hw, rsrcs, domain, args, resp);
+ if (ret) {
+ DLB_HW_ERR(hw,
+ "[%s()] Internal error: failed to verify args.\n",
+ __func__);
+
+ return ret;
+ }
+
+ list_del(&domain->func_list);
+
+ list_add(&domain->func_list, &rsrcs->used_domains);
+
+ resp->id = (vdev_req) ? domain->id.virt_id : domain->id.phys_id;
+ resp->status = 0;
+
+ return 0;
+}
+
/**
* dlb_hw_get_num_resources() - query the PCI function's available resources
* @hw: dlb_hw handle for a particular device.
diff --git a/drivers/misc/dlb/dlb_resource.h b/drivers/misc/dlb/dlb_resource.h
index 3e6d419796bc..efc5140970cd 100644
--- a/drivers/misc/dlb/dlb_resource.h
+++ b/drivers/misc/dlb/dlb_resource.h
@@ -6,12 +6,19 @@
#include <linux/types.h>
+#include <uapi/linux/dlb.h>
+
#include "dlb_hw_types.h"
int dlb_resource_init(struct dlb_hw *hw);
void dlb_resource_free(struct dlb_hw *hw);
+int dlb_hw_create_sched_domain(struct dlb_hw *hw,
+ struct dlb_create_sched_domain_args *args,
+ struct dlb_cmd_response *resp,
+ bool vdev_req, unsigned int vdev_id);
+
int dlb_hw_get_num_resources(struct dlb_hw *hw,
struct dlb_get_num_resources_args *arg,
bool vdev_req, unsigned int vdev_id);
diff --git a/include/uapi/linux/dlb.h b/include/uapi/linux/dlb.h
index b31c67de7fb4..9df80970eb50 100644
--- a/include/uapi/linux/dlb.h
+++ b/include/uapi/linux/dlb.h
@@ -6,6 +6,20 @@
#include <linux/types.h>
+enum dlb_error {
+ DLB_ST_SUCCESS = 0,
+ DLB_ST_DOMAIN_UNAVAILABLE,
+ DLB_ST_LDB_PORTS_UNAVAILABLE,
+ DLB_ST_DIR_PORTS_UNAVAILABLE,
+ DLB_ST_LDB_QUEUES_UNAVAILABLE,
+ DLB_ST_LDB_CREDITS_UNAVAILABLE,
+ DLB_ST_DIR_CREDITS_UNAVAILABLE,
+ DLB_ST_SEQUENCE_NUMBERS_UNAVAILABLE,
+ DLB_ST_ATOMIC_INFLIGHTS_UNAVAILABLE,
+ DLB_ST_HIST_LIST_ENTRIES_UNAVAILABLE,
+ DLB_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES,
+};
+
struct dlb_cmd_response {
__u32 status; /* Interpret using enum dlb_error */
__u32 id;
--
2.17.1
Hi Mike,
I love your patch! Yet something to improve:
[auto build test ERROR on soc/arm/dt]
[also build test ERROR on linus/master char-misc/char-misc-testing linux/master soc/for-next v5.11-rc5 next-20210125]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Mike-Ximing-Chen/dlb-introduce-DLB-device-driver/20210125-155257
base: https://git.kernel.org/pub/scm/linux/kernel/git/soc/soc.git arm/dt
config: x86_64-rhel (attached as .config)
compiler: gcc-9 (Debian 9.3.0-15) 9.3.0
reproduce (this is a W=1 build):
# https://github.com/0day-ci/linux/commit/693738625d24e95e7cf8f92c937f5981b3dad3d9
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Mike-Ximing-Chen/dlb-introduce-DLB-device-driver/20210125-155257
git checkout 693738625d24e95e7cf8f92c937f5981b3dad3d9
# save the attached .config to linux build tree
make W=1 ARCH=x86_64
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <[email protected]>
All errors (new ones prefixed by >>):
>> error: include/uapi/linux/dlb.h: missing "WITH Linux-syscall-note" for SPDX-License-Identifier
make[2]: *** [scripts/Makefile.headersinst:63: usr/include/linux/dlb.h] Error 1
make[2]: Target '__headers' not remade because of errors.
make[1]: *** [Makefile:1294: headers] Error 2
make[1]: Target 'headers_install' not remade because of errors.
make: *** [Makefile:185: __sub-make] Error 2
make: Target 'headers_install' not remade because of errors.
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/[email protected]
On Fri, Jan 22, 2021 at 01:01:22PM -0600, Mike Ximing Chen wrote:
> --- /dev/null
> +++ b/include/uapi/linux/dlb.h
> @@ -0,0 +1,167 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
As the bot points out, this is an "odd" license for a uapi .h file, are
you SURE about this?
If so, I need an Intel lawyer's signed-off-by on it as well, so we know
to talk to in the future about it.
thanks,
greg k-h
> -----Original Message-----
> From: Greg KH <[email protected]>
> Sent: Wednesday, January 27, 2021 7:29 AM
> To: Chen, Mike Ximing <[email protected]>
> Cc: [email protected]; [email protected]; Williams, Dan J
> <[email protected]>; [email protected]; Gage Eads
> <[email protected]>
> Subject: Re: [PATCH v9 04/20] dlb: add device ioctl layer and first three ioctls
>
> On Fri, Jan 22, 2021 at 01:01:22PM -0600, Mike Ximing Chen wrote:
> > --- /dev/null
> > +++ b/include/uapi/linux/dlb.h
> > @@ -0,0 +1,167 @@
> > +/* SPDX-License-Identifier: GPL-2.0-only */
>
> As the bot points out, this is an "odd" license for a uapi .h file, are
> you SURE about this?
>
> If so, I need an Intel lawyer's signed-off-by on it as well, so we know
> to talk to in the future about it.
>
> thanks,
>
> greg k-h
Sorry, it should be "GPL-2.0-only WITH Linux-syscall-note".
Should I correct it and submit a new patch set (v10) now, or wait for more feedback on the current patch set?
Thanks
Mike
On Wed, Jan 27, 2021 at 01:59:50PM +0000, Chen, Mike Ximing wrote:
>
> > -----Original Message-----
> > From: Greg KH <[email protected]>
> > Sent: Wednesday, January 27, 2021 7:29 AM
> > To: Chen, Mike Ximing <[email protected]>
> > Cc: [email protected]; [email protected]; Williams, Dan J
> > <[email protected]>; [email protected]; Gage Eads
> > <[email protected]>
> > Subject: Re: [PATCH v9 04/20] dlb: add device ioctl layer and first three ioctls
> >
> > On Fri, Jan 22, 2021 at 01:01:22PM -0600, Mike Ximing Chen wrote:
> > > --- /dev/null
> > > +++ b/include/uapi/linux/dlb.h
> > > @@ -0,0 +1,167 @@
> > > +/* SPDX-License-Identifier: GPL-2.0-only */
> >
> > As the bot points out, this is an "odd" license for a uapi .h file, are
> > you SURE about this?
> >
> > If so, I need an Intel lawyer's signed-off-by on it as well, so we know
> > to talk to in the future about it.
> >
> > thanks,
> >
> > greg k-h
>
> Sorry, it should be "GPL-2.0-only WITH Linux-syscall-note".
> Should I correct it and submit a new patch set (v10) now, or wait for more feedback on the current patch set?
Please consult your corporate lawyers when picking licenses for your
kernel files. I doubt they want me telling you what to do :)
good luck!
greg k-h
> -----Original Message-----
> From: Greg KH <[email protected]>
> Sent: Wednesday, January 27, 2021 9:04 AM
> To: Chen, Mike Ximing <[email protected]>
> Cc: [email protected]; [email protected]; Williams, Dan J
> <[email protected]>; [email protected]
> Subject: Re: [PATCH v9 04/20] dlb: add device ioctl layer and first three ioctls
>
> On Wed, Jan 27, 2021 at 01:59:50PM +0000, Chen, Mike Ximing wrote:
> >
> > > -----Original Message-----
> > > From: Greg KH <[email protected]>
> > > Sent: Wednesday, January 27, 2021 7:29 AM
> > > To: Chen, Mike Ximing <[email protected]>
> > > Cc: [email protected]; [email protected]; Williams, Dan J
> > > <[email protected]>; [email protected]; Gage Eads
> > > <[email protected]>
> > > Subject: Re: [PATCH v9 04/20] dlb: add device ioctl layer and first three ioctls
> > >
> > > On Fri, Jan 22, 2021 at 01:01:22PM -0600, Mike Ximing Chen wrote:
> > > > --- /dev/null
> > > > +++ b/include/uapi/linux/dlb.h
> > > > @@ -0,0 +1,167 @@
> > > > +/* SPDX-License-Identifier: GPL-2.0-only */
> > >
> > > As the bot points out, this is an "odd" license for a uapi .h file, are
> > > you SURE about this?
> > >
> > > If so, I need an Intel lawyer's signed-off-by on it as well, so we know
> > > to talk to in the future about it.
> > >
> > > thanks,
> > >
> > > greg k-h
> >
> > Sorry, it should be "GPL-2.0-only WITH Linux-syscall-note".
> > Should I correct it and submit a new patch set (v10) now, or wait for more
> feedback on the current patch set?
>
> Please consult your corporate lawyers when picking licenses for your
> kernel files. I doubt they want me telling you what to do :)
>
> good luck!
>
> greg k-h
OK. I submitted a new patch set (v10) with the fix on license.
Thanks for the help.
Mike