Received: by 2002:a05:7412:419a:b0:f3:1519:9f41 with SMTP id i26csp3372856rdh; Mon, 27 Nov 2023 12:29:29 -0800 (PST) X-Google-Smtp-Source: AGHT+IFK5EabtHwLqWahfeHIrQFkc59NEMj33kg7OIIaa5vOaPdviQOAjgcSFxyaugLnsSrwgIO/ X-Received: by 2002:a05:6a20:3d85:b0:18b:af9d:438f with SMTP id s5-20020a056a203d8500b0018baf9d438fmr14135169pzi.49.1701116969143; Mon, 27 Nov 2023 12:29:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701116969; cv=none; d=google.com; s=arc-20160816; b=k4wTtBCwqE4/EKCEEdhuHtDX5Yv4LlUea/DULsuCOxiWLKyr9/QwsP8R21qHJBoBSv Hux1+u3dcqQ8AHigPwZp3ionJ7k0a3mnR78kKrxnESBKyNJDbPpNPcvgWGfzG9x18rcO yNLnGws4//ibNKYifFNPEwjX2234PA7lM+oDaObCxEulxO04SHB+yVU0lYU5RnZz0AuG bNNs/HGsX0aSqzP0yoqh4E6udjQ0f8aEYqfzAK1fjozRB79RV8YXK7mmx1w8ET6Wee8b 2oHiGL2lZay+sV4F6SAiLEtzp7VNCbZbxQPc3LdaHbiAza69f2YuuEkH/0pC1DBkAJjI HkVQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=tfCNguiNYtPe8lYVjuPGODg0biwA6QR0ZHJkObtpjhM=; fh=CKy338m5MbqOKLCytkQoYDn9hkoAEvElo6dffmo/qoM=; b=UwzkwjxGXGkuqEoi1pCDhRS1GtQpwg8HDLlwD/H0ibsL04gHKVtaiPTmV2Dc335+cQ +24TG81ijIFOknR0LWs37a5koaajCmfKLOvB0gzRFEcylweoQW/TqZ1mye8i73TxL26x 5OX3wvL4nvvapVBbLfQk+66UI/1MECFk8bQsJbCjR1hcWtIdMzNXN/3GKwmoJqRYPvJZ TCYoAaYfDg2tOWK9PppmhU7Oa4vob7DFZ82G2JfKJaGwtVKtvWQ9pSHpyNUxj7t9RAt2 TI9ckbsZfy7L3LeqdSVAhXonlmrCGbStESH+/iT/4EjCFx4pZIXgVN+Y4fsUSRFGxCmS l3LA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=BSobYgUF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id be14-20020a656e4e000000b005bd3ca6c394si11791083pgb.43.2023.11.27.12.29.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Nov 2023 12:29:29 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=BSobYgUF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id B29C1805B9C1; Mon, 27 Nov 2023 12:29:17 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233372AbjK0U1o (ORCPT + 99 others); Mon, 27 Nov 2023 15:27:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46070 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233172AbjK0U1Y (ORCPT ); Mon, 27 Nov 2023 15:27:24 -0500 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8340D10CC; Mon, 27 Nov 2023 12:27:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1701116848; x=1732652848; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=exPEJ2NHytrZASgF46XH+MN564oA5CR2jV1mTYa4zNo=; b=BSobYgUFpi7+jqtbXw7k3cZXsBxHe1eFEMi62MtgVXuOH2+kjv8LseUh HDJUF7JmJJvJl22QGfmsrNfcpxKka9+ECoNLjNVeOJYTccmX3jd06kPaR 10VUrzxHFHxSCaMhJ47swwhrnV6jMkUjKTPGo5nY2yJ9KzbpM+r1MPIez FXZd6JtRNhpldX2kXp5Dl4hYe2zIbiwDb0j/ijxt3oXiAS4bHut/ubuxW 7BAgZdDeWkhFP/QCgIGhWRkn4YEudB4WbCL6DyxheYaGUsemiOFYQnNKS OyPDpc/fBmCasCcCzphbfNmznZk+e+dIPHl99U9yvlyn42cTLk+Yswfmf Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10907"; a="457115547" X-IronPort-AV: E=Sophos;i="6.04,231,1695711600"; d="scan'208";a="457115547" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Nov 2023 12:27:27 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.04,231,1695711600"; d="scan'208";a="16394552" Received: from rpkulapa-mobl.amr.corp.intel.com (HELO tzanussi-mobl1.hsd1.il.comcast.net) ([10.213.183.92]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Nov 2023 12:27:27 -0800 From: Tom Zanussi To: herbert@gondor.apana.org.au, davem@davemloft.net, fenghua.yu@intel.com, vkoul@kernel.org Cc: dave.jiang@intel.com, tony.luck@intel.com, wajdi.k.feghali@intel.com, james.guilford@intel.com, kanchana.p.sridhar@intel.com, vinodh.gopal@intel.com, giovanni.cabiddu@intel.com, pavel@ucw.cz, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH v10 08/14] crypto: iaa - Add Intel IAA Compression Accelerator crypto driver core Date: Mon, 27 Nov 2023 14:26:58 -0600 Message-Id: <20231127202704.1263376-9-tom.zanussi@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231127202704.1263376-1-tom.zanussi@linux.intel.com> References: <20231127202704.1263376-1-tom.zanussi@linux.intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Mon, 27 Nov 2023 12:29:18 -0800 (PST) The Intel Analytics Accelerator (IAA) is a hardware accelerator that provides very high thoughput compression/decompression compatible with the DEFLATE compression standard described in RFC 1951, which is the compression/decompression algorithm exported by this module. Users can select IAA compress/decompress acceleration by specifying one of the deflate-iaa* algorithms as the compression algorithm to use by whatever facility allows asynchronous compression algorithms to be selected. For example, zswap can select the IAA fixed deflate algorithm 'deflate-iaa' via: # echo deflate-iaa > /sys/module/zswap/parameters/compressor This patch adds iaa_crypto as an idxd sub-driver and tracks iaa devices and workqueues as they are probed or removed. [ Based on work originally by George Powley, Jing Lin and Kyung Min Park ] Signed-off-by: Tom Zanussi --- MAINTAINERS | 7 + drivers/crypto/intel/Kconfig | 1 + drivers/crypto/intel/Makefile | 1 + drivers/crypto/intel/iaa/Kconfig | 10 + drivers/crypto/intel/iaa/Makefile | 10 + drivers/crypto/intel/iaa/iaa_crypto.h | 30 ++ drivers/crypto/intel/iaa/iaa_crypto_main.c | 323 +++++++++++++++++++++ 7 files changed, 382 insertions(+) create mode 100644 drivers/crypto/intel/iaa/Kconfig create mode 100644 drivers/crypto/intel/iaa/Makefile create mode 100644 drivers/crypto/intel/iaa/iaa_crypto.h create mode 100644 drivers/crypto/intel/iaa/iaa_crypto_main.c diff --git a/MAINTAINERS b/MAINTAINERS index 97f51d5ec1cf..686ff0e1ce34 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -10719,6 +10719,13 @@ S: Supported Q: https://patchwork.kernel.org/project/linux-dmaengine/list/ F: drivers/dma/ioat* +INTEL IAA CRYPTO DRIVER +M: Tom Zanussi +L: linux-crypto@vger.kernel.org +S: Supported +F: Documentation/driver-api/crypto/iaa/iaa-crypto.rst +F: drivers/crypto/intel/iaa/* + INTEL IDLE DRIVER M: Jacob Pan M: Len Brown diff --git a/drivers/crypto/intel/Kconfig b/drivers/crypto/intel/Kconfig index 3d90c87d4094..f38cd62a3f67 100644 --- a/drivers/crypto/intel/Kconfig +++ b/drivers/crypto/intel/Kconfig @@ -3,3 +3,4 @@ source "drivers/crypto/intel/keembay/Kconfig" source "drivers/crypto/intel/ixp4xx/Kconfig" source "drivers/crypto/intel/qat/Kconfig" +source "drivers/crypto/intel/iaa/Kconfig" diff --git a/drivers/crypto/intel/Makefile b/drivers/crypto/intel/Makefile index b3d0352ae188..2f56f6d34cf0 100644 --- a/drivers/crypto/intel/Makefile +++ b/drivers/crypto/intel/Makefile @@ -3,3 +3,4 @@ obj-y += keembay/ obj-y += ixp4xx/ obj-$(CONFIG_CRYPTO_DEV_QAT) += qat/ +obj-$(CONFIG_CRYPTO_DEV_IAA_CRYPTO) += iaa/ diff --git a/drivers/crypto/intel/iaa/Kconfig b/drivers/crypto/intel/iaa/Kconfig new file mode 100644 index 000000000000..fcccb6ff7e29 --- /dev/null +++ b/drivers/crypto/intel/iaa/Kconfig @@ -0,0 +1,10 @@ +config CRYPTO_DEV_IAA_CRYPTO + tristate "Support for Intel(R) IAA Compression Accelerator" + depends on CRYPTO_DEFLATE + depends on INTEL_IDXD + default n + help + This driver supports acceleration for compression and + decompression with the Intel Analytics Accelerator (IAA) + hardware using the cryptographic API. If you choose 'M' + here, the module will be called iaa_crypto. diff --git a/drivers/crypto/intel/iaa/Makefile b/drivers/crypto/intel/iaa/Makefile new file mode 100644 index 000000000000..03859431c897 --- /dev/null +++ b/drivers/crypto/intel/iaa/Makefile @@ -0,0 +1,10 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Makefile for IAA crypto device drivers +# + +ccflags-y += -I $(srctree)/drivers/dma/idxd -DDEFAULT_SYMBOL_NAMESPACE=IDXD + +obj-$(CONFIG_CRYPTO_DEV_IAA_CRYPTO) := iaa_crypto.o + +iaa_crypto-y := iaa_crypto_main.o diff --git a/drivers/crypto/intel/iaa/iaa_crypto.h b/drivers/crypto/intel/iaa/iaa_crypto.h new file mode 100644 index 000000000000..5d1fff7f4b8e --- /dev/null +++ b/drivers/crypto/intel/iaa/iaa_crypto.h @@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright(c) 2021 Intel Corporation. All rights rsvd. */ + +#ifndef __IAA_CRYPTO_H__ +#define __IAA_CRYPTO_H__ + +#include +#include +#include + +#define IDXD_SUBDRIVER_NAME "crypto" + +/* Representation of IAA workqueue */ +struct iaa_wq { + struct list_head list; + struct idxd_wq *wq; + + struct iaa_device *iaa_device; +}; + +/* Representation of IAA device with wqs, populated by probe */ +struct iaa_device { + struct list_head list; + struct idxd_device *idxd; + + int n_wq; + struct list_head wqs; +}; + +#endif diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c new file mode 100644 index 000000000000..9a662ae49106 --- /dev/null +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -0,0 +1,323 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2021 Intel Corporation. All rights rsvd. */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "idxd.h" +#include "iaa_crypto.h" + +#ifdef pr_fmt +#undef pr_fmt +#endif + +#define pr_fmt(fmt) "idxd: " IDXD_SUBDRIVER_NAME ": " fmt + +/* number of iaa instances probed */ +static unsigned int nr_iaa; + +static LIST_HEAD(iaa_devices); +static DEFINE_MUTEX(iaa_devices_lock); + +static struct iaa_device *iaa_device_alloc(void) +{ + struct iaa_device *iaa_device; + + iaa_device = kzalloc(sizeof(*iaa_device), GFP_KERNEL); + if (!iaa_device) + return NULL; + + INIT_LIST_HEAD(&iaa_device->wqs); + + return iaa_device; +} + +static void iaa_device_free(struct iaa_device *iaa_device) +{ + struct iaa_wq *iaa_wq, *next; + + list_for_each_entry_safe(iaa_wq, next, &iaa_device->wqs, list) { + list_del(&iaa_wq->list); + kfree(iaa_wq); + } + + kfree(iaa_device); +} + +static bool iaa_has_wq(struct iaa_device *iaa_device, struct idxd_wq *wq) +{ + struct iaa_wq *iaa_wq; + + list_for_each_entry(iaa_wq, &iaa_device->wqs, list) { + if (iaa_wq->wq == wq) + return true; + } + + return false; +} + +static struct iaa_device *add_iaa_device(struct idxd_device *idxd) +{ + struct iaa_device *iaa_device; + + iaa_device = iaa_device_alloc(); + if (!iaa_device) + return NULL; + + iaa_device->idxd = idxd; + + list_add_tail(&iaa_device->list, &iaa_devices); + + nr_iaa++; + + return iaa_device; +} + +static void del_iaa_device(struct iaa_device *iaa_device) +{ + list_del(&iaa_device->list); + + iaa_device_free(iaa_device); + + nr_iaa--; +} + +static int add_iaa_wq(struct iaa_device *iaa_device, struct idxd_wq *wq, + struct iaa_wq **new_wq) +{ + struct idxd_device *idxd = iaa_device->idxd; + struct pci_dev *pdev = idxd->pdev; + struct device *dev = &pdev->dev; + struct iaa_wq *iaa_wq; + + iaa_wq = kzalloc(sizeof(*iaa_wq), GFP_KERNEL); + if (!iaa_wq) + return -ENOMEM; + + iaa_wq->wq = wq; + iaa_wq->iaa_device = iaa_device; + idxd_wq_set_private(wq, iaa_wq); + + list_add_tail(&iaa_wq->list, &iaa_device->wqs); + + iaa_device->n_wq++; + + if (new_wq) + *new_wq = iaa_wq; + + dev_dbg(dev, "added wq %d to iaa device %d, n_wq %d\n", + wq->id, iaa_device->idxd->id, iaa_device->n_wq); + + return 0; +} + +static void del_iaa_wq(struct iaa_device *iaa_device, struct idxd_wq *wq) +{ + struct idxd_device *idxd = iaa_device->idxd; + struct pci_dev *pdev = idxd->pdev; + struct device *dev = &pdev->dev; + struct iaa_wq *iaa_wq; + + list_for_each_entry(iaa_wq, &iaa_device->wqs, list) { + if (iaa_wq->wq == wq) { + list_del(&iaa_wq->list); + iaa_device->n_wq--; + + dev_dbg(dev, "removed wq %d from iaa_device %d, n_wq %d, nr_iaa %d\n", + wq->id, iaa_device->idxd->id, + iaa_device->n_wq, nr_iaa); + + if (iaa_device->n_wq == 0) + del_iaa_device(iaa_device); + break; + } + } +} + +static int save_iaa_wq(struct idxd_wq *wq) +{ + struct iaa_device *iaa_device, *found = NULL; + struct idxd_device *idxd; + struct pci_dev *pdev; + struct device *dev; + int ret = 0; + + list_for_each_entry(iaa_device, &iaa_devices, list) { + if (iaa_device->idxd == wq->idxd) { + idxd = iaa_device->idxd; + pdev = idxd->pdev; + dev = &pdev->dev; + /* + * Check to see that we don't already have this wq. + * Shouldn't happen but we don't control probing. + */ + if (iaa_has_wq(iaa_device, wq)) { + dev_dbg(dev, "same wq probed multiple times for iaa_device %p\n", + iaa_device); + goto out; + } + + found = iaa_device; + + ret = add_iaa_wq(iaa_device, wq, NULL); + if (ret) + goto out; + + break; + } + } + + if (!found) { + struct iaa_device *new_device; + struct iaa_wq *new_wq; + + new_device = add_iaa_device(wq->idxd); + if (!new_device) { + ret = -ENOMEM; + goto out; + } + + ret = add_iaa_wq(new_device, wq, &new_wq); + if (ret) { + del_iaa_device(new_device); + goto out; + } + } + + if (WARN_ON(nr_iaa == 0)) + return -EINVAL; +out: + return 0; +} + +static void remove_iaa_wq(struct idxd_wq *wq) +{ + struct iaa_device *iaa_device; + + list_for_each_entry(iaa_device, &iaa_devices, list) { + if (iaa_has_wq(iaa_device, wq)) { + del_iaa_wq(iaa_device, wq); + break; + } + } +} + +static int iaa_crypto_probe(struct idxd_dev *idxd_dev) +{ + struct idxd_wq *wq = idxd_dev_to_wq(idxd_dev); + struct idxd_device *idxd = wq->idxd; + struct idxd_driver_data *data = idxd->data; + struct device *dev = &idxd_dev->conf_dev; + int ret = 0; + + if (idxd->state != IDXD_DEV_ENABLED) + return -ENXIO; + + if (data->type != IDXD_TYPE_IAX) + return -ENODEV; + + mutex_lock(&wq->wq_lock); + + if (!idxd_wq_driver_name_match(wq, dev)) { + dev_dbg(dev, "wq %d.%d driver_name match failed: wq driver_name %s, dev driver name %s\n", + idxd->id, wq->id, wq->driver_name, dev->driver->name); + idxd->cmd_status = IDXD_SCMD_WQ_NO_DRV_NAME; + ret = -ENODEV; + goto err; + } + + wq->type = IDXD_WQT_KERNEL; + + ret = idxd_drv_enable_wq(wq); + if (ret < 0) { + dev_dbg(dev, "enable wq %d.%d failed: %d\n", + idxd->id, wq->id, ret); + ret = -ENXIO; + goto err; + } + + mutex_lock(&iaa_devices_lock); + + ret = save_iaa_wq(wq); + if (ret) + goto err_save; + + mutex_unlock(&iaa_devices_lock); +out: + mutex_unlock(&wq->wq_lock); + + return ret; + +err_save: + idxd_drv_disable_wq(wq); +err: + wq->type = IDXD_WQT_NONE; + + goto out; +} + +static void iaa_crypto_remove(struct idxd_dev *idxd_dev) +{ + struct idxd_wq *wq = idxd_dev_to_wq(idxd_dev); + + idxd_wq_quiesce(wq); + + mutex_lock(&wq->wq_lock); + mutex_lock(&iaa_devices_lock); + + remove_iaa_wq(wq); + idxd_drv_disable_wq(wq); + + mutex_unlock(&iaa_devices_lock); + mutex_unlock(&wq->wq_lock); +} + +static enum idxd_dev_type dev_types[] = { + IDXD_DEV_WQ, + IDXD_DEV_NONE, +}; + +static struct idxd_device_driver iaa_crypto_driver = { + .probe = iaa_crypto_probe, + .remove = iaa_crypto_remove, + .name = IDXD_SUBDRIVER_NAME, + .type = dev_types, +}; + +static int __init iaa_crypto_init_module(void) +{ + int ret = 0; + + ret = idxd_driver_register(&iaa_crypto_driver); + if (ret) { + pr_debug("IAA wq sub-driver registration failed\n"); + goto out; + } + + pr_debug("initialized\n"); +out: + return ret; +} + +static void __exit iaa_crypto_cleanup_module(void) +{ + idxd_driver_unregister(&iaa_crypto_driver); + + pr_debug("cleaned up\n"); +} + +MODULE_IMPORT_NS(IDXD); +MODULE_LICENSE("GPL"); +MODULE_ALIAS_IDXD_DEVICE(0); +MODULE_AUTHOR("Intel Corporation"); +MODULE_DESCRIPTION("IAA Compression Accelerator Crypto Driver"); + +module_init(iaa_crypto_init_module); +module_exit(iaa_crypto_cleanup_module); -- 2.34.1