Received: by 2002:a05:6a10:2785:0:0:0:0 with SMTP id ia5csp2930321pxb; Tue, 12 Jan 2021 02:02:39 -0800 (PST) X-Google-Smtp-Source: ABdhPJwbgoK+Ctz8RP0hSzlvZyJ/Gh9zovfIIyCn1/fgcnpXGPAOD7SUojbWEvDlgOu8+7vAoKxA X-Received: by 2002:a17:906:8058:: with SMTP id x24mr2551243ejw.262.1610445759621; Tue, 12 Jan 2021 02:02:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1610445759; cv=none; d=google.com; s=arc-20160816; b=TvxjpMqhSm117o9Sc4mgR0xM23h5vFiVauM4n55EcbGZSa0Zh2b9S1vVAbJqEG+SuO 6T9NqxTZAcoDFaGIgwqRlgv1LZMlPXGuojcPw0MdCINNaxD85IVCYgc05cjJw6N+Usu2 kc5EoJdERU0RhQP89Mpou0XVW1xZPVVV4KlhcYQ421DREsY1Mb7tg07+upKagWNYhjBK /t3c1jTL7xXLYaBhd3W82mw/+ART2ad7yZA294TVr6OJmibLW1gaF6ayfi0Lp2RpP7jG DXSU8S7617b5JaOQgaIonjuuvI26ERCDEhcEh8ME0tubmVgPTALk+QD4Nx7D/6nkjwQL sm8g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :ironport-sdr:ironport-sdr; bh=NrgBhHF62rgtrwuTRIc4xs8xINDQgK8j99KSipyos7A=; b=UDBsY79+TmdpemldbX/1CucbnDW3W9vtuH4o6KIA0xr6msUzzLUgLFc34OMPHmNCH2 fgP6yVMHf+SshvPdwzwk1ffiety9tBysEwtzbxsPUcurPDI1LcSaF7ToIovKzv5+TyjR KfQ73MaZIs70SIaH+oGzekNGfGGlPQXbQP86mfwthu9n2fSgLAtBSsEAl3mBlPlXNfDj Dwgfkt4vfZevYRlH62wWZKOHRCfeJONDfgopgtYD02iLcpuXdkU5yPM2fwEhbffON8Zx iiFb62XioIvIqgQYhHYKhBhFKIC6OqvnHyZZBdzvb7727K8MOF09ajhX+S+tvXKgB/WY vsBA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id di13si1058759edb.412.2021.01.12.02.02.16; Tue, 12 Jan 2021 02:02:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2391196AbhALAYm (ORCPT + 99 others); Mon, 11 Jan 2021 19:24:42 -0500 Received: from mga17.intel.com ([192.55.52.151]:11217 "EHLO mga17.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390793AbhAKWw1 (ORCPT ); Mon, 11 Jan 2021 17:52:27 -0500 IronPort-SDR: e0/zu56RER4gpscXReukuRm81pUWAyYGrDSWkKxUjfYoz05P33ziRei0eJrlhNPCKIvlHoXp1+ MAW2BMZ8DgDw== X-IronPort-AV: E=McAfee;i="6000,8403,9861"; a="157726531" X-IronPort-AV: E=Sophos;i="5.79,339,1602572400"; d="scan'208";a="157726531" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2021 14:51:32 -0800 IronPort-SDR: u/pYsBqHZwkELUf0+YDhzxQQE1UBrsmog5L/HjntD/YKgNifUJOGVbPmfJ3h4XdDiO/R7juq7M KCyjPH0M+53A== X-IronPort-AV: E=Sophos;i="5.79,339,1602572400"; d="scan'208";a="352777942" Received: from yyang31-mobl.amr.corp.intel.com (HELO bwidawsk-mobl5.local) ([10.252.142.71]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2021 14:51:31 -0800 From: Ben Widawsky To: linux-cxl@vger.kernel.org Cc: Dan Williams , Ben Widawsky , linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, "linux-acpi@vger.kernel.org, Ira Weiny" , Vishal Verma , "Kelley, Sean V" , Rafael Wysocki , Bjorn Helgaas , Jonathan Cameron , Jon Masters , Chris Browy , Randy Dunlap , Christoph Hellwig , daniel.lll@alibaba-inc.com Subject: [RFC PATCH v3 04/16] cxl/mem: Introduce a driver for CXL-2.0-Type-3 endpoints Date: Mon, 11 Jan 2021 14:51:08 -0800 Message-Id: <20210111225121.820014-5-ben.widawsky@intel.com> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210111225121.820014-1-ben.widawsky@intel.com> References: <20210111225121.820014-1-ben.widawsky@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Dan Williams The CXL.mem protocol allows a device to act as a provider of "System RAM" and/or "Persistent Memory" that is fully coherent as if the memory was attached to the typical CPU memory controller. With the CXL-2.0 specification a PCI endpoint can implement a "Type-3" device interface and give the operating system control over "Host Managed Device Memory". See section 2.3 Type 3 CXL Device. The memory range exported by the device may optionally be described by the platform firmware memory map, or by infrastructure like LIBNVDIMM to provision persistent memory capacity from one, or more, CXL.mem devices. A pre-requisite for Linux-managed memory-capacity provisioning is this cxl_mem driver that can speak the mailbox protocol defined in section 8.2.8.4 Mailbox Registers. For now just land the driver boiler-plate and fill it in with functionality in subsequent commits. Link: https://www.computeexpresslink.org/download-the-specification Signed-off-by: Dan Williams Signed-off-by: Ben Widawsky --- Documentation/cxl/memory-devices.rst | 9 ++++ drivers/cxl/Kconfig | 22 +++++++++ drivers/cxl/Makefile | 2 + drivers/cxl/mem.c | 69 ++++++++++++++++++++++++++++ drivers/cxl/pci.h | 20 ++++++++ 5 files changed, 122 insertions(+) create mode 100644 drivers/cxl/mem.c create mode 100644 drivers/cxl/pci.h diff --git a/Documentation/cxl/memory-devices.rst b/Documentation/cxl/memory-devices.rst index 6ce88f9d5f4f..134c9b6b4ff4 100644 --- a/Documentation/cxl/memory-devices.rst +++ b/Documentation/cxl/memory-devices.rst @@ -23,6 +23,15 @@ ACPI CXL .. kernel-doc:: drivers/cxl/acpi.c :internal: +CXL Memory Device +----------------- + +.. kernel-doc:: drivers/cxl/mem.c + :doc: cxl mem + +.. kernel-doc:: drivers/cxl/mem.c + :internal: + External Interfaces =================== diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig index 68da926ba5b1..0ac5080cd6e0 100644 --- a/drivers/cxl/Kconfig +++ b/drivers/cxl/Kconfig @@ -33,4 +33,26 @@ config CXL_ACPI specification. If unsure say 'm' + +config CXL_MEM + tristate "CXL.mem: Endpoint Support" + depends on PCI && CXL_BUS_PROVIDER + default CXL_BUS_PROVIDER + help + The CXL.mem protocol allows a device to act as a provider of + "System RAM" and/or "Persistent Memory" that is fully coherent + as if the memory was attached to the typical CPU memory + controller. + + Say 'y/m' to enable a driver (named "cxl_mem.ko" when built as + a module) that will attach to CXL.mem devices for + configuration, provisioning, and health monitoring. This + driver is required for dynamic provisioning of CXL.mem + attached memory which is a pre-requisite for persistent memory + support. Typically volatile memory is mapped by platform + firmware and included in the platform memory map, but in some + cases the OS is responsible for mapping that memory. See + Chapter 2.3 Type 3 CXL Device in the CXL 2.0 specification. + + If unsure say 'm'. endif diff --git a/drivers/cxl/Makefile b/drivers/cxl/Makefile index d38cd34a2582..97fdffb00f2d 100644 --- a/drivers/cxl/Makefile +++ b/drivers/cxl/Makefile @@ -1,5 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o +obj-$(CONFIG_CXL_MEM) += cxl_mem.o ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL cxl_acpi-y := acpi.o +cxl_mem-y := mem.o diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c new file mode 100644 index 000000000000..005404888942 --- /dev/null +++ b/drivers/cxl/mem.c @@ -0,0 +1,69 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2020 Intel Corporation. All rights reserved. */ +#include +#include +#include +#include "acpi.h" +#include "pci.h" + +static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec) +{ + int pos; + + pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_DVSEC); + if (!pos) + return 0; + + while (pos) { + u16 vendor, id; + + pci_read_config_word(pdev, pos + PCI_DVSEC_VENDOR_ID_OFFSET, + &vendor); + pci_read_config_word(pdev, pos + PCI_DVSEC_ID_OFFSET, &id); + if (vendor == PCI_DVSEC_VENDOR_ID_CXL && dvsec == id) + return pos; + + pos = pci_find_next_ext_capability(pdev, pos, + PCI_EXT_CAP_ID_DVSEC); + } + + return 0; +} + +static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) +{ + struct device *dev = &pdev->dev; + int rc, regloc; + + rc = cxl_bus_acquire(pdev); + if (rc != 0) { + dev_err(dev, "failed to acquire interface\n"); + return rc; + } + + regloc = cxl_mem_dvsec(pdev, PCI_DVSEC_ID_CXL_REGLOC); + if (!regloc) { + dev_err(dev, "register location dvsec not found\n"); + return -ENXIO; + } + + return 0; +} + +static const struct pci_device_id cxl_mem_pci_tbl[] = { + /* PCI class code for CXL.mem Type-3 Devices */ + { PCI_ANY_ID, PCI_ANY_ID, PCI_ANY_ID, PCI_ANY_ID, + PCI_CLASS_MEMORY_CXL, 0xffffff, 0 }, + { /* terminate list */ }, +}; +MODULE_DEVICE_TABLE(pci, cxl_mem_pci_tbl); + +static struct pci_driver cxl_mem_driver = { + .name = KBUILD_MODNAME, + .id_table = cxl_mem_pci_tbl, + .probe = cxl_mem_probe, +}; + +MODULE_LICENSE("GPL v2"); +module_pci_driver(cxl_mem_driver); +MODULE_IMPORT_NS(CXL); diff --git a/drivers/cxl/pci.h b/drivers/cxl/pci.h new file mode 100644 index 000000000000..a8a9935fa90b --- /dev/null +++ b/drivers/cxl/pci.h @@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2020 Intel Corporation. All rights reserved. */ +#ifndef __CXL_PCI_H__ +#define __CXL_PCI_H__ + +#define PCI_CLASS_MEMORY_CXL 0x050210 + +/* + * See section 8.1 Configuration Space Registers in the CXL 2.0 + * Specification + */ +#define PCI_EXT_CAP_ID_DVSEC 0x23 +#define PCI_DVSEC_VENDOR_ID_CXL 0x1E98 +#define PCI_DVSEC_VENDOR_ID_OFFSET 0x4 +#define PCI_DVSEC_ID_CXL 0x0 +#define PCI_DVSEC_ID_OFFSET 0x8 + +#define PCI_DVSEC_ID_CXL_REGLOC 0x8 + +#endif /* __CXL_PCI_H__ */ -- 2.30.0