Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Received-SPF: Pass (protection.outlook.com: domain of amd.com designates
 165.204.84.17 as permitted sender) receiver=protection.outlook.com;
 client-ip=165.204.84.17; helo=SATLEXMB03.amd.com; pr=C
From:   Nipun Gupta <nipun.gupta@amd.com>
To:     <robh+dt@kernel.org>, <krzysztof.kozlowski+dt@linaro.org>,
        <gregkh@linuxfoundation.org>, <rafael@kernel.org>,
        <eric.auger@redhat.com>, <alex.williamson@redhat.com>,
        <cohuck@redhat.com>, <puneet.gupta@amd.com>,
        <song.bao.hua@hisilicon.com>, <mchehab+huawei@kernel.org>,
        <maz@kernel.org>, <f.fainelli@gmail.com>,
        <jeffrey.l.hugo@gmail.com>, <saravanak@google.com>,
        <Michael.Srba@seznam.cz>, <mani@kernel.org>, <yishaih@nvidia.com>,
        <jgg@ziepe.ca>, <jgg@nvidia.com>, <robin.murphy@arm.com>,
        <will@kernel.org>, <joro@8bytes.org>, <masahiroy@kernel.org>,
        <ndesaulniers@google.com>, <linux-arm-kernel@lists.infradead.org>,
        <linux-kbuild@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        <devicetree@vger.kernel.org>, <kvm@vger.kernel.org>
CC:     <okaya@kernel.org>, <harpreet.anand@amd.com>,
        <nikhil.agarwal@amd.com>, <michal.simek@amd.com>,
        <aleksandar.radovanovic@amd.com>, <git@amd.com>,
        Nipun Gupta <nipun.gupta@amd.com>
Subject: [RFC PATCH v3 0/7] add support for CDX bus
Date:   Tue, 6 Sep 2022 19:17:54 +0530
Message-ID: <20220906134801.4079497-1-nipun.gupta@amd.com>
In-Reply-To: <20220803122655.100254-1-nipun.gupta@amd.com>
References: <20220803122655.100254-1-nipun.gupta@amd.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Sep 2022 13:48:13.5814
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: a304928f-18c6-4a1e-7623-08da900e71a9
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB03.amd.com]
X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000B8EE.namprd05.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB4939
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_PASS,SPF_PASS,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Devices in FPGA can be added/modified dynamically on run-time.
This patch series introduces AMD CDX bus, which provides a
mechanism to discover/rescan FPGA devices on run-time. These
devices are memory mapped on system bus for embedded CPUs, and
added as CDX devices in Linux framework.

This RFC:
- Intrduces the CDX bus controller and CDX devices.
- Adds rescan and reset support for the CDX bus.
- Add support for reset for CDX devices.
- Support for CDX bus in arm-smmu-v3 driver
- Support for CDX-MSI domain.
- Vfio-cdx driver support for CDX devices.

Please NOTE: This RFC change does not support the CDX bus firmware
interface as it is under development, and this series aims to get
an early feedback from the community. Firmware interaction are
stubbed as MCDI APIs which is a protocol used by AMD to interact
with Firmware.

Changes in v2:
- introduce CDX bus infrastructure
- fixed code for making compatible visible for devices
  having the 'compatible' property only (Greg K-H)
- moved CDX-MSI domain as part of CDX bus infrastructure.
  previously it was part of irqchip (Marc Zynger).
- fixed few prints (Greg K-H)
- support rescan and reset of CDX bus
- add VFIO reset module for CDX bus based devices

Changes in v3:
- Move CDX bus as a new bus type in kernel rather than
  using the platform devices (Greg K-H, Marc Zynger)
- Correspondingly update ARM SMMU v3
- Add support for vfio-cdx driver
- Updated device tree yaml with correct binding information
  (Krzysztof Kozlowski)
- remove 'compatible' sysfs platform patch which was requried
  for CDX devices exposed as platform devices


Following text provides a basic overview of CDX bus Architecture.

Contents summary
   - CDX overview
   - CDX Linux driver architecture overview
        - Bus driver
        - VFIO driver

CDX Overview
------------

CDX is a Hardware Architecture designed for AMD FPGA devices. It
consists of sophisticated mechanism for interaction between FPGA,
Firmware and the APUs (Application CPUs).

Firmware resides on RPU (Realtime CPUs) which interacts with
the FPGA program manager and the APUs. The RPU provides memory-mapped
interface (RPU if) which is used to communicate with APUs.

The diagram below shows an overview of the CDX architecture:


         +--------------------------------------+
         |    Application CPUs (APU)            |
         |                                      |
         |                    CDX device drivers|
         |     Linux OS                |        |
         |                        CDX bus       |
         |                             |        |
         +-----------------------------|--------+
                                       | (discover, config,
                                       |  reset, rescan)
                                       |
         +------------------------| RPU if |----+
         |                             |        |
         |                             V        |
         |          Realtime CPUs (RPU)         |
         |                                      |
         +--------------------------------------+
                               |
         +---------------------|----------------+
         |  FPGA               |                |
         |      +-----------------------+       |
         |      |           |           |       |
         | +-------+    +-------+   +-------+   |
         | | dev 1 |    | dev 2 |   | dev 3 |   |
         | +-------+    +-------+   +-------+   |
         +--------------------------------------+


The RPU firmware extracts the device information from the loaded FPGA
image and implements a mechanism that allows the APU drivers to
enumerate such devices (device personality and resource details) via
a dedicated communication channel. RPU mediates operations such as
discover, reset and rescan of the FPGA devices for the APU. This is
done using memory mapped interface provided by the RPU to APU.

Nipun Gupta (7):
  dt-bindings: bus: add CDX bus device tree bindings
  bus/cdx: add the cdx bus driver
  iommu/arm-smmu-v3: support ops registration for CDX bus
  bus/cdx: add cdx-MSI domain with gic-its domain as parent
  bus/cdx: add bus and device attributes
  vfio/cdx: add support for CDX bus
  vfio/cdx: add interrupt support

 Documentation/ABI/testing/sysfs-bus-cdx       |  54 ++
 .../devicetree/bindings/bus/xlnx,cdx.yaml     |  75 +++
 MAINTAINERS                                   |   9 +
 drivers/bus/Kconfig                           |   1 +
 drivers/bus/Makefile                          |   3 +
 drivers/bus/cdx/Kconfig                       |   7 +
 drivers/bus/cdx/Makefile                      |   3 +
 drivers/bus/cdx/cdx.c                         | 603 ++++++++++++++++++
 drivers/bus/cdx/cdx.h                         |  53 ++
 drivers/bus/cdx/cdx_msi.c                     | 236 +++++++
 drivers/bus/cdx/mcdi_stubs.c                  |  61 ++
 drivers/bus/cdx/mcdi_stubs.h                  |  87 +++
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   |  16 +-
 drivers/vfio/Makefile                         |   1 +
 drivers/vfio/cdx/Kconfig                      |  10 +
 drivers/vfio/cdx/Makefile                     |   4 +
 drivers/vfio/cdx/vfio_cdx.c                   | 337 ++++++++++
 drivers/vfio/cdx/vfio_cdx_intr.c              | 212 ++++++
 drivers/vfio/cdx/vfio_cdx_private.h           |  50 ++
 include/linux/cdx/cdx_bus.h                   | 120 ++++
 include/linux/mod_devicetable.h               |  13 +
 scripts/mod/devicetable-offsets.c             |   4 +
 scripts/mod/file2alias.c                      |  12 +
 23 files changed, 1969 insertions(+), 2 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-bus-cdx
 create mode 100644 Documentation/devicetree/bindings/bus/xlnx,cdx.yaml
 create mode 100644 drivers/bus/cdx/Kconfig
 create mode 100644 drivers/bus/cdx/Makefile
 create mode 100644 drivers/bus/cdx/cdx.c
 create mode 100644 drivers/bus/cdx/cdx.h
 create mode 100644 drivers/bus/cdx/cdx_msi.c
 create mode 100644 drivers/bus/cdx/mcdi_stubs.c
 create mode 100644 drivers/bus/cdx/mcdi_stubs.h
 create mode 100644 drivers/vfio/cdx/Kconfig
 create mode 100644 drivers/vfio/cdx/Makefile
 create mode 100644 drivers/vfio/cdx/vfio_cdx.c
 create mode 100644 drivers/vfio/cdx/vfio_cdx_intr.c
 create mode 100644 drivers/vfio/cdx/vfio_cdx_private.h
 create mode 100644 include/linux/cdx/cdx_bus.h

-- 
2.25.1