Received: by 2002:ac0:a5a6:0:0:0:0:0 with SMTP id m35-v6csp1297478imm; Sun, 2 Sep 2018 17:54:53 -0700 (PDT) X-Google-Smtp-Source: ANB0VdY5V5AfJ83ApuxTFIMETnS/xmSxUjGvXrU3af0qJ04EU8tITl8+s3oeDlRnemzezKH03DA1 X-Received: by 2002:a63:4207:: with SMTP id p7-v6mr9351318pga.201.1535936093030; Sun, 02 Sep 2018 17:54:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535936092; cv=none; d=google.com; s=arc-20160816; b=LsTM5bpldqh+W5eV8HP/mzTMXVnVPydZQTies/aFu46f2Un5nA3C74N42VES9KEn1G livfLGTkM9fZH3F8wqppS6zY+8JT9+1mfdooI66ZJAbaykatDNfhRWdAGycxGteGmY4/ PhZFa+AtYgBafiL1CuJr4yoHPitowxyMSTa9DcwIU2oSuxE9oW5Vfo9oRq1A/ZD2w5l2 GsgMIwX3vy1Arv/Yc/P9ZVfP6JTnV9Sjw6yPVMByroVose2N1TbTJIecxMjW9+mwAAfd YhRbpA2okWDbaqaHwGbffcYnjSxMpgyhONd5V12b/ZlqoqnM1Zgzsi+I6rvwacZz7Jyh eD8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=sa17iEnf/25lNE7MuUyFrCggJ8HGDk9yuNb+pUO+7vk=; b=XXBQw91BUolnTQ4NeYBNzgGXblx831tnKG9FHjX1F+WJSXBFqHJtjusX2w0zDAcHHf 0xnrtP/CsQvnYnbFJ/gDtl38fq4v1FUbFQ8XZnM9lrRMFls0YTn5WhflHVhW/ELy3pP1 cNzSQ2Ad1Iyl0GWBMbS3pw4GcqQgXrXvuRvNdJbrMkRM1F/YTbOuZirgVLFvDn2Q4RoB a0AAsgFjhrgsmeEJ2P9bZ6T013N5hSTNLT2Slq4WgSeGcMhTZFJK4VhqS/mgkhxFmEn9 lW+fXaKihMREcnHJqqmkGrqT1qaHmiXUBkXR0X8LQ3Ol+ZDkIGTfdLTNsxH52AFWWPvH 5OsQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=G2ojUcCl; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id p15-v6si15541323pgl.340.2018.09.02.17.54.25; Sun, 02 Sep 2018 17:54:52 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=G2ojUcCl; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726004AbeICFKC (ORCPT + 99 others); Mon, 3 Sep 2018 01:10:02 -0400 Received: from mail-pl1-f195.google.com ([209.85.214.195]:41330 "EHLO mail-pl1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725810AbeICFKC (ORCPT ); Mon, 3 Sep 2018 01:10:02 -0400 Received: by mail-pl1-f195.google.com with SMTP id b12-v6so7823294plr.8; Sun, 02 Sep 2018 17:52:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=sa17iEnf/25lNE7MuUyFrCggJ8HGDk9yuNb+pUO+7vk=; b=G2ojUcCl9aw/F1TuaQOD1xrg0vx7RUkVuTkc8vfoUxHVPSaR8bfuEjeavLWcfIQwI/ O171Qwo32R6YBkogZybmwVQt6N1SQ7eHjocaTgRUnFzNzBFs/SCdrFQLOHEmBJYGtY+X wvfTZqAa6qi02txnUbnsx31qkZb+NzgFtKT9PtBJcPezs4+6Gx97VgFQrsipGPE6hpUZ kSbXG2LAto53pnx20LbnBGAiZL2vn4tcGu0WT+BGeF0IY5qSUDHAfVP9Y428VCrp5axc 0DtZubl6KVnkPZt2U5gakqhJp7pxEJKUuzGFgusqp1p3G+46coH09MiHjZ6aI9arh+R0 Vw/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=sa17iEnf/25lNE7MuUyFrCggJ8HGDk9yuNb+pUO+7vk=; b=cnzFu0zdN1zTgmaeSUdz0D3LjfBzN+gUyCiA1fWQwDviQHeA6o44IOXEigV0vrlr0v l3sK50pWKKK6UO5KXMY8r+1o9GJ+3D/RDQN+Rbj5cxHdtvGGPMxo1eU1JYiro4E++VC1 YR0Qwa8pXBWcqyXtxooTiMz3E1Aqj53IjcwXee+sRYHl7K3VFMS06TdRrIJskhvOs580 G4gmN+GYfaDqRKzxlqFM+Z0upf+Piss4B9LZQGyPi5VNKsxymNIihdGEtaVBH9UOpI5g p/cjhESioGJC5gHHzfvCW28L+OkXBhVQiY+whgBZM/FswykNYmoBfdU5pfK4b6kTKMM6 dIMA== X-Gm-Message-State: APzg51CvAtZHqIJVnEN9RddGQU9d+48ckn2gH2aWQvGfbDa6wLLsRMmo RvHg+PwCfIVJyJFQ4cZ24Y0= X-Received: by 2002:a17:902:261:: with SMTP id 88-v6mr25736392plc.331.1535935940956; Sun, 02 Sep 2018 17:52:20 -0700 (PDT) Received: from localhost.localdomain ([203.160.89.93]) by smtp.gmail.com with ESMTPSA id w81-v6sm31487064pfk.92.2018.09.02.17.52.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Sep 2018 17:52:20 -0700 (PDT) From: Kenneth Lee To: Jonathan Corbet , Herbert Xu , "David S . Miller" , Joerg Roedel , Alex Williamson , Kenneth Lee , Hao Fang , Zhou Wang , Zaibo Xu , Philippe Ombredanne , Greg Kroah-Hartman , Thomas Gleixner , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, iommu@lists.linux-foundation.org, kvm@vger.kernel.org, linux-accelerators@lists.ozlabs.org, Lu Baolu , Sanjay Kumar Cc: linuxarm@huawei.com Subject: [PATCH 1/7] vfio/sdmdev: Add documents for WarpDrive framework Date: Mon, 3 Sep 2018 08:51:58 +0800 Message-Id: <20180903005204.26041-2-nek.in.cn@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180903005204.26041-1-nek.in.cn@gmail.com> References: <20180903005204.26041-1-nek.in.cn@gmail.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Kenneth Lee WarpDrive is a common user space accelerator framework. Its main component in Kernel is called sdmdev, Share Domain Mediated Device. It exposes the hardware capabilities to the user space via vfio-mdev. So processes in user land can obtain a "queue" by open the device and direct access the hardware MMIO space or do DMA operation via VFIO interface. WarpDrive is intended to be used with Jean Philippe Brucker's SVA patchset to support multi-process. But This is not a must. Without the SVA patches, WarpDrive can still work for one process for every hardware device. This patch add detail documents for the framework. Signed-off-by: Kenneth Lee --- Documentation/00-INDEX | 2 + Documentation/warpdrive/warpdrive.rst | 100 ++++ Documentation/warpdrive/wd-arch.svg | 728 ++++++++++++++++++++++++++ 3 files changed, 830 insertions(+) create mode 100644 Documentation/warpdrive/warpdrive.rst create mode 100644 Documentation/warpdrive/wd-arch.svg diff --git a/Documentation/00-INDEX b/Documentation/00-INDEX index 2754fe83f0d4..9959affab599 100644 --- a/Documentation/00-INDEX +++ b/Documentation/00-INDEX @@ -410,6 +410,8 @@ vm/ - directory with info on the Linux vm code. w1/ - directory with documents regarding the 1-wire (w1) subsystem. +warpdrive/ + - directory with documents about WarpDrive accelerator framework. watchdog/ - how to auto-reboot Linux if it has "fallen and can't get up". ;-) wimax/ diff --git a/Documentation/warpdrive/warpdrive.rst b/Documentation/warpdrive/warpdrive.rst new file mode 100644 index 000000000000..6d2a5d1e08c4 --- /dev/null +++ b/Documentation/warpdrive/warpdrive.rst @@ -0,0 +1,100 @@ +Introduction of WarpDrive +========================= + +*WarpDrive* is a general accelerator framework for user space. It intends to +provide interface for the user process to send request to hardware +accelerator without heavy user-kernel interaction cost. + +The *WarpDrive* user library is supposed to provide a pipe-based API, such as: + :: + int wd_request_queue(struct wd_queue *q); + void wd_release_queue(struct wd_queue *q); + + int wd_send(struct wd_queue *q, void *req); + int wd_recv(struct wd_queue *q, void **req); + int wd_recv_sync(struct wd_queue *q, void **req); + int wd_flush(struct wd_queue *q); + +*wd_request_queue* creates the pipe connection, *queue*, between the +application and the hardware. The application sends request and pulls the +answer back by asynchronized wd_send/wd_recv, which directly interact with the +hardware (by MMIO or share memory) without syscall. + +*WarpDrive* maintains a unified application address space among all involved +accelerators. With the following APIs: :: + + int wd_mem_share(struct wd_queue *q, const void *addr, + size_t size, int flags); + void wd_mem_unshare(struct wd_queue *q, const void *addr, size_t size); + +The referred process space shared by these APIs can be directly referred by the +hardware. The process can also dedicate its whole process space with flags, +*WD_SHARE_ALL* (not in this patch yet). + +The name *WarpDrive* is simply a cool and general name meaning the framework +makes the application faster. As it will be explained in this text later, the +facility in kernel is called *SDMDEV*, namely "Share Domain Mediated Device". + + +How does it work +================ + +*WarpDrive* is built upon *VFIO-MDEV*. The queue is wrapped as *mdev* in VFIO. +So memory sharing can be done via standard VFIO standard DMA interface. + +The architecture is illustrated as follow figure: + +.. image:: wd-arch.svg + :alt: WarpDrive Architecture + +Accelerator driver shares its capability via *SDMDEV* API: :: + + vfio_sdmdev_register(struct vfio_sdmdev *sdmdev); + vfio_sdmdev_unregister(struct vfio_sdmdev *sdmdev); + vfio_sdmdev_wake_up(struct spimdev_queue *q); + +*vfio_sdmdev_register* is a helper function to register the hardware to the +*VFIO_MDEV* framework. The queue creation is done by *mdev* creation interface. + +*WarpDrive* User library mmap the mdev to access its mmio space and shared +memory. Request can be sent to, or receive from, hardware in this mmap-ed +space until the queue is full or empty. + +The user library can wait on the queue by ioctl(VFIO_SDMDEV_CMD_WAIT) the mdev +if the queue is full or empty. If the queue status is changed, the hardware +driver use *vfio_sdmdev_wake_up* to wake up the waiting process. + + +Multiple processes support +========================== + +In the latest mainline kernel (4.18) when this document is written, +multi-process is not supported in VFIO yet. + +Jean Philippe Brucker has a patchset to enable it[1]_. We have tested it +with our hardware (which is known as *D06*). It works well. *WarpDrive* rely +on them to support multiple processes. If it is not enabled, *WarpDrive* can +still work, but it support only one mdev for a process, which will share the +same io map table with kernel. (But it is not going to be a security problem, +since the user application cannot access the kernel address space) + +When multiprocess is support, mdev can be created based on how many +hardware resource (queue) is available. Because the VFIO framework accepts only +one open from one mdev iommu_group. Mdev become the smallest unit for process +to use queue. And the mdev will not be released if the user process exist. So +it will need a resource agent to manage the mdev allocation for the user +process. This is not in this document's range. + + +Legacy Mode Support +=================== +For the hardware on which IOMMU is not support, WarpDrive can run on *NOIOMMU* +mode. That require some update to the mdev driver, which is not included in +this version yet. + + +References +========== +.. [1] https://patchwork.kernel.org/patch/10394851/ + +.. vim: tw=78 diff --git a/Documentation/warpdrive/wd-arch.svg b/Documentation/warpdrive/wd-arch.svg new file mode 100644 index 000000000000..2b0c467ee399 --- /dev/null +++ b/Documentation/warpdrive/wd-arch.svg @@ -0,0 +1,728 @@ + + + + + + + + + + + + + + + + + + + + + + + + generation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + WarpDrive + + + user_driver + + + + sdmdev + + + + + Device Driver + * + 1 + + <<vfio>>resource management + 1 + * + + + other standard framework(crypto/nic/others) + + <<lkm>> + register as mdev with"share domain" attribute + register to other subsystem + <<user_lib>> + <<vfio>>Hardware Accessing + wd user api + + + + Device(Hardware) + + Share Domain mdev + + -- 2.17.1