Received: by 2002:ac0:a5a6:0:0:0:0:0 with SMTP id m35-v6csp276261imm; Wed, 29 Aug 2018 21:13:08 -0700 (PDT) X-Google-Smtp-Source: ANB0Vdb9+GmtuCrve0FHxoUHhRl3p2KRx+d8Ofw03kJtYt8wclUnp/1YNBreBv7mKcLKVGg91wQN X-Received: by 2002:a63:7217:: with SMTP id n23-v6mr8064413pgc.193.1535602387927; Wed, 29 Aug 2018 21:13:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535602387; cv=none; d=google.com; s=arc-20160816; b=kmZpXBOjrtMsUYEppE0DTQ2OY7DmMyo4C2kkBpadwEDBD96y2FVb/ozaTzU9QFwFdY gLiNF0Mc3q0TJkGBqaQ5UGwk6rM2t6O0FH05bCE96HVxLxvSyiC61vFNNCwvoT7WP+6r L36/TMhAB5I7sgerTaLJMTtozrvsilmcg2U3L/Gde774t7Sgj/nq/t/ufESBoda+0RNK Qr9TNbf7yoaxmIu1d6eUdxB9K9A3ULvEHOfQ1E3J0BaagGFgxm/u/vtL3hlfUJZaz46g QjcgjnLDqemciMhHGhHjwEroRZ+WlqoAWD0avO6IQ6VN2bwQgvadrME8X0qfOTvrB9d2 JaIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:date:subject:cc:to:from :arc-authentication-results; bh=TUaqxExyTaLya2oaIcMCL5oZmH2nbn4Co4MbU3WVa5M=; b=Nrz52vXwIb7OMTWbJwWwdWjV+q0DHbViPXA9Iy2lrjuodKYZGoYS0A61ClgbKxhPAZ yCkPjUEBYSRxeM4nlG3j9OhGwqDAMLE1e8WrkLv5WPyzM+oaLMwmrq2uJ70Fpqu8LZph 6maKZOJNb2W7IOi5uSpPyPgppZBeajPP78h3D4rI4E2bzPNRI7mIne7QccIjxPw1N8L5 sOpLxhpqWLf60Se/fyujuwsb1q2rPvr0NRuRE53u63/QprVC3XRW58Tge9RbIJtAx+PO gBwqeBbXjgzgr2oQCq4FXVW5fNC/y2Z0wEbX/X7pIZBwlsr5c8+tVU2FbpkTl3DJbtQi 4gUg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id d36-v6si5502029pla.446.2018.08.29.21.12.53; Wed, 29 Aug 2018 21:13:07 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727688AbeH3ILI (ORCPT + 99 others); Thu, 30 Aug 2018 04:11:08 -0400 Received: from mga12.intel.com ([192.55.52.136]:20569 "EHLO mga12.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727193AbeH3ILI (ORCPT ); Thu, 30 Aug 2018 04:11:08 -0400 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 29 Aug 2018 21:10:58 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,306,1531810800"; d="scan'208";a="68959947" Received: from allen-box.sh.intel.com ([10.239.161.122]) by orsmga007.jf.intel.com with ESMTP; 29 Aug 2018 21:10:42 -0700 From: Lu Baolu To: Joerg Roedel , David Woodhouse , Alex Williamson , Kirti Wankhede Cc: ashok.raj@intel.com, sanjay.k.kumar@intel.com, jacob.jun.pan@intel.com, kevin.tian@intel.com, Jean-Philippe Brucker , yi.l.liu@intel.com, yi.y.sun@intel.com, peterx@redhat.com, tiwei.bie@intel.com, iommu@lists.linux-foundation.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Lu Baolu Subject: [RFC PATCH v2 00/10] vfio/mdev: IOMMU aware mediated device Date: Thu, 30 Aug 2018 12:09:12 +0800 Message-Id: <20180830040922.30426-1-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.17.1 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi, The Mediate Device is a framework for fine-grained physical device sharing across the isolated domains. Currently the mdev framework is designed to be independent of the platform IOMMU support. As the result, the DMA isolation relies on the mdev parent device in a vendor specific way. There are several cases where a mediated device could be protected and isolated by the platform IOMMU. For example, Intel vt-d rev3.0 [1] introduces a new translation mode called 'scalable mode', which enables PASID-granular translations. The vt-d scalable mode is the key ingredient for Scalable I/O Virtualization [2] [3] which allows sharing a device in minimal possible granularity (ADI - Assignable Device Interface). A mediated device backed by an ADI could be protected and isolated by the IOMMU since 1) the parent device supports tagging an unique PASID to all DMA traffic out of the mediated device; and 2) the DMA translation unit (IOMMU) supports the PASID granular translation. We can apply IOMMU protection and isolation to this kind of devices just as what we are doing with an assignable PCI device. In order to distinguish the IOMMU-capable mediated devices from those which still need to rely on parent devices, this patch set adds a domain type attribute to each mdev. enum mdev_domain_type { DOMAIN_TYPE_NO_IOMMU, /* Don't need any IOMMU support. * All isolation and protection * are handled by the parent * device driver with a device * specific mechanism. */ DOMAIN_TYPE_ATTACH_PARENT, /* IOMMU can isolate and protect * the mdev, and the isolation * domain should be attaced with * the parent device. */ }; The mdev parent device driver could opt-in whether an mdev is IOMMU capable when the device is created by invoking below interface within its @create callback: int mdev_set_domain_type(struct device *dev, enum mdev_domain_type type); In the vfio_iommu_type1_attach_group(), a domain allocated through iommu_domain_alloc() will be attached to the mdev parent device if the domain types of mdev devices in group are of type ATTACH_PARENT; Otherwise, the dummy external domain will be used and all the DMA isolation and protection are routed to parent driver as the result. On IOMMU side, a basic requirement is allowing to attach multiple domains for a PCI device if the device advertises the capability and the IOMMU hardware supports finer granularity translations than the normal PCI Source ID based translation. In order for the ease of discussion, we call "a domain in auxiliary mode' or simply 'an auxiliary domain' when a domain is attached to a device for finer granularity translations (than the Source ID based one). But we need to keep in mind that this doesn't mean two types of domains. A same domain could be bound to a device for Source ID based translation, and bound to another device for finer granularity translation at the same time. Below APIs are introduced in the IOMMU glue for device drivers to use the finer granularity translation. * iommu_capable(IOMMU_CAP_AUX_DOMAIN) - Represents the ability for supporting multiple domains per device (a.k.a. finer granularity translations) of the IOMMU hardware. * iommu_en(dis)able_aux_domain(struct device *dev) - Enable/disable the multiple domains capability for a device referenced by @dev. * iommu_auxiliary_id(struct iommu_domain *domain) - Return the index value used for finer-granularity DMA translation. The specific device driver needs to feed the hardware with this value, so that hardware device could issue the DMA transaction with this value tagged. This patch series extends both IOMMU and vfio components to support mdev device passing through when it could be isolated and protected by the IOMMU units. The first part of this series (PATCH 1/10 ~ 6/10) adds the interfaces and implementation of the multiple domains per device. The second part (PATCH 7/12 ~ 10/12) adds the domain type attribute to each mdev, determines domain type according to the attribute when attaching group in vfio type1 iommu module, and bind an auxiliary domain for the group with all mediated devices which requires its own domain. This patch series depends on a patch set posted here [4] for discussion which added the support for scalable mode in Intel IOMMU driver. References: [1] https://software.intel.com/en-us/download/intel-virtualization-technology-for-directed-io-architecture-specification [2] https://software.intel.com/en-us/download/intel-scalable-io-virtualization-technical-specification [3] https://schd.ws/hosted_files/lc32018/00/LC3-SIOV-final.pdf [4] https://lkml.org/lkml/2018/8/30/27 Best regards, Lu Baolu Change log: v1->v2: - Rewrite the patches with the concept of auxiliary domains. Lu Baolu (10): iommu: Add APIs for multiple domains per device iommu/vt-d: Add multiple domains per device query iommu/amd: Add default branch in amd_iommu_capable() iommu/vt-d: Enable/disable multiple domains per device iommu/vt-d: Attach/detach domains in auxiliary mode iommu/vt-d: Return ID associated with an auxiliary domain vfio/mdev: Add mediated device domain type vfio/type1: Add domain at(de)taching group helpers vfio/type1: Determine domain type of an mdev group vfio/type1: Attach domain for mdev group drivers/iommu/amd_iommu.c | 2 + drivers/iommu/intel-iommu.c | 208 ++++++++++++++++++++++++++++++- drivers/iommu/iommu.c | 29 +++++ drivers/vfio/mdev/mdev_core.c | 36 ++++++ drivers/vfio/mdev/mdev_private.h | 2 + drivers/vfio/vfio_iommu_type1.c | 144 +++++++++++++++++++-- include/linux/intel-iommu.h | 11 ++ include/linux/iommu.h | 13 ++ include/linux/mdev.h | 26 ++++ 9 files changed, 455 insertions(+), 16 deletions(-) -- 2.17.1