Received: by 2002:ac0:a5a6:0:0:0:0:0 with SMTP id m35-v6csp883283imm; Fri, 14 Sep 2018 07:46:39 -0700 (PDT) X-Google-Smtp-Source: ANB0VdYUeNg04AkB+JXisumpt3K9hxdfY4jsvHqdHZXj3KLoj3dD73mlRRSpo3se2KS8if6cGw0e X-Received: by 2002:a62:2119:: with SMTP id h25-v6mr13184571pfh.112.1536936399772; Fri, 14 Sep 2018 07:46:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536936399; cv=none; d=google.com; s=arc-20160816; b=dgCLBUhF9H+kG1tcwGygLPJICp/XLsYWML0u07Ah20FZPj7GW0fW24CKBIg1+fhJ4l vXNWViOR+DsR+YOEYZtFLFOGEgmAlM92/NPs39+Rw3QWSQF6htI8eqdj3P1XNM4vNlvh e/oT4W/RxYpGBSatigjk/ceK5qUn5vIS14PUOKw+fRIjW2fiTf5ffJCV40OnnkC2mDvK TZ4C1GOIOViEcSQYofYZejVLjfQnmAsL1ZwTKJSJi0t8p5lRd0KjW64e0IgNFqYkAo3Z R6DkGk5NoUGWL+2ibsC6tsf0Np7H6vHwYOmqcxhbkIVqxPvaR+APEaogNMrVKHWo6ucR q9bw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding :content-language:in-reply-to:mime-version:user-agent:date :message-id:from:references:cc:to:subject; bh=Y3EpCTmh25XeT2xw8m4lSZIm9+UvsLMptE/Wc8Rze3I=; b=nJp+ParYwR2VPTivuJr7/vZ9LSorC5uHNEz/pLfUoNzaBynwJEqkidw5pTLyi1zS+W Vn7FwSNH8XPouVh5dbqjioAk0SGXlaaDKX0rVDIokbOja+Wq38KYhEZzbT0Tuf9PF384 i/wBkQ32ktmqnW4GhKeTSJ4RVLwvebaDsnEy1MuS/eqAB+xSHOUwYD/FlZ17Ljx7SbHk xOUaW81iV44z5MBlcoYOvL30jfPyT/7C553Sg4V1yEEc1hE87hLcr2JJGaHVfonCWmAH NhjmwPRIZCPm0Zuu24V/E/gKH5VM2ME1x1XKLa0M7zJO4SKd+Rn0MjA55Y1dh2HtC8YE hpXg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l67-v6si7114292pfl.167.2018.09.14.07.46.15; Fri, 14 Sep 2018 07:46:39 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728241AbeINUA7 (ORCPT + 99 others); Fri, 14 Sep 2018 16:00:59 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:34732 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727749AbeINUA7 (ORCPT ); Fri, 14 Sep 2018 16:00:59 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CEF9080D; Fri, 14 Sep 2018 07:46:08 -0700 (PDT) Received: from [10.4.12.111] (ostrya.emea.arm.com [10.4.12.111]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 6DA7A3F575; Fri, 14 Sep 2018 07:46:06 -0700 (PDT) Subject: Re: [RFC PATCH v2 08/10] vfio/type1: Add domain at(de)taching group helpers To: "Tian, Kevin" , Lu Baolu , Joerg Roedel , David Woodhouse , Alex Williamson , Kirti Wankhede Cc: "Raj, Ashok" , "Bie, Tiwei" , "Kumar, Sanjay K" , "iommu@lists.linux-foundation.org" , "linux-kernel@vger.kernel.org" , "Sun, Yi Y" , "Pan, Jacob jun" , "kvm@vger.kernel.org" References: <20180830040922.30426-1-baolu.lu@linux.intel.com> <20180830040922.30426-9-baolu.lu@linux.intel.com> <04f5dc9d-71b1-37ec-402b-fae5f9e08664@arm.com> From: Jean-Philippe Brucker Message-ID: Date: Fri, 14 Sep 2018 15:45:52 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 13/09/2018 01:35, Tian, Kevin wrote: >>> Let's consider it from the API point of view. >>> >>> We have iommu_a(de)ttach_device() APIs to attach or detach a domain >>> to/from a device. We should avoid applying a limitation of "these are >>> only for single domain case, for multiple domains, use another API". >> >> That's an idealized view of the API, the actual semantics aren't as >> simple. For IOMMU drivers that implement IOMMU_DOMAIN_DMA in their >> domain_alloc operation (Arm SMMU, AMD IOMMU, ...), attach_dev >> attaches >> an unmanaged domain, detach_dev reattaches the default DMA domain, >> and >> the detach_dev IOMMU op is not called. We can't change that now, so it's >> difficult to add more functionality to attach_dev and detach_dev. >> > > Now we have four possible usages on a(de)ttach_device: > > 1) Normal DMA API path i.e. IOMMU_DOMAIN_DMA, for DMA operations > in host/parent device driver; > > 2) VFIO passthru path, when the physical device is assigned to user space > or guest driver > > 3) mdev passthru path 1, when mdev is assigned to user space or guest > driver. Here mdev is a wrap on random PCI device > > 4) mdev passthru path 2, when mdev is assigned to user space or guest > driver. Here mdev is in a smaller granularity (e.g. tagged by PASID) as > supported by vt-d scalable mode > > 1) and 2) are existing usages. What you described (unmanaged vs. default) > falls into this category. > > 3) is actually same as 2) in IOMMU layer. sort of passing through DMA > capability to guest. Though there is still a parent driver, the parent driver > itself should not do DMA - i.e. unmanaged in host side. > > 4) is a new code path introduced in IOMMU layer, which is what > vfio_a(de)tach_aux_domain is trying to address. In this case parent device > can still have its own DMA capability, using existing code path 1) (thus > default domain still applies). and the parent device can have multiple > aux domains (and associated structures), using code path 4). We still have the problem that detach() doesn't propagate to the IOMMU driver. Here's the current flow, without mdev: 1) At boot, the PF's (parent device) driver is probed, and the IOMMU core sets up a default DMA domain, to be used by dma_alloc by the PF's driver. -> iommu.c calls default_domain->ops->attach_dev(default_domain, dev) 2) or 3) Later userspace wants to control the PF, replaces the PF's driver with vfio-pci. When userspace creates a container, VFIO allocates a new IOMMU domain, and calls iommu_attach_group. -> iommu.c calls domain->ops->attach_dev(domain, dev) This detaches the PF from the default domain, and attaches it to the new domain. 1) When the container is closed, VFIO calls iommu_detach_group. This detaches the PF from its current domain, and attaches it back to the default domain. -> iommu.c calls default_domain->ops->attach_dev(default_domain, dev) ----- Now with mdev, we still attach the DMA domain in 1). Then: 4) Userspace opens an mdev and creates a container. VFIO enables aux domain for the device. VFIO allocates a new IOMMU domain, and calls iommu_attach_device(domain1, parent_dev). -> iommu.c calls domain->ops->attach_dev(domain1, dev) Because the device is in "aux domain" state, the IOMMU driver does not detach from the default domain, but instead allocates a PASID and attaches the aux domain. (Side note: for SMMU we couldn't detach from the default domain, because we need it for MSI mappings.) 4) Userspace opens another mdev. -> iommu.c calls domain->ops->attach_dev(domain2, dev) 1)? When the container is closed, VFIO calls iommu_detach_device(domain2, parent_dev) -> iommu.c calls default_domain->ops->attach_dev(default_domain, dev) Given that auxiliary domains are attached, the IOMMU driver could deduce that this actually means "detach an auxiliary domain". But which one? So the proposed interface doesn't seem to work as is. If we want to use iommu_attach/detach_device for auxiliary domains, the existing behavior of iommu.c, and IOMMU drivers that rely on it, have to change. Any change I can think of right now seems more daunting than introducing a different path for auxiliary domains, like iommu_attach_aux_domain for example. Thanks, Jean