Received: by 2002:ac0:bc90:0:0:0:0:0 with SMTP id a16csp4081196img;
        Tue, 26 Mar 2019 02:35:31 -0700 (PDT)
X-Google-Smtp-Source: APXvYqx5oa1BIQqkncPvaicTaCGvQ1tZx9H1svm6THl4jb1//WLc8zDpEdqtZFaZ/82vg9SmUHP4
X-Received: by 2002:a17:902:d889:: with SMTP id b9mr27681888plz.294.1553592931888;
        Tue, 26 Mar 2019 02:35:31 -0700 (PDT)
ARC-Seal: i=1; a=rsa-sha256; t=1553592931; cv=none;
        d=google.com; s=arc-20160816;
        b=H4eTVKj47/tk6bsrnhrLiBiIru8ixy4ydwfRhXxdRS/Z8uV/xq24n9gGJ7lzq4Hkng
         lm73NBNW8Iou4qPG5GSWUYrkwpdfb6QW/zwmsF/Z3/VQv9kKvmdHxWIHb5e3a3r7v/kT
         DB54SwrtwUKmcn70KcHDnuD/2FCCOgrtiKlGpvDQPdzNkIYpy4L5lx8L5G4AJVk7lDcR
         Xrjb29Z2zvvTpIEqJ8WSd9yPbLnznlvN17Ea9ddYklk5NAY/VOYTWsj70qFxHlxQdSy2
         waaFRviJya5Gktu6Eo+qi4Jksc4CzSkcwQzxYawKcjxytrRoVSAJrb1nZI6YFKY9xSxE
         2rrA==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
        h=list-id:precedence:sender:dkim-signature:content-transfer-encoding
         :content-language:in-reply-to:mime-version:date:message-id:from
         :references:cc:to:subject;
        bh=368TaH0Fb1bLjmVRrNFt63DwQGMzKT1SSsjZM/MSzB8=;
        b=efb/V/rmQDC5qkD6J7FFHEoEHxhWoBueGs5pWT8dfA+IY56qQpVvObiPaPv1aBWKGs
         h0prh5yZyVSoceGW8XdgjtO4WmWJvkDGl4UWu4eTcVYzHRBsT1XajQk2ibok4lJ1VyzE
         XtmHpd5zVyW9bI/NdmISnYhLyNprO/lxYnE1CzFJDqyN3M28ucdfyk+X00gR29SSwGhc
         8Mhtj2P+9G/4ASeVkpHm1Vm38sQmFSQAik9X8FyZ6XYC1cgTtWqezQWJTn9GqTZAEfhm
         28xoGa9kPpqQHuL3eYEmu1YJBrzVij8H0Wz2gtCHfJhm1hsoZxiPae661FnuLGhusc6i
         Bb7w==
ARC-Authentication-Results: i=1; mx.google.com;
       dkim=pass header.i=@nvidia.com header.s=n1 header.b=No71KLEG;
       spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org;
       dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=nvidia.com
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67])
        by mx.google.com with ESMTP id 70si17123898ple.294.2019.03.26.02.35.16;
        Tue, 26 Mar 2019 02:35:31 -0700 (PDT)
Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Authentication-Results: mx.google.com;
       dkim=pass header.i=@nvidia.com header.s=n1 header.b=No71KLEG;
       spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org;
       dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=nvidia.com
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1731223AbfCZJdV (ORCPT <rfc822;morganleezd2003@gmail.com>
        + 99 others); Tue, 26 Mar 2019 05:33:21 -0400
Received: from hqemgate16.nvidia.com ([216.228.121.65]:17510 "EHLO
        hqemgate16.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1726261AbfCZJdU (ORCPT
        <rfc822;linux-kernel@vger.kernel.org>);
        Tue, 26 Mar 2019 05:33:20 -0400
Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqemgate16.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA)
        id <B5c99f1db0000>; Tue, 26 Mar 2019 02:33:15 -0700
Received: from hqmail.nvidia.com ([172.20.161.6])
  by hqpgpgate101.nvidia.com (PGP Universal service);
  Tue, 26 Mar 2019 02:33:17 -0700
X-PGP-Universal: processed;
        by hqpgpgate101.nvidia.com on Tue, 26 Mar 2019 02:33:17 -0700
Received: from [10.24.70.251] (10.124.1.5) by HQMAIL101.nvidia.com
 (172.20.187.10) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 26 Mar
 2019 09:33:10 +0000
Subject: Re: [PATCH v7 8/9] vfio/type1: Add domain at(de)taching group helpers
To:     Lu Baolu <baolu.lu@linux.intel.com>,
        Joerg Roedel <joro@8bytes.org>,
        David Woodhouse <dwmw2@infradead.org>,
        Alex Williamson <alex.williamson@redhat.com>
CC:     <ashok.raj@intel.com>, <sanjay.k.kumar@intel.com>,
        <jacob.jun.pan@intel.com>, <kevin.tian@intel.com>,
        Jean-Philippe Brucker <jean-philippe.brucker@arm.com>,
        <yi.l.liu@intel.com>, <yi.y.sun@intel.com>, <peterx@redhat.com>,
        <tiwei.bie@intel.com>, <xin.zeng@intel.com>,
        <iommu@lists.linux-foundation.org>, <kvm@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>,
        Jacob Pan <jacob.jun.pan@linux.intel.com>,
        Neo Jia <cjia@nvidia.com>
References: <20190222021927.13132-1-baolu.lu@linux.intel.com>
 <20190222021927.13132-9-baolu.lu@linux.intel.com>
X-Nvconfidentiality: public
From:   Kirti Wankhede <kwankhede@nvidia.com>
Message-ID: <4ed10087-1e96-2aea-91a5-93ded001cb0b@nvidia.com>
Date:   Tue, 26 Mar 2019 15:03:06 +0530
MIME-Version: 1.0
In-Reply-To: <20190222021927.13132-9-baolu.lu@linux.intel.com>
X-Originating-IP: [10.124.1.5]
X-ClientProxiedBy: HQMAIL101.nvidia.com (172.20.187.10) To
 HQMAIL101.nvidia.com (172.20.187.10)
Content-Type: text/plain; charset="utf-8"
Content-Language: en-US
Content-Transfer-Encoding: 7bit
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1;
        t=1553592796; bh=368TaH0Fb1bLjmVRrNFt63DwQGMzKT1SSsjZM/MSzB8=;
        h=X-PGP-Universal:Subject:To:CC:References:X-Nvconfidentiality:From:
         Message-ID:Date:MIME-Version:In-Reply-To:X-Originating-IP:
         X-ClientProxiedBy:Content-Type:Content-Language:
         Content-Transfer-Encoding;
        b=No71KLEGNdZKsNVTq7S4CoBXpKsJFvrU5HI+RuiDpoDbvDty7MgdwQko4CT6nb1Ee
         Gs5fEzc1P6gJZWOZDn6IE4LtUJTUaYNLvhhWxWRbhVBEac0QHJ6dCTA8s77XyjbeYo
         LQLQ4xuLetAAi84At+dhJf64Vd00w9gueJG0M5Xt7weZNy6Qc+0rkpCZ5sbFRwoEo0
         BsW1H/5jZNqxEz8Gs7sTR4u45OTj5uZyh0I1VvMBcW5u8dR3+FesQQkPs1yMT4GwlB
         CAMlQZ/vb1fVuNO0l87fQgk/FgqCiVvzTFnxaDGuZZG3uiYOUFrOcJGa6KU2r/F3/i
         28/evKoa91H0w==
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org



On 2/22/2019 7:49 AM, Lu Baolu wrote:
> This adds helpers to attach or detach a domain to a
> group. This will replace iommu_attach_group() which
> only works for non-mdev devices.
> 
> If a domain is attaching to a group which includes the
> mediated devices, it should attach to the iommu device
> (a pci device which represents the mdev in iommu scope)
> instead. The added helper supports attaching domain to
> groups for both pci and mdev devices.
> 
> Cc: Ashok Raj <ashok.raj@intel.com>
> Cc: Jacob Pan <jacob.jun.pan@linux.intel.com>
> Cc: Kevin Tian <kevin.tian@intel.com>
> Signed-off-by: Sanjay Kumar <sanjay.k.kumar@intel.com>
> Signed-off-by: Liu Yi L <yi.l.liu@intel.com>
> Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
> Reviewed-by: Jean-Philippe Brucker <jean-philippe.brucker@arm.com>

Reviewed-by: Kirti Wankhede <kwankhede@nvidia.com>

Thanks,
Kirti


> ---
>  drivers/vfio/vfio_iommu_type1.c | 84 ++++++++++++++++++++++++++++++---
>  1 file changed, 77 insertions(+), 7 deletions(-)
> 
> diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
> index 73652e21efec..ccc4165474aa 100644
> --- a/drivers/vfio/vfio_iommu_type1.c
> +++ b/drivers/vfio/vfio_iommu_type1.c
> @@ -91,6 +91,7 @@ struct vfio_dma {
>  struct vfio_group {
>  	struct iommu_group	*iommu_group;
>  	struct list_head	next;
> +	bool			mdev_group;	/* An mdev group */
>  };
>  
>  /*
> @@ -1298,6 +1299,75 @@ static bool vfio_iommu_has_sw_msi(struct iommu_group *group, phys_addr_t *base)
>  	return ret;
>  }
>  
> +static struct device *vfio_mdev_get_iommu_device(struct device *dev)
> +{
> +	struct device *(*fn)(struct device *dev);
> +	struct device *iommu_device;
> +
> +	fn = symbol_get(mdev_get_iommu_device);
> +	if (fn) {
> +		iommu_device = fn(dev);
> +		symbol_put(mdev_get_iommu_device);
> +
> +		return iommu_device;
> +	}
> +
> +	return NULL;
> +}
> +
> +static int vfio_mdev_attach_domain(struct device *dev, void *data)
> +{
> +	struct iommu_domain *domain = data;
> +	struct device *iommu_device;
> +
> +	iommu_device = vfio_mdev_get_iommu_device(dev);
> +	if (iommu_device) {
> +		if (iommu_dev_feature_enabled(iommu_device, IOMMU_DEV_FEAT_AUX))
> +			return iommu_aux_attach_device(domain, iommu_device);
> +		else
> +			return iommu_attach_device(domain, iommu_device);
> +	}
> +
> +	return -EINVAL;
> +}
> +
> +static int vfio_mdev_detach_domain(struct device *dev, void *data)
> +{
> +	struct iommu_domain *domain = data;
> +	struct device *iommu_device;
> +
> +	iommu_device = vfio_mdev_get_iommu_device(dev);
> +	if (iommu_device) {
> +		if (iommu_dev_feature_enabled(iommu_device, IOMMU_DEV_FEAT_AUX))
> +			iommu_aux_detach_device(domain, iommu_device);
> +		else
> +			iommu_detach_device(domain, iommu_device);
> +	}
> +
> +	return 0;
> +}
> +
> +static int vfio_iommu_attach_group(struct vfio_domain *domain,
> +				   struct vfio_group *group)
> +{
> +	if (group->mdev_group)
> +		return iommu_group_for_each_dev(group->iommu_group,
> +						domain->domain,
> +						vfio_mdev_attach_domain);
> +	else
> +		return iommu_attach_group(domain->domain, group->iommu_group);
> +}
> +
> +static void vfio_iommu_detach_group(struct vfio_domain *domain,
> +				    struct vfio_group *group)
> +{
> +	if (group->mdev_group)
> +		iommu_group_for_each_dev(group->iommu_group, domain->domain,
> +					 vfio_mdev_detach_domain);
> +	else
> +		iommu_detach_group(domain->domain, group->iommu_group);
> +}
> +
>  static int vfio_iommu_type1_attach_group(void *iommu_data,
>  					 struct iommu_group *iommu_group)
>  {
> @@ -1373,7 +1443,7 @@ static int vfio_iommu_type1_attach_group(void *iommu_data,
>  			goto out_domain;
>  	}
>  
> -	ret = iommu_attach_group(domain->domain, iommu_group);
> +	ret = vfio_iommu_attach_group(domain, group);
>  	if (ret)
>  		goto out_domain;
>  
> @@ -1405,8 +1475,8 @@ static int vfio_iommu_type1_attach_group(void *iommu_data,
>  	list_for_each_entry(d, &iommu->domain_list, next) {
>  		if (d->domain->ops == domain->domain->ops &&
>  		    d->prot == domain->prot) {
> -			iommu_detach_group(domain->domain, iommu_group);
> -			if (!iommu_attach_group(d->domain, iommu_group)) {
> +			vfio_iommu_detach_group(domain, group);
> +			if (!vfio_iommu_attach_group(d, group)) {
>  				list_add(&group->next, &d->group_list);
>  				iommu_domain_free(domain->domain);
>  				kfree(domain);
> @@ -1414,7 +1484,7 @@ static int vfio_iommu_type1_attach_group(void *iommu_data,
>  				return 0;
>  			}
>  
> -			ret = iommu_attach_group(domain->domain, iommu_group);
> +			ret = vfio_iommu_attach_group(domain, group);
>  			if (ret)
>  				goto out_domain;
>  		}
> @@ -1440,7 +1510,7 @@ static int vfio_iommu_type1_attach_group(void *iommu_data,
>  	return 0;
>  
>  out_detach:
> -	iommu_detach_group(domain->domain, iommu_group);
> +	vfio_iommu_detach_group(domain, group);
>  out_domain:
>  	iommu_domain_free(domain->domain);
>  out_free:
> @@ -1531,7 +1601,7 @@ static void vfio_iommu_type1_detach_group(void *iommu_data,
>  		if (!group)
>  			continue;
>  
> -		iommu_detach_group(domain->domain, iommu_group);
> +		vfio_iommu_detach_group(domain, group);
>  		list_del(&group->next);
>  		kfree(group);
>  		/*
> @@ -1596,7 +1666,7 @@ static void vfio_release_domain(struct vfio_domain *domain, bool external)
>  	list_for_each_entry_safe(group, group_tmp,
>  				 &domain->group_list, next) {
>  		if (!external)
> -			iommu_detach_group(domain->domain, group->iommu_group);
> +			vfio_iommu_detach_group(domain, group);
>  		list_del(&group->next);
>  		kfree(group);
>  	}
>