Received: by 2002:ac0:a5a6:0:0:0:0:0 with SMTP id m35-v6csp208794imm; Wed, 29 Aug 2018 18:40:48 -0700 (PDT) X-Google-Smtp-Source: ANB0VdaZax6UBy4mfSBlRHHebEGfYH2PqpFQy85K6M4JrpvVF2fB9boJ7eCnu25iSokoDVN8KMhP X-Received: by 2002:a63:4c54:: with SMTP id m20-v6mr7809221pgl.292.1535593248603; Wed, 29 Aug 2018 18:40:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535593248; cv=none; d=google.com; s=arc-20160816; b=vFgBPrM7tCqief1PXS4RxZNmuJGMSanh/RSLYqQUsMIbQTe8F83KU4BaW/SZTUrLTP De0YGFRvL0yPsQdllY7Ub6afo/VduSMqrmjWjpI42STdScnbwBlM80wvwU0+buAPUn8e ZhMkfU/vqH5KNi4jbOIHhOO/3GPKq0+7wqCteO0KjSvZ0jsvFJibAOkAr/oVtbNrdgW8 ThiyBeUHmtkI3x8kTFPqqOZIxOWG8DdkpAPe0bbyq5XjvAohozwCrDu12f2xe8W+lZMk SGHHhcBi576QkK3idIGg3WsLvYynPuXgJpnJ2ngW/SSdN5tKWD2ooATZXuyTlWFMd9RP MVAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=3TmKepMvXw1KD6fmv+tyrZdeT8haoR+P2hcSd01hJCs=; b=l4BKYLrqi9GonxfDpay4w/jTaaTbZhViuv/I7UkmgjFKlZCsRiInBYR1/ypqjFv9gb 7xk5Aplt6aNEkAUnimSizLf5EYtfPVV1+d4RyFZcSSSqm+6mdX9f2uKX/CV1L2z/IXMU Qu5tvjS0vbxjESe9B0f5VTSDOB+UKlotHwmguoerEWbXcQggFVGnHHYDTpb3CGe7SRN9 uUbdIRN62spom+1QNbikyNo8/ijhM/NzMKCRVqg4i7GEL68pdDcTMAfdIBdSzeiy3lU7 cw5MecE95OTU60ud33HP9HfkP91fDoUULYKi0h+oQ8XpH0WgUfDG6jrWdfoMAqqrI5ya Y9nA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l1-v6si5402926pgo.377.2018.08.29.18.40.33; Wed, 29 Aug 2018 18:40:48 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727336AbeH3FgY (ORCPT + 99 others); Thu, 30 Aug 2018 01:36:24 -0400 Received: from mga11.intel.com ([192.55.52.93]:9350 "EHLO mga11.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727258AbeH3FgX (ORCPT ); Thu, 30 Aug 2018 01:36:23 -0400 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 29 Aug 2018 18:36:45 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,305,1531810800"; d="scan'208";a="79442236" Received: from allen-box.sh.intel.com ([10.239.161.122]) by orsmga003.jf.intel.com with ESMTP; 29 Aug 2018 18:36:37 -0700 From: Lu Baolu To: Joerg Roedel , David Woodhouse Cc: ashok.raj@intel.com, sanjay.k.kumar@intel.com, jacob.jun.pan@intel.com, kevin.tian@intel.com, yi.l.liu@intel.com, yi.y.sun@intel.com, peterx@redhat.com, Jean-Philippe Brucker , iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org, Lu Baolu , Jacob Pan Subject: [PATCH v2 02/12] iommu/vt-d: Manage scalalble mode PASID tables Date: Thu, 30 Aug 2018 09:35:14 +0800 Message-Id: <20180830013524.28743-3-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180830013524.28743-1-baolu.lu@linux.intel.com> References: <20180830013524.28743-1-baolu.lu@linux.intel.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In scalable mode, pasid structure is a two level table with a pasid directory table and a pasid table. Any pasid entry can be identified by a pasid value in below way. 1 9 6 5 0 .-----------------------.-------. | PASID | | '-----------------------'-------' .-------------. | | | | | | | | | | | | | .-----------. | .-------------. | | | |----->| PASID Entry | | | | | '-------------' | | | |Plus | | | .-----------. | | | |---->| DIR Entry |-------->| | | '-----------' '-------------' .---------. |Plus | | | Context | | | | | Entry |------->| | '---------' '-----------' This changes the pasid table APIs to support scalable mode PASID directory and PASID table. It also adds a helper to get the PASID table entry according to the pasid value. Cc: Ashok Raj Cc: Jacob Pan Cc: Kevin Tian Cc: Liu Yi L Signed-off-by: Sanjay Kumar Signed-off-by: Lu Baolu Reviewed-by: Ashok Raj --- drivers/iommu/intel-iommu.c | 2 +- drivers/iommu/intel-pasid.c | 72 ++++++++++++++++++++++++++++++++----- drivers/iommu/intel-pasid.h | 10 +++++- drivers/iommu/intel-svm.c | 6 +--- 4 files changed, 74 insertions(+), 16 deletions(-) diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c index 5845edf4dcf9..b0da4f765274 100644 --- a/drivers/iommu/intel-iommu.c +++ b/drivers/iommu/intel-iommu.c @@ -2507,7 +2507,7 @@ static struct dmar_domain *dmar_insert_one_dev_info(struct intel_iommu *iommu, if (dev) dev->archdata.iommu = info; - if (dev && dev_is_pci(dev) && info->pasid_supported) { + if (dev && dev_is_pci(dev) && sm_supported(iommu)) { ret = intel_pasid_alloc_table(dev); if (ret) { __dmar_remove_one_dev_info(info); diff --git a/drivers/iommu/intel-pasid.c b/drivers/iommu/intel-pasid.c index fe95c9bd4d33..d6e90cd5b062 100644 --- a/drivers/iommu/intel-pasid.c +++ b/drivers/iommu/intel-pasid.c @@ -127,8 +127,7 @@ int intel_pasid_alloc_table(struct device *dev) int ret, order; info = dev->archdata.iommu; - if (WARN_ON(!info || !dev_is_pci(dev) || - !info->pasid_supported || info->pasid_table)) + if (WARN_ON(!info || !dev_is_pci(dev) || info->pasid_table)) return -EINVAL; /* DMA alias device already has a pasid table, use it: */ @@ -143,8 +142,9 @@ int intel_pasid_alloc_table(struct device *dev) return -ENOMEM; INIT_LIST_HEAD(&pasid_table->dev); - size = sizeof(struct pasid_entry); + size = sizeof(struct pasid_dir_entry); count = min_t(int, pci_max_pasids(to_pci_dev(dev)), intel_pasid_max_id); + count >>= PASID_PDE_SHIFT; order = get_order(size * count); pages = alloc_pages_node(info->iommu->node, GFP_ATOMIC | __GFP_ZERO, @@ -154,7 +154,7 @@ int intel_pasid_alloc_table(struct device *dev) pasid_table->table = page_address(pages); pasid_table->order = order; - pasid_table->max_pasid = count; + pasid_table->max_pasid = count << PASID_PDE_SHIFT; attach_out: device_attach_pasid_table(info, pasid_table); @@ -162,14 +162,33 @@ int intel_pasid_alloc_table(struct device *dev) return 0; } +/* Get PRESENT bit of a PASID directory entry. */ +static inline bool +pasid_pde_is_present(struct pasid_dir_entry *pde) +{ + return READ_ONCE(pde->val) & PASID_PTE_PRESENT; +} + +/* Get PASID table from a PASID directory entry. */ +static inline struct pasid_entry * +get_pasid_table_from_pde(struct pasid_dir_entry *pde) +{ + if (!pasid_pde_is_present(pde)) + return NULL; + + return phys_to_virt(READ_ONCE(pde->val) & PDE_PFN_MASK); +} + void intel_pasid_free_table(struct device *dev) { struct device_domain_info *info; struct pasid_table *pasid_table; + struct pasid_dir_entry *dir; + struct pasid_entry *table; + int i, max_pde; info = dev->archdata.iommu; - if (!info || !dev_is_pci(dev) || - !info->pasid_supported || !info->pasid_table) + if (!info || !dev_is_pci(dev) || !info->pasid_table) return; pasid_table = info->pasid_table; @@ -178,6 +197,14 @@ void intel_pasid_free_table(struct device *dev) if (!list_empty(&pasid_table->dev)) return; + /* Free scalable mode PASID directory tables: */ + dir = pasid_table->table; + max_pde = pasid_table->max_pasid >> PASID_PDE_SHIFT; + for (i = 0; i < max_pde; i++) { + table = get_pasid_table_from_pde(&dir[i]); + free_pgtable_page(table); + } + free_pages((unsigned long)pasid_table->table, pasid_table->order); kfree(pasid_table); } @@ -206,17 +233,37 @@ int intel_pasid_get_dev_max_id(struct device *dev) struct pasid_entry *intel_pasid_get_entry(struct device *dev, int pasid) { + struct device_domain_info *info; struct pasid_table *pasid_table; + struct pasid_dir_entry *dir; struct pasid_entry *entries; + int dir_index, index; pasid_table = intel_pasid_get_table(dev); if (WARN_ON(!pasid_table || pasid < 0 || pasid >= intel_pasid_get_dev_max_id(dev))) return NULL; - entries = pasid_table->table; + dir = pasid_table->table; + info = dev->archdata.iommu; + dir_index = pasid >> PASID_PDE_SHIFT; + index = pasid & PASID_PTE_MASK; + + spin_lock(&pasid_lock); + entries = get_pasid_table_from_pde(&dir[dir_index]); + if (!entries) { + entries = alloc_pgtable_page(info->iommu->node); + if (!entries) { + spin_unlock(&pasid_lock); + return NULL; + } + + WRITE_ONCE(dir[dir_index].val, + (u64)virt_to_phys(entries) | PASID_PTE_PRESENT); + } + spin_unlock(&pasid_lock); - return &entries[pasid]; + return &entries[index]; } /* @@ -224,7 +271,14 @@ struct pasid_entry *intel_pasid_get_entry(struct device *dev, int pasid) */ static inline void pasid_clear_entry(struct pasid_entry *pe) { - WRITE_ONCE(pe->val, 0); + WRITE_ONCE(pe->val[0], 0); + WRITE_ONCE(pe->val[1], 0); + WRITE_ONCE(pe->val[2], 0); + WRITE_ONCE(pe->val[3], 0); + WRITE_ONCE(pe->val[4], 0); + WRITE_ONCE(pe->val[5], 0); + WRITE_ONCE(pe->val[6], 0); + WRITE_ONCE(pe->val[7], 0); } void intel_pasid_clear_entry(struct device *dev, int pasid) diff --git a/drivers/iommu/intel-pasid.h b/drivers/iommu/intel-pasid.h index 1c05ed6fc5a5..12f480c2bb8b 100644 --- a/drivers/iommu/intel-pasid.h +++ b/drivers/iommu/intel-pasid.h @@ -12,11 +12,19 @@ #define PASID_MIN 0x1 #define PASID_MAX 0x100000 +#define PASID_PTE_MASK 0x3F +#define PASID_PTE_PRESENT 1 +#define PDE_PFN_MASK PAGE_MASK +#define PASID_PDE_SHIFT 6 -struct pasid_entry { +struct pasid_dir_entry { u64 val; }; +struct pasid_entry { + u64 val[8]; +}; + /* The representative of a PASID table */ struct pasid_table { void *table; /* pasid table pointer */ diff --git a/drivers/iommu/intel-svm.c b/drivers/iommu/intel-svm.c index 4a03e5090952..6c0bd9ee9602 100644 --- a/drivers/iommu/intel-svm.c +++ b/drivers/iommu/intel-svm.c @@ -65,8 +65,6 @@ int intel_svm_init(struct intel_iommu *iommu) order = get_order(sizeof(struct pasid_entry) * iommu->pasid_max); if (ecap_dis(iommu->ecap)) { - /* Just making it explicit... */ - BUILD_BUG_ON(sizeof(struct pasid_entry) != sizeof(struct pasid_state_entry)); pages = alloc_pages(GFP_KERNEL | __GFP_ZERO, order); if (pages) iommu->pasid_state_table = page_address(pages); @@ -406,9 +404,7 @@ int intel_svm_bind_mm(struct device *dev, int *pasid, int flags, struct svm_dev_ pasid_entry_val |= PASID_ENTRY_FLPM_5LP; entry = intel_pasid_get_entry(dev, svm->pasid); - entry->val = pasid_entry_val; - - wmb(); + WRITE_ONCE(entry->val[0], pasid_entry_val); /* * Flush PASID cache when a PASID table entry becomes -- 2.17.1