Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Thu, 12 Dec 2019 17:30:13 -0700
From:   Jerry Snitselaar <jsnitsel@redhat.com>
To:     Lu Baolu <baolu.lu@linux.intel.com>
Cc:     Joerg Roedel <joro@8bytes.org>,
        David Woodhouse <dwmw2@infradead.org>, ashok.raj@intel.com,
        jacob.jun.pan@intel.com, kevin.tian@intel.com,
        iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org,
        stable@vger.kernel.org
Subject: Re: [PATCH 1/1] iommu/vt-d: Fix dmar pte read access not set error
Message-ID: <20191213003013.gc3zg3fpzpjntnzg@cantor>
Reply-To: Jerry Snitselaar <jsnitsel@redhat.com>
Mail-Followup-To: Lu Baolu <baolu.lu@linux.intel.com>,
        Joerg Roedel <joro@8bytes.org>,
        David Woodhouse <dwmw2@infradead.org>, ashok.raj@intel.com,
        jacob.jun.pan@intel.com, kevin.tian@intel.com,
        iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org,
        stable@vger.kernel.org
References: <20191211014015.7898-1-baolu.lu@linux.intel.com>
 <20191212014952.vlrmxrk2cebwxjnp@cantor>
 <6f3bcad9-b9b3-b349-fdad-ce53a79a665b@linux.intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <6f3bcad9-b9b3-b349-fdad-ce53a79a665b@linux.intel.com>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Thu Dec 12 19, Lu Baolu wrote:
>Hi,
>
>On 12/12/19 9:49 AM, Jerry Snitselaar wrote:
>>On Wed Dec 11 19, Lu Baolu wrote:
>>>If the default DMA domain of a group doesn't fit a device, it
>>>will still sit in the group but use a private identity domain.
>>>When map/unmap/iova_to_phys come through iommu API, the driver
>>>should still serve them, otherwise, other devices in the same
>>>group will be impacted. Since identity domain has been mapped
>>>with the whole available memory space and RMRRs, we don't need
>>>to worry about the impact on it.
>>>
>>>Link: https://www.spinics.net/lists/iommu/msg40416.html
>>>Cc: Jerry Snitselaar <jsnitsel@redhat.com>
>>>Reported-by: Jerry Snitselaar <jsnitsel@redhat.com>
>>>Fixes: 942067f1b6b97 ("iommu/vt-d: Identify default domains 
>>>replaced with private")
>>>Cc: stable@vger.kernel.org # v5.3+
>>>Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
>>
>>Reviewed-by: Jerry Snitselaar <jsnitsel@redhat.com>
>
>Can you please try this fix and check whether it can fix your problem?
>If it helps, do you mind adding a Tested-by?
>
>Best regards,
>baolu
>

I'm testing with this patch, my patch that moves the direct mapping call,
and Alex's patch for the ISA bridge. It solved the 2 iommu mapping errors
I was seeing with default passthrough, I no longer see all the dmar pte
read access errors, and the system boots allowing me to login. I'm tracking
down 2 issues at the moment. With passthrough I see a problem with 01:00.4
that I mentioned in the earlier email:

[   78.978573] uhci_hcd: USB Universal Host Controller Interface driver
[   78.980842] uhci_hcd 0000:01:00.4: UHCI Host Controller
[   78.982738] uhci_hcd 0000:01:00.4: new USB bus registered, assigned bus number 3
[   78.985222] uhci_hcd 0000:01:00.4: detected 8 ports
[   78.986907] uhci_hcd 0000:01:00.4: port count misdetected? forcing to 2 ports
[   78.989316] uhci_hcd 0000:01:00.4: irq 16, io base 0x00003c00
[   78.994634] uhci_hcd 0000:01:00.4: DMAR: 32bit DMA uses non-identity mapping
[   7 0000:01:00.4: unable to allocate consistent memory for frame list
[   79.499891] uhci_hcd 0000:01:00.4: startup error -16
[   79.501588] uhci_hcd 0000:01:00.4: USB bus 3 deregistered
[   79.503494] uhci_hcd 0000:01:00.4: init 0000:01:00.4 fail, -16
[   79.505497] uhci_hcd: probe of 0000:01:00.4 failed with error -16

If I boot the system with iommu=nopt I see an iommu map failure due to
the prot check in __domain_mapping:

[   40.940589] pci 0000:00:1f.0: iommu_group_add_device: calling iommu_group_create_direct_mappings
[   40.943558] pci 0000:00:1f.0: iommu_group_create_direct_mappings: iterating through mappings
[   40.946402] pci 0000:00:1f.0: iommu_group_create_direct_mappings: calling apply_resv_region
[   40.949184] pci 0000:00:1f.0: iommu_group_create_direct_mappings: entry type is direct
[   40.951819] DMAR: intel_iommu_map: enter
[   40.953128] DMAR: __domain_mapping: prot & (DMA_PTE_READ|DMA_PTE_WRITE) == 0
[   40.955486] DMAR: domain_mapping: __domain_mapping failed
[   40.957348] DMAR: intel_iommu_map: domain_pfn_mapping returned -22
[   40.959466] DMAR: intel_iommu_map: leave
[   40.959468] iommu: iommu_map: ops->map failed iova 0x0 pa 0x0000000000000000 pgsize 0x1000
[   40.963511] pci 0000:00:1f.0: iommu_group_create_direct_mappings: iommu_map failed
[   40.966026] pci 0000:00:1f.0: iommu_group_create_direct_mappings: leaving func
[   40.968487] pci 0000:00:1f.0: iommu_group_add_device: calling __iommu_attach_device
[   40.971016] pci 0000:00:1f.0: Adding to iommu group 19
[   40.972731] pci 0000:00:1f.0: DMAR: domain->type is dma

/sys/kernel/iommu_groups/19
[root@hp-dl388g8-07 19]# cat reserved_regions 
0x0000000000000000 0x0000000000ffffff direct
0x00000000bdf6e000 0x00000000bdf84fff direct
0x00000000fee00000 0x00000000feefffff msi

00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller

>>
>>>---
>>>drivers/iommu/intel-iommu.c | 8 --------
>>>1 file changed, 8 deletions(-)
>>>
>>>diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
>>>index 0c8d81f56a30..b73bebea9148 100644
>>>--- a/drivers/iommu/intel-iommu.c
>>>+++ b/drivers/iommu/intel-iommu.c
>>>@@ -5478,9 +5478,6 @@ static int intel_iommu_map(struct 
>>>iommu_domain *domain,
>>>    int prot = 0;
>>>    int ret;
>>>
>>>-    if (dmar_domain->flags & DOMAIN_FLAG_LOSE_CHILDREN)
>>>-        return -EINVAL;
>>>-
>>>    if (iommu_prot & IOMMU_READ)
>>>        prot |= DMA_PTE_READ;
>>>    if (iommu_prot & IOMMU_WRITE)
>>>@@ -5523,8 +5520,6 @@ static size_t intel_iommu_unmap(struct 
>>>iommu_domain *domain,
>>>    /* Cope with horrid API which requires us to unmap more than the
>>>       size argument if it happens to be a large-page mapping. */
>>>    BUG_ON(!pfn_to_dma_pte(dmar_domain, iova >> VTD_PAGE_SHIFT, &level));
>>>-    if (dmar_domain->flags & DOMAIN_FLAG_LOSE_CHILDREN)
>>>-        return 0;
>>>
>>>    if (size < VTD_PAGE_SIZE << level_to_offset_bits(level))
>>>        size = VTD_PAGE_SIZE << level_to_offset_bits(level);
>>>@@ -5556,9 +5551,6 @@ static phys_addr_t 
>>>intel_iommu_iova_to_phys(struct iommu_domain *domain,
>>>    int level = 0;
>>>    u64 phys = 0;
>>>
>>>-    if (dmar_domain->flags & DOMAIN_FLAG_LOSE_CHILDREN)
>>>-        return 0;
>>>-
>>>    pte = pfn_to_dma_pte(dmar_domain, iova >> VTD_PAGE_SHIFT, &level);
>>>    if (pte)
>>>        phys = dma_pte_addr(pte);
>>>-- 
>>>2.17.1
>>>
>>
>_______________________________________________
>iommu mailing list
>iommu@lists.linux-foundation.org
>https://lists.linuxfoundation.org/mailman/listinfo/iommu