From: Don Brace <don.brace@microsemi.com>
To: Xunlei Pang <xlpang@redhat.com>, Joerg Roedel <joro@8bytes.org>,
        "David Woodhouse" <dwmw2@infradead.org>
CC: "iommu@lists.linux-foundation.org" <iommu@lists.linux-foundation.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Myron Stowe <myron.stowe@gmail.com>,
        Joseph Szczypek <jszczype@redhat.com>, Baoquan He <bhe@redhat.com>,
        Dave Young <dyoung@redhat.com>
Subject: RE: [PATCH v3] iommu/vt-d: Flush old iommu caches for kdump when
 the device gets context mapped
Thread-Topic: [PATCH v3] iommu/vt-d: Flush old iommu caches for kdump when
 the device gets context mapped
Thread-Index: AQHSTvA63D4aFyvMQEO5E76Bba3rhKD7ewaA
Date: Tue, 6 Dec 2016 22:03:20 +0000
Message-ID: <4993A297653ECB4581FA5C3C31323D1941809326@avsrvexchmbx1.microsemi.net>
References: <1480939747-31916-1-git-send-email-xlpang@redhat.com>
In-Reply-To: <1480939747-31916-1-git-send-email-xlpang@redhat.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
SpamDiagnosticOutput: 1:99
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Dec 2016 22:03:22.6951
 (UTC)
X-MS-Exchange-CrossTenant-Id: f267a5c8-86d8-4cc9-af71-1fd2c67c8fad
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f267a5c8-86d8-4cc9-af71-1fd2c67c8fad;Ip=[208.19.100.21];Helo=[avsrvexchhts1.microsemi.net]
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR02MB2256
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by mail.home.local id uB6Mb3aR018415
Content-Length: 4680
Lines: 112

> -----Original Message-----
> From: Xunlei Pang [mailto:xlpang@redhat.com]
> Sent: Monday, December 05, 2016 6:09 AM
> To: Joerg Roedel; David Woodhouse
> Cc: iommu@lists.linux-foundation.org; linux-kernel@vger.kernel.org; Xunlei
> Pang; Myron Stowe; Joseph Szczypek; Don Brace; Baoquan He; Dave Young
> Subject: [PATCH v3] iommu/vt-d: Flush old iommu caches for kdump when
> the device gets context mapped
> 
> EXTERNAL EMAIL
> 
> 
> We met the DMAR fault both on hpsa P420i and P421 SmartArray controllers
> under kdump, it can be steadily reproduced on several different machines,
> the dmesg log is like:
> HP HPSA Driver (v 3.4.16-0)
> hpsa 0000:02:00.0: using doorbell to reset controller
> hpsa 0000:02:00.0: board ready after hard reset.
> hpsa 0000:02:00.0: Waiting for controller to respond to no-op
> DMAR: Setting identity map for device 0000:02:00.0 [0xe8000 - 0xe8fff]
> DMAR: Setting identity map for device 0000:02:00.0 [0xf4000 - 0xf4fff]
> DMAR: Setting identity map for device 0000:02:00.0 [0xbdf6e000 -
> 0xbdf6efff]
> DMAR: Setting identity map for device 0000:02:00.0 [0xbdf6f000 - 0xbdf7efff]
> DMAR: Setting identity map for device 0000:02:00.0 [0xbdf7f000 - 0xbdf82fff]
> DMAR: Setting identity map for device 0000:02:00.0 [0xbdf83000 - 0xbdf84fff]
> DMAR: DRHD: handling fault status reg 2
> DMAR: [DMA Read] Request device [02:00.0] fault addr fffff000 [fault reason
> 06] PTE Read access is not set
> hpsa 0000:02:00.0: controller message 03:00 timed out
> hpsa 0000:02:00.0: no-op failed; re-trying
> 
> After some debugging, we found that the fault addr is from DMA initiated at
> the driver probe stage after reset(not in-flight DMA), and the corresponding
> pte entry value is correct, the fault is likely due to the old iommu caches
> of the in-flight DMA before it.
> 
> Thus we need to flush the old cache after context mapping is setup for the
> device, where the device is supposed to finish reset at its driver probe
> stage and no in-flight DMA exists hereafter.
> 
> I'm not sure if the hardware is responsible for invalidating all the related
> caches allocated in the iommu hardware before, but seems not the case for
> hpsa,
> actually many device drivers have problems in properly resetting the
> hardware.
> Anyway flushing (again) by software in kdump kernel when the device gets
> context
> mapped which is a quite infrequent operation does little harm.
> 
> With this patch, the problematic machine can survive the kdump tests.
> 
> CC: Myron Stowe <myron.stowe@gmail.com>
> CC: Joseph Szczypek <jszczype@redhat.com>
> CC: Don Brace <don.brace@microsemi.com>
> CC: Baoquan He <bhe@redhat.com>
> CC: Dave Young <dyoung@redhat.com>
> Fixes: 091d42e43d21 ("iommu/vt-d: Copy translation tables from old kernel")
> Fixes: dbcd861f252d ("iommu/vt-d: Do not re-use domain-ids from the old
> kernel")
> Fixes: cf484d0e6939 ("iommu/vt-d: Mark copied context entries")
> Signed-off-by: Xunlei Pang <xlpang@redhat.com>
> ---
> v2->v3:
> Flush context cache only and add Fixes-tag, according to Joerg's comments.
> 
>  drivers/iommu/intel-iommu.c | 19 +++++++++++++++++++
>  1 file changed, 19 insertions(+)
> 
> diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
> index 3965e73..624eac9 100644
> --- a/drivers/iommu/intel-iommu.c
> +++ b/drivers/iommu/intel-iommu.c
> @@ -2024,6 +2024,25 @@ static int domain_context_mapping_one(struct
> dmar_domain *domain,
>         if (context_present(context))
>                 goto out_unlock;
> 
> +       /*
> +        * For kdump cases, old valid entries may be cached due to the
> +        * in-flight DMA and copied pgtable, but there is no unmapping
> +        * behaviour for them, thus we need an explicit cache flush for
> +        * the newly-mapped device. For kdump, at this point, the device
> +        * is supposed to finish reset at its driver probe stage, so no
> +        * in-flight DMA will exist, and we don't need to worry anymore
> +        * hereafter.
> +        */
> +       if (context_copied(context)) {
> +               u16 did_old = context_domain_id(context);
> +
> +               if (did_old >= 0 && did_old < cap_ndoms(iommu->cap))
> +                       iommu->flush.flush_context(iommu, did_old,
> +                                                  (((u16)bus) << 8) | devfn,
> +                                                  DMA_CCMD_MASK_NOBIT,
> +                                                  DMA_CCMD_DEVICE_INVL);
> +       }
> +
>         pgd = domain->pgd;
> 
>         context_clear_entry(context);
> --
> 1.8.3.1

Tested-by: Don Brace <don.brace@microsemi.com>

Thanks,
Don Brace
ESC - Smart Storage
Microsemi Corporation