Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754172AbcKQCqL (ORCPT ); Wed, 16 Nov 2016 21:46:11 -0500 Received: from mx1.redhat.com ([209.132.183.28]:49524 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752999AbcKQCqK (ORCPT ); Wed, 16 Nov 2016 21:46:10 -0500 Reply-To: xlpang@redhat.com Subject: Re: [PATCH] iommu/vt-d: Flush old iotlb for kdump when the device gets context mapped References: <1479286950-21885-1-git-send-email-xlpang@redhat.com> <582C232F.6080205@redhat.com> To: Myron Stowe , xlpang@redhat.com Cc: iommu@lists.linux-foundation.org, Joerg Roedel , Don Brace , Dave Young , kexec@lists.infradead.org, LKML , Myron Stowe , David Woodhouse From: Xunlei Pang Message-ID: <582D1A40.409@redhat.com> Date: Thu, 17 Nov 2016 10:47:28 +0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Thu, 17 Nov 2016 02:46:09 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4280 Lines: 81 On 2016/11/16 at 22:58, Myron Stowe wrote: > On Wed, Nov 16, 2016 at 2:13 AM, Xunlei Pang wrote: >> Ccing David >> On 2016/11/16 at 17:02, Xunlei Pang wrote: >>> We met the DMAR fault both on hpsa P420i and P421 SmartArray controllers >>> under kdump, it can be steadily reproduced on several different machines, >>> the dmesg log is like: >>> HP HPSA Driver (v 3.4.16-0) >>> hpsa 0000:02:00.0: using doorbell to reset controller >>> hpsa 0000:02:00.0: board ready after hard reset. >>> hpsa 0000:02:00.0: Waiting for controller to respond to no-op >>> DMAR: Setting identity map for device 0000:02:00.0 [0xe8000 - 0xe8fff] >>> DMAR: Setting identity map for device 0000:02:00.0 [0xf4000 - 0xf4fff] >>> DMAR: Setting identity map for device 0000:02:00.0 [0xbdf6e000 - 0xbdf6efff] >>> DMAR: Setting identity map for device 0000:02:00.0 [0xbdf6f000 - 0xbdf7efff] >>> DMAR: Setting identity map for device 0000:02:00.0 [0xbdf7f000 - 0xbdf82fff] >>> DMAR: Setting identity map for device 0000:02:00.0 [0xbdf83000 - 0xbdf84fff] >>> DMAR: DRHD: handling fault status reg 2 >>> DMAR: [DMA Read] Request device [02:00.0] fault addr fffff000 [fault reason 06] PTE Read access is not set >>> hpsa 0000:02:00.0: controller message 03:00 timed out >>> hpsa 0000:02:00.0: no-op failed; re-trying >>> >>> After some debugging, we found that the corresponding pte entry value >>> is correct, and the value of the iommu caching mode is 0, the fault is >>> probably due to the old iotlb cache of the in-flight DMA. >>> >>> Thus need to flush the old iotlb after context mapping is setup for the >>> device, where the device is supposed to finish reset at its driver probe >>> stage and no in-flight DMA exists hereafter. >>> >>> With this patch, all our problematic machines can survive the kdump tests. >>> >>> CC: Myron Stowe >>> CC: Don Brace >>> CC: Baoquan He >>> CC: Dave Young >>> Tested-by: Joseph Szczypek >>> Signed-off-by: Xunlei Pang >>> --- >>> drivers/iommu/intel-iommu.c | 11 +++++++++-- >>> 1 file changed, 9 insertions(+), 2 deletions(-) >>> >>> diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c >>> index 3965e73..eb79288 100644 >>> --- a/drivers/iommu/intel-iommu.c >>> +++ b/drivers/iommu/intel-iommu.c >>> @@ -2067,9 +2067,16 @@ static int domain_context_mapping_one(struct dmar_domain *domain, >>> * It's a non-present to present mapping. If hardware doesn't cache >>> * non-present entry we only need to flush the write-buffer. If the >>> * _does_ cache non-present entries, then it does so in the special > If this does get accepted then we should fix the above grammar also - > "If the _does_ cache ..." -> "If the hardware _does_ cache ..." Yes, but this reminds me of something. As per the comment, the code here only needs to flush context caches for the special domain 0 which is used to tag the non-present/erroneous caches, seems we should flush the old domain id of present entries for kdump according to the analysis, other than the new-allocated domain id. Let me ponder more on this. Regards, Xunlei > >>> - * domain #0, which we have to flush: >>> + * domain #0, which we have to flush. >>> + * >>> + * For kdump cases, present entries may be cached due to the in-flight >>> + * DMA and copied old pgtable, but there is no unmapping behaviour for >>> + * them, so we need an explicit iotlb flush for the newly-mapped device. >>> + * For kdump, at this point, the device is supposed to finish reset at >>> + * the driver probe stage, no in-flight DMA will exist, thus we do not >>> + * need to worry about that anymore hereafter. >>> */ >>> - if (cap_caching_mode(iommu->cap)) { >>> + if (is_kdump_kernel() || cap_caching_mode(iommu->cap)) { >>> iommu->flush.flush_context(iommu, 0, >>> (((u16)bus) << 8) | devfn, >>> DMA_CCMD_MASK_NOBIT, >> _______________________________________________ >> iommu mailing list >> iommu@lists.linux-foundation.org >> https://lists.linuxfoundation.org/mailman/listinfo/iommu