Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751115AbdGMGsU (ORCPT ); Thu, 13 Jul 2017 02:48:20 -0400 Received: from mail-it0-f47.google.com ([209.85.214.47]:34757 "EHLO mail-it0-f47.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750916AbdGMGsS (ORCPT ); Thu, 13 Jul 2017 02:48:18 -0400 MIME-Version: 1.0 In-Reply-To: <20170713031234.GD4486@bhelgaas-glaptop.roam.corp.google.com> References: <20170710155230.8622-1-marc.zyngier@arm.com> <20170713031234.GD4486@bhelgaas-glaptop.roam.corp.google.com> From: Ard Biesheuvel Date: Thu, 13 Jul 2017 07:48:16 +0100 Message-ID: Subject: Re: [PATCH 0/2] Workaround for uPD72020x USB3 chips To: Bjorn Helgaas Cc: Marc Zyngier , Bjorn Helgaas , Mathias Nyman , Greg Kroah-Hartman , linux-pci , "linux-kernel@vger.kernel.org" , linux-usb Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3109 Lines: 69 On 13 July 2017 at 04:12, Bjorn Helgaas wrote: > On Mon, Jul 10, 2017 at 04:52:28PM +0100, Marc Zyngier wrote: >> Ard and myself have just spent quite some time lately trying to pin >> down an issue in the DMA code which was taking the form of a PCIe USB3 >> controller issuing a DMA access at some bizarre address, and being >> caught red-handed by the IOMMU. >> >> After much head scratching and most of a week-end spent on tracing the >> damn thing, I'm now convinced that the DMA code is fine, the XHCI >> driver is correct, but that the HW (a Renesas uPD720202 chip) is a >> nasty piece of work. >> >> The issue is as follow: >> >> - EFI initializes the controller using physical addresses above the >> 4GB limit (this is on an arm64 box where the memory starts at >> 0x80_00000000...). >> >> - The kernel takes over, sends a XHCI reset to the controller, and >> because we have an IOMMU sitting between the controller and memory, >> provides *virtual* addresses. Trying to make things a bit faster for >> our controller, it issues IOVAs in the low 4GB range). >> >> - Low and behold, the controller is now issuing transactions with a >> 0x80 prefix in front of our IOVA. Yes, the same prefix that was >> programmed during the EFI configuration. IOMMU fault, not happy. >> >> If the kernel is hacked to only generate IOVAs that are more than >> 32bit wide, the HW behaves correctly. The only way I can explain this >> behaviour is that the HW latches the top 32bit of the ERST (it is >> always the ERST IOVA that appears in my traces) in some internal >> register, and that the XHCI reset fails to clear it. Writing zero in >> the top bits is not enough to clear it either. >> >> So far, the only solution we have for this lovely piece of kit is to >> force a PCI reset at probe time, which puts it right. The patches are >> pretty ugly, but that's the best I could come up with so far. >> >> Tested on a pair of AMD Opteron 1100 boxes with Renesas uPD720201 and >> uPD720202 controllers. >> >> Marc Zyngier (2): >> PCI: Implement pci_reset_function_locked >> usb: host: pci_quirks: Force hard reset of Renesas uPD72020x USB >> controller >> >> drivers/pci/pci.c | 35 +++++++++++++++++++++++++++++++++++ >> drivers/usb/host/pci-quirks.c | 20 ++++++++++++++++++++ >> drivers/usb/host/pci-quirks.h | 1 + >> drivers/usb/host/xhci-pci.c | 7 +++++++ >> include/linux/pci.h | 1 + >> 5 files changed, 64 insertions(+) > > I provisionally applied this to pci/virtualization. I'd like to have an > XHCI ack before going further, though. > > I assume this only affects boxes where the firmware uses addresses above > 4GB, i.e., not very many? So this is v4.14 material? Or do you think it's > important for v4.13? > As I mentioned, it would be nice if this could at least go into v4.11 and later, given that v4.11 contains a patch that switches all PCI devices to 32-bit addressing only when the IOMMU is involved in DMA, and this is what triggered the issue on arm64 boards with such a PCI card and no DRAM below 4 GB. Thanks, Ard.