Received: by 2002:ab2:23c8:0:b0:1f2:fdbc:cb93 with SMTP id a8csp171950lqe; Wed, 27 Mar 2024 02:14:57 -0700 (PDT) X-Forwarded-Encrypted: i=3; AJvYcCV5KxXv0Nbg/F/9QMcGWh2l5QUtBYvIDIlPbU78917o8WKEWweZNXfYuSDpT9yCf/hfKYv0c9jFSk41n+y0kmf2AQQ27vo6n1MxKi8rRQ== X-Google-Smtp-Source: AGHT+IHHYsjBGJNpYPyq+aTictX72JPzd6XqHSLHn5ap9A9lfJgJRjWAKoVh09cr+gPugC/qEl6x X-Received: by 2002:a05:620a:311:b0:78a:3fa2:cd7d with SMTP id s17-20020a05620a031100b0078a3fa2cd7dmr5987801qkm.25.1711530897408; Wed, 27 Mar 2024 02:14:57 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1711530897; cv=pass; d=google.com; s=arc-20160816; b=rvYQHWmvi1t5gh8AQ+9AGs8uc6xL3vT79PyxHznbQZVenZKf56AEZnN64KbKAhM6iG MYw/3bIWwh1J8m8oB8t+W7OAX0Nx5PwC2Cc++F+Mvk/MMuM56zmOtKlWXovkwiHe6WFc sEbznoQeinXaGIB578R/X3Qt+MgAGgDD+LFAfTxoL2JI/c7Nw6O5vR/vXPFd7TXHnm57 fcGrneelW8I5vpF/K358QY9WfQLY1SSGIrKreY60lEmrF0ThugDn6a3sC1/KLyeUSh+L zSedIzqgJadP3H4kIVH65BFQKSxF22x4FR1cKKvoM1se7/Lf8hkSF/YBD65BQ2/6OfHP qjYw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:list-unsubscribe:list-subscribe:list-id:precedence :subject:date:from:dkim-signature; bh=n+x0WiOVlEoxEWj6JmzpWEM32pTOAvBFw/2lqI7CQ3Y=; fh=v8SCn8Ql//vSpDn0N2fLA9mALwTlveToI6wrVravzxQ=; b=ys5Hx4oYcLAMdRy/ApCfa3G/HJ8ajVWV0NvPdJv6cR0mMrjZdUukXeHffKwxh9Hjym AbgUPalJpqJoLtFtA5N8pPtdu7MWZhkee7WNgSEm2ahLelEF66o6n+GfcTtxoWTOt9Iv OGbc1P2g+1j6+y6hOrTi6OiUC7RNZfNQQeVfa3uAnS63ZJBNIxuzENZdv2V/hb4LiKcT 6HZKQSvmtiIh6gim1wVu+80fnn6dkCiVy80tpbVAqRDUrbmbT99JHhJCRPBSMGwKxb2r IUuH27ASmqzbrho2EH6ixwtkDQSHd8X0zwPDLbQpHim/LYPXuPE+YfKA0sNBdIvtWZTi FX9A==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ucsCVpns; arc=pass (i=1 spf=pass spfdomain=linaro.org dkim=pass dkdomain=linaro.org dmarc=pass fromdomain=linaro.org); spf=pass (google.com: domain of linux-kernel+bounces-120595-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-120595-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id h15-20020a05620a13ef00b0078850338897si9343913qkl.195.2024.03.27.02.14.57 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Mar 2024 02:14:57 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel+bounces-120595-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ucsCVpns; arc=pass (i=1 spf=pass spfdomain=linaro.org dkim=pass dkdomain=linaro.org dmarc=pass fromdomain=linaro.org); spf=pass (google.com: domain of linux-kernel+bounces-120595-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-120595-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id C5D921C268F8 for ; Wed, 27 Mar 2024 09:14:56 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D7A4741757; Wed, 27 Mar 2024 09:14:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="ucsCVpns" Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CDD4738382 for ; Wed, 27 Mar 2024 09:14:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.47 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711530852; cv=none; b=sksGRDlaNCLX+DWv21NPwI79t+Z2pRKUaHwLmEuCDZPvkCKYXjnMb3hOEjDDXcVP97ecPZ8W1EwOQhbPNTIHhhZPT0DxblDHX9dhodG12WLiUhCv//U8Z0GL12tB4mGvkQ6u4MWROBTFxBdTLHxNN5L9aYLNZRRr+AKqXzs3ieQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711530852; c=relaxed/simple; bh=C9itwwrFPiwdNm8Zd8w7cqdwK6rVpgcx54kNU3MQW98=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=t70OWbsuwmIUPGezK3AlfBUAiFZovHUhHQQTZL3UaEhoe6AaXjjpqTSju+5uWTtY185WIt+ka7/8tvm6m/a4G81Uk7roKdbaDOJWQs2WrasUwLBSyCTLfR80lsR7wA6IbvhjyVAzMtG9AUpF9avl0q9oiBYE/KmKW5Pm3yMc1XE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org; spf=pass smtp.mailfrom=linaro.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b=ucsCVpns; arc=none smtp.client-ip=209.85.216.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linaro.org Received: by mail-pj1-f47.google.com with SMTP id 98e67ed59e1d1-2a074187a42so1768180a91.0 for ; Wed, 27 Mar 2024 02:14:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1711530850; x=1712135650; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=n+x0WiOVlEoxEWj6JmzpWEM32pTOAvBFw/2lqI7CQ3Y=; b=ucsCVpnsUGhjrywQhz6t7mVi0L22t6KzHgofmYLvUhje5KS8JLa2y1bv3f+MLoVfMB KWsqHXc+/8g71Bj/fenIcU80XiQATaY8tDwIIsvBec38nDndLdAACVq2RCD0RfEIdIlG o5v6gIooAz+QojLYOo1KoNoE/NTDxt2Q9QoYTjdnyRqPgP1xg+JSgh+Cq0byOirsUN9c PVdQJd8TOuc/bKcsJmGgXaF+8FbYUTN/9WTq3Y8Tm+WQ3jEDiyWJ/uOtbG+J46b+Y0uD 1G2IngqC5jE0x56dHzhRpayUaI++topFQQ/v1LOro5gFrznvGEMesnlqMOUPhTawk2We eQjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711530850; x=1712135650; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=n+x0WiOVlEoxEWj6JmzpWEM32pTOAvBFw/2lqI7CQ3Y=; b=gmfEbHga56vJqx7X6IyAPImAPpqgbh12AClUuWJHIpk7wJd2RnXB52zVamsTpxTKeo es3vwBwOIJNrFtdbcA66bHwkLcpBxxbK8DivGoO6A86b2lCA9UlyTcsKRjF+eQX/4H8O Knzzru3xswE6WIrdXf2q45bFAxybBeKjEFgeloFQibiw6eRrpf2fevj/AiInvszSC0Aj Q/JwakoypIW3YiNS87lEeTkmHA1UJkLXlKF3pu+VXAk/SnkDi2qAKsKJxACuz7xdBHUT MuX8iYPxJXJLOhjhhIXdxhyWmmXx6aDlBvT8c4x0svAzolCjSknfTUBcVHCfk77PIajQ 5e0Q== X-Forwarded-Encrypted: i=1; AJvYcCUStJSHuA5X2Bv2XNzWerYonD2KAFIb5N4Pp6jvPCkGaQln/wnBCsx/DxMgT4uNsEhxzlxRyKFkHfOthxGF+upbYLWlvWQnx0arcU+N X-Gm-Message-State: AOJu0YyM7xOqSrI6mGJVgD/3dgWWfC09dJChUree/+3IQLcXDv5i/+t5 AOUvCzDk7K8HHwfxg1Ah5qjLuhxOjgAQzvWrbOa7A62HH/NLW6ar6W1ooub1Xw== X-Received: by 2002:a17:90a:ba8b:b0:29b:961a:29c3 with SMTP id t11-20020a17090aba8b00b0029b961a29c3mr4336581pjr.49.1711530849946; Wed, 27 Mar 2024 02:14:09 -0700 (PDT) Received: from [127.0.1.1] ([120.60.52.77]) by smtp.gmail.com with ESMTPSA id h190-20020a6383c7000000b005dc4fc80b21sm8673871pge.70.2024.03.27.02.13.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Mar 2024 02:14:09 -0700 (PDT) From: Manivannan Sadhasivam Date: Wed, 27 Mar 2024 14:43:30 +0530 Subject: [PATCH v12 1/8] PCI: dwc: ep: Fix DBI access failure for drivers requiring refclk from host Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20240327-pci-dbi-rework-v12-1-082625472414@linaro.org> References: <20240327-pci-dbi-rework-v12-0-082625472414@linaro.org> In-Reply-To: <20240327-pci-dbi-rework-v12-0-082625472414@linaro.org> To: Jingoo Han , Lorenzo Pieralisi , =?utf-8?q?Krzysztof_Wilczy=C5=84ski?= , Rob Herring , Bjorn Helgaas , Marek Vasut , Yoshihiro Shimoda , Thierry Reding , Jonathan Hunter , Vidya Sagar , Vignesh Raghavendra , Richard Zhu , Lucas Stach , Shawn Guo , Sascha Hauer , Pengutronix Kernel Team , Fabio Estevam , NXP Linux Team , Minghuan Lian , Mingkai Hu , Roy Zang , Kunihiko Hayashi , Masami Hiramatsu , Kishon Vijay Abraham I , Jesper Nilsson , Srikanth Thokala , Shawn Lin , Heiko Stuebner , Kishon Vijay Abraham I Cc: linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, linux-renesas-soc@vger.kernel.org, linux-arm-msm@vger.kernel.org, linux-tegra@vger.kernel.org, linux-omap@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Niklas Cassel , linux-arm-kernel@axis.com, linux-rockchip@lists.infradead.org, Manivannan Sadhasivam , Frank Li X-Mailer: b4 0.12.4 X-Developer-Signature: v=1; a=openpgp-sha256; l=7480; i=manivannan.sadhasivam@linaro.org; h=from:subject:message-id; bh=C9itwwrFPiwdNm8Zd8w7cqdwK6rVpgcx54kNU3MQW98=; b=owEBbQGS/pANAwAKAVWfEeb+kc71AcsmYgBmA+NIJnfXD7LMZio8W0A249GNJA5H9smQ0/YIV 2Mbshx2fmCJATMEAAEKAB0WIQRnpUMqgUjL2KRYJ5dVnxHm/pHO9QUCZgPjSAAKCRBVnxHm/pHO 9SFSCACcl92q4s6I4nZd3WCcfhQumw0iLwZ/wn0ylz4u11oziIW8gOj04TPmEFpvKK4yUo2H2gV emRIqlv3a2MowhrvoJ/FySPsqbtDFadni6vd5WA+H1hK0Cc5vAmsMuverqXHYM37THcdHfKjGVf dnUP8okf0Y594v6qqTsidArO8n/hzakaqI++awJiG9/LH8RFIR5CvkdtH0gnsksv9fhZJsmXJLv HHU5iUqaNDOnOKsBxS/m+1G5GP12FiFevRHj3ySuW/KGbiQ7rAc7qe76wzQXksTMbMWHBG7llAa aZWRSGu6qekGHKePMBoUsGEAOFcBvwCqKD8NKC0q7hQEWBg2 X-Developer-Key: i=manivannan.sadhasivam@linaro.org; a=openpgp; fpr=C668AEC3C3188E4C611465E7488550E901166008 The DWC glue drivers requiring an active reference clock from the PCIe host for initializing their PCIe EP core, set a flag called 'core_init_notifier' to let DWC driver know that these drivers need a special attention during initialization. In these drivers, access to the hw registers (like DBI) before receiving the active refclk from host will result in access failure and also could cause a whole system hang. But the current DWC EP driver doesn't honor the requirements of the drivers setting 'core_init_notifier' flag and tries to access the DBI registers during dw_pcie_ep_init(). This causes the system hang for glue drivers such as Tegra194 and Qcom EP as they depend on refclk from host and have set the above mentioned flag. To workaround this issue, users of the affected platforms have to maintain the dependency with the PCIe host by booting the PCIe EP after host boot. But this won't provide a good user experience, since PCIe EP is _one_ of the features of those platforms and it doesn't make sense to delay the whole platform booting due to PCIe requiring active refclk. So to fix this issue, let's move all the DBI access from dw_pcie_ep_init() in the DWC EP driver to the dw_pcie_ep_init_complete() API. This API will only be called by the drivers setting 'core_init_notifier' flag once refclk is received from host. For the rest of the drivers that gets the refclk locally, this API will be called within dw_pcie_ep_init(). Fixes: e966f7390da9 ("PCI: dwc: Refactor core initialization code for EP mode") Co-developed-by: Vidya Sagar Signed-off-by: Vidya Sagar Reviewed-by: Frank Li Reviewed-by: Niklas Cassel Signed-off-by: Manivannan Sadhasivam --- drivers/pci/controller/dwc/pcie-designware-ep.c | 120 ++++++++++++++---------- 1 file changed, 71 insertions(+), 49 deletions(-) diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c index 746a11dcb67f..c43a1479de2c 100644 --- a/drivers/pci/controller/dwc/pcie-designware-ep.c +++ b/drivers/pci/controller/dwc/pcie-designware-ep.c @@ -604,11 +604,16 @@ static unsigned int dw_pcie_ep_find_ext_capability(struct dw_pcie *pci, int cap) int dw_pcie_ep_init_complete(struct dw_pcie_ep *ep) { struct dw_pcie *pci = to_dw_pcie_from_ep(ep); + struct dw_pcie_ep_func *ep_func; + struct device *dev = pci->dev; + struct pci_epc *epc = ep->epc; unsigned int offset, ptm_cap_base; unsigned int nbars; u8 hdr_type; + u8 func_no; + int i, ret; + void *addr; u32 reg; - int i; hdr_type = dw_pcie_readb_dbi(pci, PCI_HEADER_TYPE) & PCI_HEADER_TYPE_MASK; @@ -619,6 +624,58 @@ int dw_pcie_ep_init_complete(struct dw_pcie_ep *ep) return -EIO; } + dw_pcie_version_detect(pci); + + dw_pcie_iatu_detect(pci); + + ret = dw_pcie_edma_detect(pci); + if (ret) + return ret; + + if (!ep->ib_window_map) { + ep->ib_window_map = devm_bitmap_zalloc(dev, pci->num_ib_windows, + GFP_KERNEL); + if (!ep->ib_window_map) + goto err_remove_edma; + } + + if (!ep->ob_window_map) { + ep->ob_window_map = devm_bitmap_zalloc(dev, pci->num_ob_windows, + GFP_KERNEL); + if (!ep->ob_window_map) + goto err_remove_edma; + } + + if (!ep->outbound_addr) { + addr = devm_kcalloc(dev, pci->num_ob_windows, sizeof(phys_addr_t), + GFP_KERNEL); + if (!addr) + goto err_remove_edma; + ep->outbound_addr = addr; + } + + for (func_no = 0; func_no < epc->max_functions; func_no++) { + + ep_func = dw_pcie_ep_get_func_from_ep(ep, func_no); + if (ep_func) + continue; + + ep_func = devm_kzalloc(dev, sizeof(*ep_func), GFP_KERNEL); + if (!ep_func) + goto err_remove_edma; + + ep_func->func_no = func_no; + ep_func->msi_cap = dw_pcie_ep_find_capability(ep, func_no, + PCI_CAP_ID_MSI); + ep_func->msix_cap = dw_pcie_ep_find_capability(ep, func_no, + PCI_CAP_ID_MSIX); + + list_add_tail(&ep_func->list, &ep->func_list); + } + + if (ep->ops->init) + ep->ops->init(ep); + offset = dw_pcie_ep_find_ext_capability(pci, PCI_EXT_CAP_ID_REBAR); ptm_cap_base = dw_pcie_ep_find_ext_capability(pci, PCI_EXT_CAP_ID_PTM); @@ -658,14 +715,17 @@ int dw_pcie_ep_init_complete(struct dw_pcie_ep *ep) dw_pcie_dbi_ro_wr_dis(pci); return 0; + +err_remove_edma: + dw_pcie_edma_remove(pci); + + return ret; } EXPORT_SYMBOL_GPL(dw_pcie_ep_init_complete); int dw_pcie_ep_init(struct dw_pcie_ep *ep) { int ret; - void *addr; - u8 func_no; struct resource *res; struct pci_epc *epc; struct dw_pcie *pci = to_dw_pcie_from_ep(ep); @@ -673,7 +733,6 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep) struct platform_device *pdev = to_platform_device(dev); struct device_node *np = dev->of_node; const struct pci_epc_features *epc_features; - struct dw_pcie_ep_func *ep_func; INIT_LIST_HEAD(&ep->func_list); @@ -691,26 +750,6 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep) if (ep->ops->pre_init) ep->ops->pre_init(ep); - dw_pcie_version_detect(pci); - - dw_pcie_iatu_detect(pci); - - ep->ib_window_map = devm_bitmap_zalloc(dev, pci->num_ib_windows, - GFP_KERNEL); - if (!ep->ib_window_map) - return -ENOMEM; - - ep->ob_window_map = devm_bitmap_zalloc(dev, pci->num_ob_windows, - GFP_KERNEL); - if (!ep->ob_window_map) - return -ENOMEM; - - addr = devm_kcalloc(dev, pci->num_ob_windows, sizeof(phys_addr_t), - GFP_KERNEL); - if (!addr) - return -ENOMEM; - ep->outbound_addr = addr; - epc = devm_pci_epc_create(dev, &epc_ops); if (IS_ERR(epc)) { dev_err(dev, "Failed to create epc device\n"); @@ -724,23 +763,6 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep) if (ret < 0) epc->max_functions = 1; - for (func_no = 0; func_no < epc->max_functions; func_no++) { - ep_func = devm_kzalloc(dev, sizeof(*ep_func), GFP_KERNEL); - if (!ep_func) - return -ENOMEM; - - ep_func->func_no = func_no; - ep_func->msi_cap = dw_pcie_ep_find_capability(ep, func_no, - PCI_CAP_ID_MSI); - ep_func->msix_cap = dw_pcie_ep_find_capability(ep, func_no, - PCI_CAP_ID_MSIX); - - list_add_tail(&ep_func->list, &ep->func_list); - } - - if (ep->ops->init) - ep->ops->init(ep); - ret = pci_epc_mem_init(epc, ep->phys_base, ep->addr_size, ep->page_size); if (ret < 0) { @@ -756,25 +778,25 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep) goto err_exit_epc_mem; } - ret = dw_pcie_edma_detect(pci); - if (ret) - goto err_free_epc_mem; - if (ep->ops->get_features) { epc_features = ep->ops->get_features(ep); if (epc_features->core_init_notifier) return 0; } + /* + * NOTE:- Avoid accessing the hardware (Ex:- DBI space) before this + * step as platforms that implement 'core_init_notifier' feature may + * not have the hardware ready (i.e. core initialized) for access + * (Ex: tegra194). Any hardware access on such platforms result + * in system hang. + */ ret = dw_pcie_ep_init_complete(ep); if (ret) - goto err_remove_edma; + goto err_free_epc_mem; return 0; -err_remove_edma: - dw_pcie_edma_remove(pci); - err_free_epc_mem: pci_epc_mem_free_addr(epc, ep->msi_mem_phys, ep->msi_mem, epc->mem->window.page_size); -- 2.25.1