Received: by 2002:ab2:3141:0:b0:1ed:23cc:44d1 with SMTP id i1csp1722528lqg; Mon, 4 Mar 2024 01:29:34 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVKJm4LgggHRHQt6KBox4Mvu5a3cV58CTSIkwYxePKCnCQNZ8ZoFt1LfEKGxTN7NAP8xGLDiu6BTsToit/sBU1zX4AIBpb+FT8Jne3M8w== X-Google-Smtp-Source: AGHT+IEt+KPXf2gYXHHW02tR5j3knHzDSIrdu8v4ODgvcfBRBCcWaanh4urtPf868pVuHD9AxB/v X-Received: by 2002:a05:6a00:2d28:b0:6e5:591e:308e with SMTP id fa40-20020a056a002d2800b006e5591e308emr9512917pfb.28.1709544574251; Mon, 04 Mar 2024 01:29:34 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1709544574; cv=pass; d=google.com; s=arc-20160816; b=kTEKY/pFuRUkE+euwsDktLEzh9yKpCQdDy2dQjQQfmLNanffW6HiAGi6xSqJ1VAxbd VoKT0ziDQEXbD0626jV+s0V8q1Nh7SuEGeJasyxg4uHQtoZS/PJWlOxs5VhKVwLbCzic atmz48cnP3BdtkDaoYX5WXFv4w+J9QZQtckZ4Ci6m5MPtY4WKcS2Q9AwKAkSL/gi9dSX 3TyhAYJWrNEDS8DdcIBlQ0LonvDfywwJZVmUGO8PjgOmGgq2d8V0hfaMFcL+KQ6oV7Di EbediJZzTnPQv9be77s9NbgDwI40uWaIw/il+QMDDRQO/66wEMtyuVhil9cAeGxz5Hrp /uOg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:list-unsubscribe:list-subscribe:list-id:precedence :subject:date:from:dkim-signature; bh=5fT/y0AtsQLIjDs/rZqEeULAe7sJBBbmOb/17oQI9IE=; fh=A3vtGSBOwDchHmTnPKa9DJn55bmTkgHZGh52FeyE4L0=; b=YvfpRGqijGOBndnx/CXjZrmoySRd5sK6DawXxrwTcQZ0RGT0H4Jp+/9wNo7/gnlpwU kxhNWKgnb+Gx1mi/dv/kkDat02AHlNebhybzUdvAHphFXva8fZ34fkNWCyq5gIY1z1KQ Qo52PN+03IvLOZjw5mh2Xo0RB5Rhd6ylHDh/Ugwloqsf/gLrRak9uywLY/5iBZhRrVpk D3NhM3fLARllitFYAyNpj0aJS8QZxcR5PhuL+V6KpXA5eKKN+qpqB4L0c3QhZydlIinb EhSZNf2jAvr8xgzj0i61uSEypnW+KFAUQQozzPkOfeR5h9kysWftqUR/nVtfNydXnsKJ VLIw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=eB6+IhW9; arc=pass (i=1 spf=pass spfdomain=linaro.org dkim=pass dkdomain=linaro.org dmarc=pass fromdomain=linaro.org); spf=pass (google.com: domain of linux-kernel+bounces-90307-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-90307-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id y9-20020a631809000000b005dc48449b0bsi8143403pgl.690.2024.03.04.01.29.33 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Mar 2024 01:29:34 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-90307-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=eB6+IhW9; arc=pass (i=1 spf=pass spfdomain=linaro.org dkim=pass dkdomain=linaro.org dmarc=pass fromdomain=linaro.org); spf=pass (google.com: domain of linux-kernel+bounces-90307-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-90307-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id C9DB5B23016 for ; Mon, 4 Mar 2024 09:24:28 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 5FAE122097; Mon, 4 Mar 2024 09:23:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="eB6+IhW9" Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 61D44241E0 for ; Mon, 4 Mar 2024 09:23:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709544210; cv=none; b=C5K0ShUc3yof1ylwsQ+IiHOscvsUQR/pInz95h+yfXEoCuipPDtBhBya6c23qwakwbjIsMhGUgPUtBBxDf1wFFtfkdcHdAVMqb7r/b69DCfzNEVwPBvDgkE5//J7p7y52DkwwYtyaITLc6Ud3VvjTmBzQsUXhvdJT6xxIz5fUn4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709544210; c=relaxed/simple; bh=twJXmkPkyrcH3emb1jz1IubDpI6MxLeiqeCjK6e/8uk=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=YPY5flHB+Hg/jpsN0tQBuVhUqn+Jzyx/FsmU37wtokfl/h6GIX87hcBISipEcGfnFb+oLZ3mUzux5kQw70kmZ318EIpES2Nc7lg3dn17PBLYTmFykEVTRijqRW6gASe1VbC+N5tbweo3G/TOoM+cUwVZUxJey4BuXttj+9cB098= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org; spf=pass smtp.mailfrom=linaro.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b=eB6+IhW9; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linaro.org Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-1dbd32cff0bso35014835ad.0 for ; Mon, 04 Mar 2024 01:23:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1709544207; x=1710149007; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=5fT/y0AtsQLIjDs/rZqEeULAe7sJBBbmOb/17oQI9IE=; b=eB6+IhW9nOJsOy5TRg1/ijv9oOITHHRezirYF8sdo9uNI742FgYyhIxmqtZ5Uynx3S CrTc7du0ZlgSkJ7VaqpIbni3DXhxph3Vi78vRRR55XKavRNUoENAoUk2cOyKaYDzUbTK eUxZN5D6djJQphqm9n9bXNAXEO6LmM8nUjTceL8mgBxDwJD3VdnGEG4wrjPKausB2GW4 lXDGhPj17Y4WA6H1+zcV6JucAafTfuUGo4L8ZrpBsQZLFfq+C14Knn2LFvdX9jAv1pjI h6RBi50nLLzvlg6vp7rVhuiQFEapeHnmnZhi3srUz8EpFGNUhjNrB7XTsCpOltCujKKs hdQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709544207; x=1710149007; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5fT/y0AtsQLIjDs/rZqEeULAe7sJBBbmOb/17oQI9IE=; b=e8T9ZA5C7RrcYNXdHT2CMQIR/vzCKy6cuwlagqT0dJDLeyqylRwWDvZfFyAlnLBz/Q aPPHaw39ZZhdErnBFux9AmmQfBAHmQPjJKl21X7CW1nXDd/lgfiDQTTP9twpNspwLbPs 3YxjCIXNDlKH0fsfPpm7O3PO5/fC4MvTvjJlLIgf3R8cc2HQpAtB98heE9INnEnRzD23 eB28fTZIPvT8DdhjRYclM4YSFIi9E9GscwIuMg4605HWV/w3w3Qb4A4qqqO3CqtfqeBR 3U/aTdDIYeLC8tu0pTxNKiLfthZUSpZmdw3M+MyvXe+H6UoV1/VA+f3XXjwoJRaPEwIE LluA== X-Forwarded-Encrypted: i=1; AJvYcCWOiT9VmOxO2Xu0KILsRZSIGHc6prEOEIF7tcELxCPwp5kNzETivC3wX8YLC4YjJ0ZL+hlaiy82kYwhG4/jJ/Ygjh7/yWCUz669R7+5 X-Gm-Message-State: AOJu0YyKakdSIz+70Bw1kIOR3H0jxcC1wTyqXaxVT6HCUDPwi8GrQA/E o8oQ5Sz2gtLBNkqpFutY9Ae2EoG8gg9HyfqJRpS9wAOB4RWHZu4ITzH28yCrcA== X-Received: by 2002:a17:902:7283:b0:1dc:4bc2:4923 with SMTP id d3-20020a170902728300b001dc4bc24923mr6497738pll.65.1709544207590; Mon, 04 Mar 2024 01:23:27 -0800 (PST) Received: from [127.0.1.1] ([117.202.187.165]) by smtp.gmail.com with ESMTPSA id c5-20020a170902c1c500b001dbb06b6138sm7996648plc.252.2024.03.04.01.23.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Mar 2024 01:23:27 -0800 (PST) From: Manivannan Sadhasivam Date: Mon, 04 Mar 2024 14:52:16 +0530 Subject: [PATCH v9 04/10] PCI: dwc: ep: Fix DBI access failure for drivers requiring refclk from host Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20240304-pci-dbi-rework-v9-4-29d433d99cda@linaro.org> References: <20240304-pci-dbi-rework-v9-0-29d433d99cda@linaro.org> In-Reply-To: <20240304-pci-dbi-rework-v9-0-29d433d99cda@linaro.org> To: Jingoo Han , Gustavo Pimentel , Lorenzo Pieralisi , =?utf-8?q?Krzysztof_Wilczy=C5=84ski?= , Rob Herring , Bjorn Helgaas , Marek Vasut , Yoshihiro Shimoda , Thierry Reding , Jonathan Hunter , Kishon Vijay Abraham I , Vidya Sagar , Vignesh Raghavendra , Richard Zhu , Lucas Stach , Shawn Guo , Sascha Hauer , Pengutronix Kernel Team , Fabio Estevam , NXP Linux Team , Minghuan Lian , Mingkai Hu , Roy Zang , Kunihiko Hayashi , Masami Hiramatsu , Kishon Vijay Abraham I , Jesper Nilsson , Srikanth Thokala Cc: linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, linux-renesas-soc@vger.kernel.org, linux-arm-msm@vger.kernel.org, linux-tegra@vger.kernel.org, linux-omap@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Niklas Cassel , linux-arm-kernel@axis.com, Manivannan Sadhasivam , Frank Li X-Mailer: b4 0.12.4 X-Developer-Signature: v=1; a=openpgp-sha256; l=7432; i=manivannan.sadhasivam@linaro.org; h=from:subject:message-id; bh=twJXmkPkyrcH3emb1jz1IubDpI6MxLeiqeCjK6e/8uk=; b=owEBbQGS/pANAwAKAVWfEeb+kc71AcsmYgBl5ZLXQs+vyYRsCgCYeHMQhAizXHwVS0MTuu3zU s6j/vnxYteJATMEAAEKAB0WIQRnpUMqgUjL2KRYJ5dVnxHm/pHO9QUCZeWS1wAKCRBVnxHm/pHO 9TbkB/4mDMy2D9rIvZZoeNU74EDlQen5NR9hkWfz1/dRRY4YKuQscPObWH2YGbVYs25WH/nmcV5 DMHLRB1TPAWYfowXO07gllCjpCEHb0U3klTJBU87uUmYeC19uxUWrPVQzg3TUCI+MotgahAPXHs Vsb6xjY67CV+dSmqwWPgpmKp3gAj34JXVcibYOvcArp5VJyjFcZeWTEvHU717tlbRi3qFIfWA+C bRCGOildneJtxA0CGSwKx5piCtRGGdS1qvJ/U9b4ESOmTn7NjPMFY0Adg+/vVUSHsoFP7K8buxf JvEo81WiEh1KXsXJqiB2FdaGShOSBVTBRXvlMbbx0hLP3hA0 X-Developer-Key: i=manivannan.sadhasivam@linaro.org; a=openpgp; fpr=C668AEC3C3188E4C611465E7488550E901166008 The DWC glue drivers requiring an active reference clock from the PCIe host for initializing their PCIe EP core, set a flag called 'core_init_notifier' to let DWC driver know that these drivers need a special attention during initialization. In these drivers, access to the hw registers (like DBI) before receiving the active refclk from host will result in access failure and also could cause a whole system hang. But the current DWC EP driver doesn't honor the requirements of the drivers setting 'core_init_notifier' flag and tries to access the DBI registers during dw_pcie_ep_init(). This causes the system hang for glue drivers such as Tegra194 and Qcom EP as they depend on refclk from host and have set the above mentioned flag. To workaround this issue, users of the affected platforms have to maintain the dependency with the PCIe host by booting the PCIe EP after host boot. But this won't provide a good user experience, since PCIe EP is _one_ of the features of those platforms and it doesn't make sense to delay the whole platform booting due to PCIe requiring active refclk. So to fix this issue, let's move all the DBI access from dw_pcie_ep_init() in the DWC EP driver to the dw_pcie_ep_init_complete() API. This API will only be called by the drivers setting 'core_init_notifier' flag once refclk is received from host. For the rest of the drivers that gets the refclk locally, this API will be called within dw_pcie_ep_init(). Fixes: e966f7390da9 ("PCI: dwc: Refactor core initialization code for EP mode") Co-developed-by: Vidya Sagar Signed-off-by: Vidya Sagar Reviewed-by: Frank Li Signed-off-by: Manivannan Sadhasivam --- drivers/pci/controller/dwc/pcie-designware-ep.c | 120 ++++++++++++++---------- 1 file changed, 71 insertions(+), 49 deletions(-) diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c index 1205bfba8310..99d66b0fa59b 100644 --- a/drivers/pci/controller/dwc/pcie-designware-ep.c +++ b/drivers/pci/controller/dwc/pcie-designware-ep.c @@ -606,11 +606,16 @@ static unsigned int dw_pcie_ep_find_ext_capability(struct dw_pcie *pci, int cap) int dw_pcie_ep_init_complete(struct dw_pcie_ep *ep) { struct dw_pcie *pci = to_dw_pcie_from_ep(ep); + struct dw_pcie_ep_func *ep_func; + struct device *dev = pci->dev; + struct pci_epc *epc = ep->epc; unsigned int offset, ptm_cap_base; unsigned int nbars; u8 hdr_type; + u8 func_no; + int i, ret; + void *addr; u32 reg; - int i; hdr_type = dw_pcie_readb_dbi(pci, PCI_HEADER_TYPE) & PCI_HEADER_TYPE_MASK; @@ -621,6 +626,58 @@ int dw_pcie_ep_init_complete(struct dw_pcie_ep *ep) return -EIO; } + dw_pcie_version_detect(pci); + + dw_pcie_iatu_detect(pci); + + ret = dw_pcie_edma_detect(pci); + if (ret) + return ret; + + if (!ep->ib_window_map) { + ep->ib_window_map = devm_bitmap_zalloc(dev, pci->num_ib_windows, + GFP_KERNEL); + if (!ep->ib_window_map) + goto err_remove_edma; + } + + if (!ep->ob_window_map) { + ep->ob_window_map = devm_bitmap_zalloc(dev, pci->num_ob_windows, + GFP_KERNEL); + if (!ep->ob_window_map) + goto err_remove_edma; + } + + if (!ep->outbound_addr) { + addr = devm_kcalloc(dev, pci->num_ob_windows, sizeof(phys_addr_t), + GFP_KERNEL); + if (!addr) + goto err_remove_edma; + ep->outbound_addr = addr; + } + + for (func_no = 0; func_no < epc->max_functions; func_no++) { + + ep_func = dw_pcie_ep_get_func_from_ep(ep, func_no); + if (ep_func) + continue; + + ep_func = devm_kzalloc(dev, sizeof(*ep_func), GFP_KERNEL); + if (!ep_func) + goto err_remove_edma; + + ep_func->func_no = func_no; + ep_func->msi_cap = dw_pcie_ep_find_capability(ep, func_no, + PCI_CAP_ID_MSI); + ep_func->msix_cap = dw_pcie_ep_find_capability(ep, func_no, + PCI_CAP_ID_MSIX); + + list_add_tail(&ep_func->list, &ep->func_list); + } + + if (ep->ops->init) + ep->ops->init(ep); + offset = dw_pcie_ep_find_ext_capability(pci, PCI_EXT_CAP_ID_REBAR); ptm_cap_base = dw_pcie_ep_find_ext_capability(pci, PCI_EXT_CAP_ID_PTM); @@ -655,14 +712,17 @@ int dw_pcie_ep_init_complete(struct dw_pcie_ep *ep) dw_pcie_dbi_ro_wr_dis(pci); return 0; + +err_remove_edma: + dw_pcie_edma_remove(pci); + + return ret; } EXPORT_SYMBOL_GPL(dw_pcie_ep_init_complete); int dw_pcie_ep_init(struct dw_pcie_ep *ep) { int ret; - void *addr; - u8 func_no; struct resource *res; struct pci_epc *epc; struct dw_pcie *pci = to_dw_pcie_from_ep(ep); @@ -670,7 +730,6 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep) struct platform_device *pdev = to_platform_device(dev); struct device_node *np = dev->of_node; const struct pci_epc_features *epc_features; - struct dw_pcie_ep_func *ep_func; INIT_LIST_HEAD(&ep->func_list); @@ -688,26 +747,6 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep) if (ep->ops->pre_init) ep->ops->pre_init(ep); - dw_pcie_version_detect(pci); - - dw_pcie_iatu_detect(pci); - - ep->ib_window_map = devm_bitmap_zalloc(dev, pci->num_ib_windows, - GFP_KERNEL); - if (!ep->ib_window_map) - return -ENOMEM; - - ep->ob_window_map = devm_bitmap_zalloc(dev, pci->num_ob_windows, - GFP_KERNEL); - if (!ep->ob_window_map) - return -ENOMEM; - - addr = devm_kcalloc(dev, pci->num_ob_windows, sizeof(phys_addr_t), - GFP_KERNEL); - if (!addr) - return -ENOMEM; - ep->outbound_addr = addr; - epc = devm_pci_epc_create(dev, &epc_ops); if (IS_ERR(epc)) { dev_err(dev, "Failed to create epc device\n"); @@ -721,23 +760,6 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep) if (ret < 0) epc->max_functions = 1; - for (func_no = 0; func_no < epc->max_functions; func_no++) { - ep_func = devm_kzalloc(dev, sizeof(*ep_func), GFP_KERNEL); - if (!ep_func) - return -ENOMEM; - - ep_func->func_no = func_no; - ep_func->msi_cap = dw_pcie_ep_find_capability(ep, func_no, - PCI_CAP_ID_MSI); - ep_func->msix_cap = dw_pcie_ep_find_capability(ep, func_no, - PCI_CAP_ID_MSIX); - - list_add_tail(&ep_func->list, &ep->func_list); - } - - if (ep->ops->init) - ep->ops->init(ep); - ret = pci_epc_mem_init(epc, ep->phys_base, ep->addr_size, ep->page_size); if (ret < 0) { @@ -753,25 +775,25 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep) goto err_exit_epc_mem; } - ret = dw_pcie_edma_detect(pci); - if (ret) - goto err_free_epc_mem; - if (ep->ops->get_features) { epc_features = ep->ops->get_features(ep); if (epc_features->core_init_notifier) return 0; } + /* + * NOTE:- Avoid accessing the hardware (Ex:- DBI space) before this + * step as platforms that implement 'core_init_notifier' feature may + * not have the hardware ready (i.e. core initialized) for access + * (Ex: tegra194). Any hardware access on such platforms result + * in system hang. + */ ret = dw_pcie_ep_init_complete(ep); if (ret) - goto err_remove_edma; + goto err_free_epc_mem; return 0; -err_remove_edma: - dw_pcie_edma_remove(pci); - err_free_epc_mem: pci_epc_mem_free_addr(epc, ep->msi_mem_phys, ep->msi_mem, epc->mem->window.page_size); -- 2.25.1