Received: by 2002:ab2:1149:0:b0:1f3:1f8c:d0c6 with SMTP id z9csp217442lqz; Fri, 29 Mar 2024 14:08:07 -0700 (PDT) X-Forwarded-Encrypted: i=3; AJvYcCUbATo7NLaYAGI+4W8d4dRuUhf8812EQ6LGqd9D2fM45VLJ7ZgY/4drtGgXGhKXoq3fSdXzfurLGrdSamXZBTs/sSpOkRJVFW6mk/zR0g== X-Google-Smtp-Source: AGHT+IEqefhMhn5QWkSBS+/R0TFvRHr7Eu7mPS0IZcNnZFUgIBD1YKfSqsMOcdMG7kQNFPydF0py X-Received: by 2002:a05:6402:13c3:b0:566:2f24:b063 with SMTP id a3-20020a05640213c300b005662f24b063mr2672546edx.23.1711746487000; Fri, 29 Mar 2024 14:08:07 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1711746486; cv=pass; d=google.com; s=arc-20160816; b=P+udLia5vqRe7FUVlhbqNXPbdSsCRa8MjupVWBjH9QLonzfVoMJsBbzLFgNbUaB9ay c+3K5Zu1ysM9iSPGskG3NgoI2TZkfsQmNnIR+LAqBUCODqMCDu9qEAfFbR6r1g6qbFeZ tXUgWg4nUe4yxK3rnZb/QLIrO/WwnAbuhxRLPf4n4Z2mvEWivCkeoLzJZAGpvmJdTJep vXmxSl6Lpay49JNixSjpDxhPNROmXDoaMAmti2FsZ1q3qtnXSyvEVHrQy0wnWa3KiRnG Rzz7IJ7r3mE9WwPMSlLWZ0DMVIE2tqpSTILwj7rTmsCaadsbboMst38nBqVvfLmNi3nq YIcQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=GgbUCbOMHRnVhyOJ7okeqYGH3I8nzYQhPX7ZVNZDAOQ=; fh=gTQnRIxpB4/GKsMy5s7HvzYf0xa4wk69uZmT/C+IYXQ=; b=j2dYYz5HAYpFcbaGs/sQgGEq5ddE5DQfKF3B3nxJ4hH54MZnydW5WI9Q8/UPt3nBv9 SoSE5ZI2wSMT52C+ogl+YpKcpkHO8Dk8XwgEvN6EntMqqpElGOwJkY5s1EiRxfkz2xlC zbqYpcA6YQ5+11W3lP/KKtsqlbzJHSCPxbJfeEyyQnBGi1d/np4kfnjrHxmCrsH4F6Lr Y/6Ly94/Jl0aw8v95v3+znzt5LSib4ky7zVGZMXArUIkQrKiAfagYcQ4if/ZIS8bvF2X 6LXMrLtvS5XsvAijLTLWM6hNpioxdh33QHdBUQbGQSUQUFbkCwJlFL1kCY1c+pi2Yn6b ycsg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@quicinc.com header.s=qcppdkim1 header.b=i+vB3WIF; arc=pass (i=1 spf=pass spfdomain=quicinc.com dkim=pass dkdomain=quicinc.com dmarc=pass fromdomain=quicinc.com); spf=pass (google.com: domain of linux-kernel+bounces-125432-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-125432-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=quicinc.com Return-Path: Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id f20-20020a0564021e9400b0056c3771c31csi2261760edf.131.2024.03.29.14.08.06 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Mar 2024 14:08:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel+bounces-125432-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@quicinc.com header.s=qcppdkim1 header.b=i+vB3WIF; arc=pass (i=1 spf=pass spfdomain=quicinc.com dkim=pass dkdomain=quicinc.com dmarc=pass fromdomain=quicinc.com); spf=pass (google.com: domain of linux-kernel+bounces-125432-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-125432-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=quicinc.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 8E86D1F2342D for ; Fri, 29 Mar 2024 21:08:06 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 0415313D518; Fri, 29 Mar 2024 21:07:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=quicinc.com header.i=@quicinc.com header.b="i+vB3WIF" Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 351F613AA38; Fri, 29 Mar 2024 21:07:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=205.220.180.131 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711746443; cv=none; b=AwyekX09Ibt+fwM4x6UPFciXpUwWsDfP5iCYvxvdCyW5nw37WUviuEhHa7zIewqRoFVk8NdH+H0FXeicc77Nlp7ukAqGzuE83hhSgIcoOCFhg9JotACw9soKIFyEY5UOQQD3C6hclVVwm+h1vfi1RNc0hXk5ZQa3KaBt/1RLXaM= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711746443; c=relaxed/simple; bh=WqzlNtdzGny6DAImT5I05x8Ye222sB7lrR73IdbxyJg=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=V/YSfmWufCdK6lxBqH4+FZoT5xZlLHiECRK5OMcHZDA/h5IPzt+UomI7HzCkJbldz765b8kCaq4grXWnC6ERhNtiSfkfHJYOtRbETDMWXaAtfaWQ0fq3bzYBoqrzlnbuPkwbm8wbYNOnyZHUSgYVM9k/8UI0kLHQkPH/dx2pdtQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=quicinc.com; spf=pass smtp.mailfrom=quicinc.com; dkim=pass (2048-bit key) header.d=quicinc.com header.i=@quicinc.com header.b=i+vB3WIF; arc=none smtp.client-ip=205.220.180.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=quicinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=quicinc.com Received: from pps.filterd (m0279871.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.24/8.17.1.24) with ESMTP id 42TJkc29032588; Fri, 29 Mar 2024 21:07:09 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding:content-type; s= qcppdkim1; bh=GgbUCbOMHRnVhyOJ7okeqYGH3I8nzYQhPX7ZVNZDAOQ=; b=i+ vB3WIF7Qm6Q02kRm5QvPcgV3pRAvzBQOT4LiHG0I2uZoQ4lzkzeUj1+Db945kzfU r4NYotteNq9gdAZdlBhaJE/dWchkeAbbv5VD+cyMs1ufVcQEHOoi4Ag7b6OT1IyA RwNdgkQUH2yyzkm7uQTMmr420cpv0GLEa8cb59z5UBYe7qlyAt/EWGESF+2rNn06 AowbVS8LYOlNtFDVbZDXICpfKZYP1okpokvjQAskuf2qxnpZniH4FpUdJYPqrlvo REzMFWX47eAsiKetawb5B0ipi5m1bjN+f7EPXjw+GQ0SFwy2PNrU3/MlxgzBUFYV 3CtA+uca7lv7MZDNVt1A== Received: from nasanppmta02.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3x5ybmrux8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 29 Mar 2024 21:07:09 +0000 (GMT) Received: from nasanex01a.na.qualcomm.com (nasanex01a.na.qualcomm.com [10.52.223.231]) by NASANPPMTA02.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 42TL78bL026925 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 29 Mar 2024 21:07:08 GMT Received: from hu-c-gdjako-lv.qualcomm.com (10.49.16.6) by nasanex01a.na.qualcomm.com (10.52.223.231) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1118.40; Fri, 29 Mar 2024 14:07:07 -0700 From: Georgi Djakov To: , , , CC: , , , , , , , , , , , , , , Subject: [PATCH v7 4/7] iommu/arm-smmu-qcom: Use a custom context fault handler for sdm845 Date: Fri, 29 Mar 2024 14:06:35 -0700 Message-ID: <20240329210638.3647523-5-quic_c_gdjako@quicinc.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240329210638.3647523-1-quic_c_gdjako@quicinc.com> References: <20240329210638.3647523-1-quic_c_gdjako@quicinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: nalasex01c.na.qualcomm.com (10.47.97.35) To nasanex01a.na.qualcomm.com (10.52.223.231) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-ORIG-GUID: dVyLsjI418FMsnhjGU9XDT_maAVkXv2o X-Proofpoint-GUID: dVyLsjI418FMsnhjGU9XDT_maAVkXv2o X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-03-29_13,2024-03-28_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 impostorscore=0 lowpriorityscore=0 malwarescore=0 bulkscore=0 phishscore=0 mlxlogscore=999 suspectscore=0 adultscore=0 priorityscore=1501 mlxscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2403210001 definitions=main-2403290188 The sdm845 platform now supports TBUs, so let's get additional debug info from the TBUs when a context fault occurs. Implement a custom context fault handler that does both software + hardware page table walks and TLB Invalidate All. Signed-off-by: Georgi Djakov --- .../iommu/arm/arm-smmu/arm-smmu-qcom-tbu.c | 143 ++++++++++++++++++ drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 4 + 2 files changed, 147 insertions(+) diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom-tbu.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom-tbu.c index e3202ed89566..7c7f02618a08 100644 --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom-tbu.c +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom-tbu.c @@ -312,6 +312,149 @@ static phys_addr_t qcom_iova_to_phys(struct arm_smmu_domain *smmu_domain, return phys; } +static phys_addr_t qcom_smmu_iova_to_phys_hard(struct arm_smmu_domain *smmu_domain, dma_addr_t iova) +{ + struct arm_smmu_device *smmu = smmu_domain->smmu; + int idx = smmu_domain->cfg.cbndx; + u32 frsynra; + u16 sid; + + frsynra = arm_smmu_gr1_read(smmu, ARM_SMMU_GR1_CBFRSYNRA(idx)); + sid = FIELD_GET(ARM_SMMU_CBFRSYNRA_SID, frsynra); + + return qcom_iova_to_phys(smmu_domain, iova, sid); +} + +static phys_addr_t qcom_smmu_verify_fault(struct arm_smmu_domain *smmu_domain, dma_addr_t iova, u32 fsr) +{ + struct io_pgtable *iop = io_pgtable_ops_to_pgtable(smmu_domain->pgtbl_ops); + struct arm_smmu_device *smmu = smmu_domain->smmu; + phys_addr_t phys_post_tlbiall; + phys_addr_t phys; + + phys = qcom_smmu_iova_to_phys_hard(smmu_domain, iova); + io_pgtable_tlb_flush_all(iop); + phys_post_tlbiall = qcom_smmu_iova_to_phys_hard(smmu_domain, iova); + + if (phys != phys_post_tlbiall) { + dev_err(smmu->dev, + "ATOS results differed across TLBIALL... (before: %pa after: %pa)\n", + &phys, &phys_post_tlbiall); + } + + return (phys == 0 ? phys_post_tlbiall : phys); +} + +irqreturn_t qcom_smmu_context_fault(int irq, void *dev) +{ + struct arm_smmu_domain *smmu_domain = dev; + struct io_pgtable_ops *ops = smmu_domain->pgtbl_ops; + struct arm_smmu_device *smmu = smmu_domain->smmu; + u32 fsr, fsynr, cbfrsynra, resume = 0; + int idx = smmu_domain->cfg.cbndx; + phys_addr_t phys_soft; + unsigned long iova; + int ret, tmp; + + static DEFINE_RATELIMIT_STATE(_rs, + DEFAULT_RATELIMIT_INTERVAL, + DEFAULT_RATELIMIT_BURST); + + fsr = arm_smmu_cb_read(smmu, idx, ARM_SMMU_CB_FSR); + if (!(fsr & ARM_SMMU_FSR_FAULT)) + return IRQ_NONE; + + fsynr = arm_smmu_cb_read(smmu, idx, ARM_SMMU_CB_FSYNR0); + iova = arm_smmu_cb_readq(smmu, idx, ARM_SMMU_CB_FAR); + cbfrsynra = arm_smmu_gr1_read(smmu, ARM_SMMU_GR1_CBFRSYNRA(idx)); + + if (list_empty(&tbu_list)) { + ret = report_iommu_fault(&smmu_domain->domain, NULL, iova, + fsynr & ARM_SMMU_FSYNR0_WNR ? IOMMU_FAULT_WRITE : IOMMU_FAULT_READ); + + if (ret == -ENOSYS) + dev_err_ratelimited(smmu->dev, + "Unhandled context fault: fsr=0x%x, iova=0x%08lx, fsynr=0x%x, cbfrsynra=0x%x, cb=%d\n", + fsr, iova, fsynr, cbfrsynra, idx); + + arm_smmu_cb_write(smmu, idx, ARM_SMMU_CB_FSR, fsr); + return IRQ_HANDLED; + } + + phys_soft = ops->iova_to_phys(ops, iova); + + tmp = report_iommu_fault(&smmu_domain->domain, NULL, iova, + fsynr & ARM_SMMU_FSYNR0_WNR ? IOMMU_FAULT_WRITE : IOMMU_FAULT_READ); + if (!tmp || tmp == -EBUSY) { + dev_dbg(smmu->dev, + "Context fault handled by client: iova=0x%08lx, fsr=0x%x, fsynr=0x%x, cb=%d\n", + iova, fsr, fsynr, idx); + dev_dbg(smmu->dev, "soft iova-to-phys=%pa\n", &phys_soft); + ret = IRQ_HANDLED; + resume = ARM_SMMU_RESUME_TERMINATE; + } else { + phys_addr_t phys_atos = qcom_smmu_verify_fault(smmu_domain, iova, fsr); + + if (__ratelimit(&_rs)) { + dev_err(smmu->dev, + "Unhandled context fault: fsr=0x%x, iova=0x%08lx, fsynr=0x%x, cbfrsynra=0x%x, cb=%d\n", + fsr, iova, fsynr, cbfrsynra, idx); + dev_err(smmu->dev, + "FSR = %08x [%s%s%s%s%s%s%s%s%s], SID=0x%x\n", + fsr, + (fsr & 0x02) ? "TF " : "", + (fsr & 0x04) ? "AFF " : "", + (fsr & 0x08) ? "PF " : "", + (fsr & 0x10) ? "EF " : "", + (fsr & 0x20) ? "TLBMCF " : "", + (fsr & 0x40) ? "TLBLKF " : "", + (fsr & 0x80) ? "MHF " : "", + (fsr & 0x40000000) ? "SS " : "", + (fsr & 0x80000000) ? "MULTI " : "", + cbfrsynra); + + dev_err(smmu->dev, + "soft iova-to-phys=%pa\n", &phys_soft); + if (!phys_soft) + dev_err(smmu->dev, + "SOFTWARE TABLE WALK FAILED! Looks like %s accessed an unmapped address!\n", + dev_name(smmu->dev)); + if (phys_atos) + dev_err(smmu->dev, "hard iova-to-phys (ATOS)=%pa\n", + &phys_atos); + else + dev_err(smmu->dev, "hard iova-to-phys (ATOS) failed\n"); + } + ret = IRQ_NONE; + resume = ARM_SMMU_RESUME_TERMINATE; + } + + /* + * If the client returns -EBUSY, do not clear FSR and do not RESUME + * if stalled. This is required to keep the IOMMU client stalled on + * the outstanding fault. This gives the client a chance to take any + * debug action and then terminate the stalled transaction. + * So, the sequence in case of stall on fault should be: + * 1) Do not clear FSR or write to RESUME here + * 2) Client takes any debug action + * 3) Client terminates the stalled transaction and resumes the IOMMU + * 4) Client clears FSR. The FSR should only be cleared after 3) and + * not before so that the fault remains outstanding. This ensures + * SCTLR.HUPCF has the desired effect if subsequent transactions also + * need to be terminated. + */ + if (tmp != -EBUSY) { + /* Clear the faulting FSR */ + arm_smmu_cb_write(smmu, idx, ARM_SMMU_CB_FSR, fsr); + + /* Retry or terminate any stalled transactions */ + if (fsr & ARM_SMMU_FSR_SS) + arm_smmu_cb_write(smmu, idx, ARM_SMMU_CB_RESUME, resume); + } + + return ret; +} + static int qcom_tbu_probe(struct platform_device *pdev) { struct of_phandle_args args = { .args_count = 2 }; diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c index 5c7cfc51b57c..7a58b1b96bca 100644 --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c @@ -422,6 +422,10 @@ static const struct arm_smmu_impl sdm845_smmu_500_impl = { .reset = qcom_sdm845_smmu500_reset, .write_s2cr = qcom_smmu_write_s2cr, .tlb_sync = qcom_smmu_tlb_sync, +#ifdef CONFIG_ARM_SMMU_QCOM_TBU + .context_fault = qcom_smmu_context_fault, + .context_fault_needs_threaded_irq = true, +#endif }; static const struct arm_smmu_impl qcom_adreno_smmu_v2_impl = {