From:   Ganapatrao Kulkarni <gankulkarni@os.amperecomputing.com>
To:     linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org,
        bhelgaas@google.com
Cc:     jean-philippe@linaro.org, darren@os.amperecomputing.com,
        scott@os.amperecomputing.com, gankulkarni@os.amperecomputing.com
Subject: [PATCH] PCI/ATS:  Allow to enable ATS on VFs even if it is not enabled on PF
Date:   Wed,  8 Feb 2023 10:43:21 -0800
Message-Id: <20230208184321.867666-1-gankulkarni@os.amperecomputing.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?D+5FJUp9Gva8CD3G0rJ/rtJBvd/Gvim6t2RZjMZtsrvO8uh95uiTumllv6Z3?=
 =?us-ascii?Q?CPxHhTWp7u1d4fleLvBqcPT/rCzAiaorEZ7cItc81f1IPEQd/xytb1TcTjGK?=
 =?us-ascii?Q?4sYCV2DWTU9rTj9Y/ddv3D/heWqoXuJU39QqwmL3fFpCTYNM1IPkoemzmWab?=
 =?us-ascii?Q?HtMyd2ADW4s7sMpGFWUThpHMwSxlq9SU8oKzifH7l1aJjrbl321wnawZVQsy?=
 =?us-ascii?Q?KCBRuEHq5tLdZCn52SWBirBhbeFgLd+pZRk+IyrSAyw8iOSJh6/f0dThKtGZ?=
 =?us-ascii?Q?BXK1azkdOe/bhOGZaO+PKXQX787qliqMEFOq4ntRdbbNKlQal8K2GWI0/qjZ?=
 =?us-ascii?Q?x0MzTi2hyhmkxJgLU/GHRb5gNohmqQMphqNK426Mwf5tQQAwT97otQnS+FMY?=
 =?us-ascii?Q?C78PzCMoeuDEhJzgLAdFOuZp004sxS3EjMjMNDMwhHxsKZit3MTrGfSHbHtp?=
 =?us-ascii?Q?lDQafwqWsNVfHsZvVGSlQr4i13OC79e4/9fqrgzk1+utD/2hiLzOf8x0tbhp?=
 =?us-ascii?Q?LsVvXGDrqGfbP9f+im1Nbq2wRY7ztxGHuunf9qWZd3g2UL/TNkJxnOjHBtq0?=
 =?us-ascii?Q?c8HXWr1/xfBtB/ziiXIdzhx4PPU0iBMWGOmvchtbWO+xJY+JWUsGFWsJZvDr?=
 =?us-ascii?Q?ZW00lCGIce4+EpM+kT7/wjJpt0FUQ48YbgPGqdUBR7+RRxXn14ACTxTL856a?=
 =?us-ascii?Q?qek2hpEBG4rGywlkD4JGdQ7hO2OIzkukoORluRhktUTyRJfv+oxJuBPk6hkr?=
 =?us-ascii?Q?s3caqyMsomAgrP78sH0jgjXOZCPfWvII74fKmgtB9eJoak2RHhJeidB6kcue?=
 =?us-ascii?Q?y0Tokc4WF+PmIc8omxu99PID6JEwj+edqFRtq1ZDLgQ1qH65Bj+lXbV+BGgR?=
 =?us-ascii?Q?8iRw4JVNU1JcnW9RGLGU339X7zFjSMBANCNAl5Q8NXpqbsWO3Qraf2Tdz6Vo?=
 =?us-ascii?Q?/7v+K5esU1bqsgJsjVXfmyY1fJaYiv1kOicuFhYNHOQnyZ9S17RAQLgkDJxq?=
 =?us-ascii?Q?rtyqJA6DDOY/iAZ28jKh3S8f8rCX2ip2UTTIlPYD+Xogp3+Pp8FEdYJmIJbB?=
 =?us-ascii?Q?7mXA1SeHs/rri75L3PeJKZHK4mMq23brDHQK5ypTTSv+Jkh9UJPy5dkhBI3b?=
 =?us-ascii?Q?28BBztYf79Ai9OXBZy1X2Pfa9RehLAOdF/tyyrVZlXUSISaASO6RuZ4e8SGn?=
 =?us-ascii?Q?9EjIadG2rvKJYjiB1WyyPoFvOj14H5bmllWdktysaVG459PFxsDk/jSQjEAW?=
 =?us-ascii?Q?qhDPeFZk5IILpATwGQIpqfvEsfJopU953OeFE4Mtucn+agl22em8DGCgOdKs?=
 =?us-ascii?Q?9/OzsLPlEyrS7xXTCvYlDNMsqQuK4zUdOo+XMOnjRNqSQCCxfQaR/K9FN13M?=
 =?us-ascii?Q?DtCjJtbz7Sm9efanIFNyRFQQz5sobyWZGG4H6rCVuctXDtsTMrQpsNLSX2kV?=
 =?us-ascii?Q?F5yTrpbR2v+6NR799MkFCQNnTCL6pABn+lW7PCIPR1TZehGT9R3i/ElhQ7oB?=
 =?us-ascii?Q?HEhLxr7ajLWS8EkL3MzCpcakn8qyboBI2KPsG05w52Dkk7/T9yffRQwgpUMA?=
 =?us-ascii?Q?BVq9tQOgMttQ80M5WsDqiMM4oYkpU7cmAWSsjriK3nUMxcEd3kAU5Wsw7ZDx?=
 =?us-ascii?Q?UhKBeL+sun3sFjUKHcGk1qk=3D?=
X-OriginatorOrg: os.amperecomputing.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ef94d597-c804-4cae-101d-08db0a0465a2
X-MS-Exchange-CrossTenant-AuthSource: DM8PR01MB6824.prod.exchangelabs.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Feb 2023 18:43:40.7462
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: fznqysSwWkWUguFZ9vEb3wCu43X2us7bw/pTyTuAIcGvkYnNYN0ol5qzKtNXibBu7NPZbdvXo2oVfnqIhKB2q8LQL4yJRjTOP+EkO07KHNuHzFbeepVWC1QOdmzk3sU8
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR01MB5176
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

As per PCIe specification(section 10.5), If a VF implements an
ATS capability, its associated PF must implement an ATS capability.
The ATS Capabilities in VFs and their associated PFs are permitted to
be enabled independently.
Also, it states that the Smallest Translation Unit (STU) for VFs must be
hardwired to Zero and the associated PF's value applies to VFs STU.

The current code allows to enable ATS on VFs only if it is already
enabled on associated PF, which is not necessary as per the specification.

It is only required to have valid STU programmed on PF to enable
ATS on VFs. Adding code to write the first VFs STU to a PF's STU
when PFs ATS is not enabled.

Signed-off-by: Ganapatrao Kulkarni <gankulkarni@os.amperecomputing.com>
---
 drivers/pci/ats.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/drivers/pci/ats.c b/drivers/pci/ats.c
index f9cc2e10b676..a97ec67201d1 100644
--- a/drivers/pci/ats.c
+++ b/drivers/pci/ats.c
@@ -67,13 +67,20 @@ int pci_enable_ats(struct pci_dev *dev, int ps)
 	if (ps < PCI_ATS_MIN_STU)
 		return -EINVAL;
 
-	/*
-	 * Note that enabling ATS on a VF fails unless it's already enabled
-	 * with the same STU on the PF.
-	 */
 	ctrl = PCI_ATS_CTRL_ENABLE;
 	if (dev->is_virtfn) {
 		pdev = pci_physfn(dev);
+
+		if (!pdev->ats_enabled &&
+				(pdev->ats_stu < PCI_ATS_MIN_STU)) {
+			u16 ctrl2;
+
+			/* Associated PF's STU value applies to VFs. */
+			pdev->ats_stu = ps;
+			ctrl2 = PCI_ATS_CTRL_STU(pdev->ats_stu - PCI_ATS_MIN_STU);
+			pci_write_config_word(pdev, pdev->ats_cap + PCI_ATS_CTRL, ctrl2);
+		}
+
 		if (pdev->ats_stu != ps)
 			return -EINVAL;
 	} else {
-- 
2.39.1