Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751904AbdL0JVA (ORCPT ); Wed, 27 Dec 2017 04:21:00 -0500 Received: from mail-sn1nam02on0065.outbound.protection.outlook.com ([104.47.36.65]:31552 "EHLO NAM02-SN1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751145AbdL0JU5 (ORCPT ); Wed, 27 Dec 2017 04:20:57 -0500 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Suravee.Suthikulpanit@amd.com; From: Suravee Suthikulpanit To: linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org Cc: joro@8bytes.org, jroedel@suse.de, alex.williamson@redhat.com, Suravee Suthikulpanit Subject: [RFC PATCH v2 0/2] Reduce IOTLB flush when pass-through dGPU devices Date: Wed, 27 Dec 2017 04:20:33 -0500 Message-Id: <1514366435-12723-1-git-send-email-suravee.suthikulpanit@amd.com> X-Mailer: git-send-email 1.8.3.1 MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [114.109.128.54] X-ClientProxiedBy: HK2PR02CA0155.apcprd02.prod.outlook.com (10.171.30.15) To MWHPR12MB1742.namprd12.prod.outlook.com (10.175.55.13) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 806ff4da-0441-4c60-20a8-08d54d0b223d X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(4534020)(4602075)(4627115)(201703031133081)(201702281549075)(5600026)(4604075)(48565401081)(2017052603307)(7153060);SRVR:MWHPR12MB1742; X-Microsoft-Exchange-Diagnostics: 1;MWHPR12MB1742;3:aopidUggbGQrBuc3hCAaE4UytL2dp2qkHcDuma4dz6TQWsp3WrwXhVxPpD0B71zGE5OqFT4hz7zpo6W8gHipgw5pdXAnR/aBMlrtOBfPTl/yCmUi+4d3rknvqw7ZeMw8aHC+7weLnYg2HZqk7q3i0T8z/0498rjKzMGO2TcjZoivyCKOkr3ix9srwNNcRCMOHhiY1Ixes8wzQOg8o93Z4C6rlrJLtcbY8l4xxx2ImVy+MnvqMPtUu2jA3CEsuQM9;25:e5vQ/hEXbAlw3+uqvWlNfrTWMXRf+hW4gc4L2d6zfJFOsrzttClvVoMcS5Z6ff6wpe1eLoK4ObLPOFAPLqU1KizoeWtjyCTXXMlFqe6jtoR9Ry5Uwc0VubNe9zTZAg3Jq4adRyrcq69kzKMzN9wm53HqB+QvQoGMPRSygRKP8wMvm+5TxDaMOUl5JgtYW1fRvnMNDlsNJ6T0h8GP0I+9WVXyupg15YZM1jITILyOBJhDmsuSn0ER3+sKUIzNNkq/PObrooP/5GqkAY/XhaOaZ/sdjnU8JEqWb11oFf7XxhoM0mw7iRvhVkFoX8zsQSvbOyEka/qjSuUdj/IiaQNLrA==;31:QalQ/R/Kxo5jf9xXjwJfT3Of5z5QZVBXUC7SiBNELBdxO4lJ4iEpdYTDUxgBPvAE3fks99YHHE95agI8jP9mCHIX5oMJR+K2LFCSu/6BPgiMn5TeGCTZLKj6eOvTtvh3lhyCQJDGnDYaZqjVVBV09QODNsI35Bj8IjWe1lNR+QixJb5rdgrgObrhDmczRDpJo3hKLn8DInA/QIMwoUlJWmdts9fGF3aeTiAWFLiUFvc= X-MS-TrafficTypeDiagnostic: MWHPR12MB1742: X-Microsoft-Exchange-Diagnostics: 1;MWHPR12MB1742;20:g5+AIlP+SYOa8FmiFQ14IPwRAX5KWsmx+NkLczDVXLHQ3h83udZtdaICw3GRNIRIPycyUWqFfhWbeUEz+gtaqMyT64hTTLebklYVC5Kv7VHJsvK6ORlDt6SEGzBCaLAhAPyPXz+QR3YFv76nZNJUyilfnBdqeLz7grVdMSmMnjuZAY5fBsVncBU2d1R8z1DmlMhPRbAe+p9XQi8m8G8//uMPTU8sQd7r4B7YmU8Prr1KzyvNGCtB1EN61blm1c4M5LeCYx+QEXDGhTgxe/RCgNTP+CMO0pzd6uxHjEJAHdu73LrcAHVq6p/+k38hgkSxQE7RnXsA58tsSG3CVvLLPIlZaeARTsqPpo4AT2ng1mWXlEq1z4qYWG4Al+Jzhpx+KBu3THsWHAsXqP/LfwmBtLm9kry4hCfU09p3uJ3njSznLQQezyLlmjXbEDRAUHn7q1MA0Qs7cZDczIfyhPj9LoNr0GeB0mxxM3llQ420J5fEwWF7R+AIU0NcnBCoJlQf;4:RD/GW0dNyHDuJVHl53x0ualijsQV1wlaxcNYLYbYo9egh307ORIDjKrUI/dW6QZlHTlHAZ/cpK8p5DFuR3EfGAbZAx2jpWoFtbjWuCyqoJIM7QIrNiefcBxTOMiwNrVwSg6e0WaFiT0gpJ/UFN1RpuCnFnxt8OGzDoAjsv0m+FZuL03qCJyMIMeh6VNQ3siriQSwV93FbL/tX1/23gTGpJKvcbd+D8Zfck8YNP/OCsVhxVD+JeLi2NAha4x3cbnKywgldxTKLECBGNfUeIl0XQ== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040470)(2401047)(5005006)(8121501046)(3231023)(944501075)(93006095)(93001095)(10201501046)(3002001)(6055026)(6041268)(20161123564045)(20161123562045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(20161123558120)(6072148)(201708071742011);SRVR:MWHPR12MB1742;BCL:0;PCL:0;RULEID:(100000803101)(100110400095);SRVR:MWHPR12MB1742; X-Forefront-PRVS: 0534947130 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(396003)(376002)(39380400002)(366004)(346002)(39860400002)(199004)(189003)(68736007)(25786009)(106356001)(105586002)(6486002)(8936002)(97736004)(66066001)(86362001)(47776003)(3846002)(6116002)(4720700003)(5660300001)(7736002)(305945005)(6666003)(8676002)(81166006)(81156014)(50226002)(72206003)(478600001)(316002)(16526018)(16586007)(2906002)(4326008)(53936002)(51416003)(50466002)(6306002)(48376002)(6512007)(36756003)(52116002)(386003)(6506007)(217873001);DIR:OUT;SFP:1101;SCL:1;SRVR:MWHPR12MB1742;H:ssuthiku-rhel74.localdomain;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;MWHPR12MB1742;23:Wko27ZM9K4oDggRKq7gzyemDPYerOHLu6gdcsNPwi?= =?us-ascii?Q?sQMLNK4Z7Tw0knX2hw2giFgpvUAXKoTxs1w2TjJMnY5MIxLyauA2UdIU0+Yq?= =?us-ascii?Q?WYtEgkHSP774LJZLl9JqXRimwj+TP7dMXcylmfZD16iXjxozj1GwbP+l7cC1?= =?us-ascii?Q?uXEMPfqagLETFlYeuw1T9Q7n3YKcdQRY9QUl6wosLOmRmNaNECW1SeQNXMV9?= =?us-ascii?Q?5VNuy0Qclvl9Ha4kL+RtfoPOeXGENd3FrFT2T9kE21LGfPBow3mn3XEEVNsW?= =?us-ascii?Q?YHahxKD7XbMWJhWoT4AkilkelJlbZu2J+SEXZfr5xO5yswRyPNOgnPRvMiOh?= =?us-ascii?Q?aitxxET4T/vBL7iapyJoaLh41uZY3/VKFC6WstYKfYDCEJx4rNW0XkZFPDLH?= =?us-ascii?Q?7jnk/BOUNEun2uL5Vk4QW55l2oAetSiKRSsTqEIclC3XWdA9OA8rxrwCybAx?= =?us-ascii?Q?UzUkZ/MXBDeHpfZPmwoTysolA8YCVY4b0CvlXfHa6zjBDC9CIy27F9VuT9il?= =?us-ascii?Q?oRr2LukKFo9s0ZpAYfQ2HRoq4aIoJvslM7qVpQBR4dnTSD81GVPmDVC+rfFg?= =?us-ascii?Q?5/NixW9HpHmslVLZmSPhUr/DHfQ6/1V8K4IgfYAtPcEEUcVTzLD4Exd55j+d?= =?us-ascii?Q?EQR6YXMguD3O2st2XzCDNVlzC9DkBMSjrqG1oSQlkOTwShrTrqYCSCMtniac?= =?us-ascii?Q?nERWai9SnTImI87UuipObRO5j1tvrMTo4SoOSWIF7JS9BJH5Wia+MqSeht+G?= =?us-ascii?Q?mKZ42pqWc+OMCXQLxOAoIeJQHgAfdk6zlwz6N5CKQy4bmijd1sovsanWpm2s?= =?us-ascii?Q?jCMd8cPLvbkGN5UqItxUSp22TEvKYMu04yOYwQidDWKPtjZb6EmNtD0JCR2H?= =?us-ascii?Q?Sa7uqffCtaORC3ZkkE/bw5TCc7nvzpTiXPPeigvsmRJUaOemDlAODS6lx7Gl?= =?us-ascii?Q?kgrq1Ptw5CFGSsKJJ9vc9Veyq/SvuIxpIqZfUfYokXWGOTZmI2oP6wCfV4gf?= =?us-ascii?Q?sgxn31C0XrfkYhFsL0D1d14acSlRi/yhj6zV/QhsJzQgrmoEQgz0BrDVjUdD?= =?us-ascii?Q?364eqJuWQ51kiR003nhW8JGxb5XriQPhMRqgd72qsJa7T1N2w=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1;MWHPR12MB1742;6:8xK+yJfe8gfJNXm9qcezR2vkFJQHewMIPHpdj+t4IrRR5nJ6+Z/x8dFN99/a/KbKT0nzOwIHgCIxM1d6A8M76sGfzLGFVTiluPWQAZnu1TS3xWHBmYirHsPwpvcmsPpbZzspoWt7IkwPWlRZjnQE46Q0a4y7fAADls/2wPE1XVjSqLjRImnIoFSx85QAZAEwlqEdC2XLB2zh1kkN+OGVuxq20sdT2G/gyoW/folKsdnDucaww/ealhCtx3+AwnEUwIsjfU6QTBiwFOCMv7K85PMsfGYewwS7v1imBGm+U2hNi4oPExW2rQp6dd/fY+v70lYvQaHdR7RtK5+67GF8ymRwemRM7k5BcxmdMl8iJ14=;5:rsxfs0csizcVf9tf6QXCNdHf1XDdmrNKK0IwPYrLvm0UCMMSLvKzAxagCf5jXBhCqPcLXR5ceOUGYmVlUseq+gLj32O1ws8NOqC1LUVdXJdGyxUap0fTp5wqdSRipXX8D2DKSGNffQdIUtzaba4ShqBCn1Q8x2kvtBCemv/MGmo=;24:6v/z0U31RQjibqttMU8ZrTGRbrntlsQq1LBkyZbbnKyOk9gdLIKqLqP0esHy/b4DblWOLMthEFcQNH0FTDIYrAlNsL+2zLxoTp9e4cO4oE8=;7:7h8RV5R+cXMAQlzXs5iY5nBXVeGnfFG0dU0gWI4fLVIM54PxBUa2ximM8mGIYWrXaedT3zj02rFJ0PPvPiOA1OtquBHizAw8NL3p37DELYm0PgP/owuanIgmrs/uUjb+u7PmZKmnB3pSvgBdXFeuMgo9AD4Svc04vPP+zAURWx6Hp/y7T4sQMcnSG6wgmjL7A63kh6pJ1fe3v+p8Aahdcjc4m+4NzP1ZK7OFUFsmo1S+4eSZr4TufS+1WT4Vb98W SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;MWHPR12MB1742;20:2zyJSFyhTWOqRmNj2JyzvpUeUjzKBYO319nhs8hh96P2b9I59FKSfGzuMda74r3kiUo5P2QXgZBz/4WFYeoeJYnIRHCRIZ6zezPwPJpxmxXP9cenOEOJMGnRQmlA8DkwrYOpqv8gpxRJk66UyEjcw+V5KRJYJwwFRti2wfldPIYIsKHzkM8WQc20GqQS1DH315AAHGj/l2fzvdJ3PFu4n2v9FhgCBEWplyhtnJmNgAoPvDnekek7CjIfxtNUMgob X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Dec 2017 09:20:53.7047 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 806ff4da-0441-4c60-20a8-08d54d0b223d X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR12MB1742 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1347 Lines: 37 From: Suravee Suthikulpanit Currently, when pass-through dGPU to a guest VM, there are thousands of IOTLB flush commands sent from IOMMU to end-point-device. This cause performance issue when launching new VMs, and could cause IOTLB invalidate time-out issue on certain dGPUs. This can be avoided by adopting the new fast IOTLB flush APIs. Cc: Alex Williamson Cc: Joerg Roedel Changes from V1: (https://lkml.org/lkml/2017/11/17/764) * Rebased on top of v4.15-rc5 * Patch 1/2: Fix iommu_tlb_range_add() size parameter to use unmapped instead of len. (per Alex) * Patch 1/2: Use a list to keep track unmapped IOVAs for VFIO remote unpinning. Although, I am still not sure if using a list is the best way to keep track the IOVAs. (per Alex) * Patch 2/2: Fix logic due to missing spin unlock. (per Tom) Suravee Suthikulpanit (2): vfio/type1: Adopt fast IOTLB flush interface when unmap IOVAs iommu/amd: Add support for fast IOTLB flushing drivers/iommu/amd_iommu.c | 73 ++++++++++++++++++++++++++++++++- drivers/iommu/amd_iommu_init.c | 7 ++++ drivers/iommu/amd_iommu_types.h | 7 ++++ drivers/vfio/vfio_iommu_type1.c | 89 +++++++++++++++++++++++++++++++++++------ 4 files changed, 163 insertions(+), 13 deletions(-) -- 1.8.3.1