Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S934365AbdCVLv0 (ORCPT ); Wed, 22 Mar 2017 07:51:26 -0400 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:59563 "EHLO mx0b-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932606AbdCVLvZ (ORCPT ); Wed, 22 Mar 2017 07:51:25 -0400 Authentication-Results: suse.de; dkim=none (message not signed) header.d=none;suse.de; dmarc=none action=none header.from=fb.com; Date: Wed, 22 Mar 2017 07:50:56 -0400 From: Shaohua Li To: Joerg Roedel CC: , , , , , , , Subject: Re: [RFC] x86/tboot: add an option to disable iommu force on Message-ID: <20170322115055.GA35752@dhcp-172-20-162-56.dhcp.thefacebook.com> References: <20170322104900.GE8329@suse.de> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <20170322104900.GE8329@suse.de> User-Agent: Mutt/1.8.0 (2017-02-23) X-Originating-IP: [2620:10d:c090:180::ad2c] X-ClientProxiedBy: BN6PR18CA0012.namprd18.prod.outlook.com (10.175.188.22) To MWHPR15MB1647.namprd15.prod.outlook.com (10.175.138.151) X-MS-Office365-Filtering-Correlation-Id: 4207c1f2-962d-4cd5-3660-08d47119bcb2 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:MWHPR15MB1647; X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1647;3:v3FVIYLBC4P9vNW9irHGkkM+XcpKbuS4+iF8BVkCrb0yjP//tMsKKoNNtanmtlHcAL0dg11xFIw1E8n+e8eO9LaFq+v7DRz8+p8iB6yAUowP0TzCNOpEupusYC4mNU1buC9T4Ni8CB9oThYOr+NRmkZP3WdFmV+x0my5ZFSK7KnAHNZGpU7uryifQXEa0Pfm421U+qNUzWtI8quTpqOe1MMRu+Bu4f+HznYEiODgYKZBNeUkQD1iUu1hFHPnKy4+OTv6MRrsSLsbYb0sajojzQ==;25:04tncu9HqafLc5sr+T9HcATA6IrLek2ZzJc+5TLWBs21/h+r+gG+slnK4o5e6InpFX2OMgjbUVC79uoRoLPUuk/LbA6rvGiHYg/KtJI+hRNFWqFoykhG+p4SWYcNkj8U8tMD7VId2zrirDayL33IzgTfVYnCvmURtkzg0xN6JBIas238jAekQCd2htZbTJTuQp6zMc1QFLn1QzxX5sJH+34KCmU1CKBemzv+wfEEH1n28eg6LeBdGOEZ6E5POqEEUCIrlVGj+vYgPZFPHd2iS7lQRJamy92p4DmFrkGxwavsRSQ/64LhNA8em/BG2jMYlX/xQbooNlXTxrOkl5vFQwcuTUvJs0sro1A5MgYKYj1Vn2uoV3+qqTLISWDx/xLoEEKe/f30VSkZ1DBpsFrhQuj6t3C7xMTdIGB6sdu0PWOQ20sdV5cGaPLIvvKz3IL4zBK8RHEtWktha7AoVFN86A== X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1647;31:Sj5NpfvyVM0XXGl11Tp3uCsrYZjcpjIc2GC27bNSuKictSwk6cEmeuNxwm3EFG9oOXPE2KOILmluc0pVG3f4bvnhqaPcgXk4YcnKB4opJz2QpjWG8nJ2LlhPnP+EkfGwwRvA5elh9Adu8SHx7wSTu7vvmIUQuZA3+4wKya7iTEhLVBqTcp+/8FW9LAYf3mdtXwVKb++e/C9+OJYoYL7iZAyPkOWdYo76/fPC7F2DjYkuxczRO0zO5GQ5oDPSrjcF;20:8UHetojIV/puHXa0LHANgq/0qg5vJn3ny6p5bPaOx8MIwrg0FNkXBZoFCR1E/EgMjgYd8snYzIuh7avCu9qLG2qG3YPjhDWFG9xN5y1W8Vghl51KXZpJ8DR9iscg1ypkIevxJWtzcH7/gx3lSe6A0otwwzAz2O2i0NpCovfb8XcZIbkXsOXU+qZXZ/bbqEjuBXn52qYUd29s3jTC9LfUPLCYwh6qP9wydOl3H3+b44XGT6a9yVLAs1XyoUCJeYM0Pf9UAgsOnRsd4hOIDedFquy/oz4LUHtmRrnPKlwGVhIu9c9gREaCwbrQgKDnPGjkvkYtPYJBWLnZbs+9+PhjPyrnS3WBAuBoIZ2KaoFC2IWKEkM6F9EfXYIS81/yT0GLOUyTOxmHNcrQcmQnCh5WbCEfSgMy8i2/29c/FR4RxasHHYer+G6gMj80BjfRQ8770zPvwjZNpp0VJo4oDDV+m8RFq3O8pjCRkEYL4TouXmqW1lxlIt/Y8A7/CsgRPWbb X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(192374486261705); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040375)(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001)(6041248)(20161123564025)(20161123562025)(20161123555025)(20161123558025)(20161123560025)(6072148);SRVR:MWHPR15MB1647;BCL:0;PCL:0;RULEID:;SRVR:MWHPR15MB1647; X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1647;4:hY4bnc3M+tnfT/Cp3YGS8+LX1wiUP+4l44obKky5TyfxVQjmd6qTuUDYFdfL3sPT18PUdhClULfmQ5guwtMlcCq3mjmtirvVdw5K8TPN1Su3854Rhmue0MykHy0cDXZ4s4ekxOkxOK39h17BMh7quAnCaTHqCdWRY/4fY+N3vejfUgcRajpFh7eERumMc5fyAfH8u6ZuUL2JvxxS3mbSwRgAn8WD/e5RYcbdI44NXgXouveuwtg0Q4bvg0abvBrIppYMpmxlbj2vftk0Q+NwQ8IgASpYIUbsOkqPeGeURu1piFo3KAIJOUxFYQPxRBGW2m8QVsU9d4jw6fwjnGn+KCbolwquzoTfaNzQFpy9Z3v7WBxLsfU2HtjD4lD+CMLSo87BLufeq8QREiKx9ADWHapGHahIH5T0JI3WVuoiys2IHjyfzPk1j1vAxp4cK38o9ALj8ARrFKmZF3uRYrH2OTULz4wPMnHtqf5Ozg3OR5Qu5uccZAwr9VnVCNjitWzLO+XVl8gMa8MI+wPIAaZelTIuIpQwk3ICrX1KfrynZp+01Q3SLDV/8B9mc/iamiJUYagHPiW6UrdwTT7KgBM1saNSCRZIKlcvulfQHC/finoznFKA+yDNNkGD6RooXdkQtvzQiWvM+GUhAQzGOeq9Gg== X-Forefront-PRVS: 02543CD7CD X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6009001)(39410400002)(39830400002)(39450400003)(24454002)(6666003)(4326008)(53936002)(2906002)(9686003)(2950100002)(6916009)(47776003)(4001350100001)(8676002)(97756001)(81166006)(189998001)(25786009)(6506006)(76176999)(6246003)(38730400002)(55016002)(110136004)(50986999)(54356999)(229853002)(46406003)(305945005)(6116002)(50466002)(5660300001)(7736002)(33656002)(83506001)(1076002)(86362001)(42186005)(23726003)(18370500001)(102196002);DIR:OUT;SFP:1102;SCL:1;SRVR:MWHPR15MB1647;H:dhcp-172-20-162-56.dhcp.thefacebook.com;FPR:;SPF:None;MLV:sfv;LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;MWHPR15MB1647;23:6jSxOQQxaaPygntwGEmuo7OCRH5ZsmDoKALFUI/lZ?= =?us-ascii?Q?zoGPfWW+rsfP4fj6B6KAylY0cL3VS59QqaygPfDhPaL15F1aXUnb6ZqEKyWm?= =?us-ascii?Q?k74cB1vxG0aU7ZBY6XYtDJEmeZJioFkysMcu+r/+SD7vFx0zs/5/lMHHfDzL?= =?us-ascii?Q?0etumEWYPq+x0eD4C388gXPk5eXl95gS7NXNlx4ZJDtj42nKpcYg5CtDSs4e?= =?us-ascii?Q?I3m2wp7qX38NTRPrOO11zFlgSeTvxTWoSClk2/c86XeoV9MXBxQaFj3uy2sZ?= =?us-ascii?Q?7+2IAT6ct0F4eAa0q+lFdXfvqW93FJmFks6MEzZvAf9oAoyNYMOpMVqySCR3?= =?us-ascii?Q?aoHeV5e5UIvJNl3C8phkI+sog3YzfXauIsgH4nMkCr1CQfJe5040pCfu/4Io?= =?us-ascii?Q?h4RpRp3bFElHsoOiGFzy9VM7bRF3CKJKPt3ie54dJpWD2vquVnPomwpsChPD?= =?us-ascii?Q?IRm2uCD6ko5DE45TBUEXG1sOL0SEppMpOTg6TJ1vKIBE6GDL2DPD28RQa6WZ?= =?us-ascii?Q?1K4X8GEAgHYBSUG2a3EiZNxibntzwWIRcmHDL9GuPpvxfPLTmI7VsyA19rJg?= =?us-ascii?Q?+Qgl5RUtVR/Pel3DogltnZlkob2grnpudup4CdLWekDDOdQALdDglI50GCBa?= =?us-ascii?Q?0YcW86nbgWJiBQSpSnQmzvvCUa/OxN37OBhTBqCkCc8YugHMVXEOLnMlmD35?= =?us-ascii?Q?zn46nYZUDdQdv121qCVjgsI63EheWK6XJ5yyhkQRKFv0dK1Adz889Y+cHLKs?= =?us-ascii?Q?6hR4/Tvo9hv57wLTyOoogbcOmmtLlvsVv760azh+oy9yXM7GcstyESV0QoyA?= =?us-ascii?Q?Uolpuu2xElXFhI4GUWmLWnOZtfRfRfHD02IQlCeoQU4oggWbjTBv+prM9FO1?= =?us-ascii?Q?karhOedF9YOYiRpzDLOnubdBK9F4qhyF/LuMsHgzTO6KfBSFPTCLekEIkZSY?= =?us-ascii?Q?YvoRPpLMSM3ncoZioIrRxudjv2ai2f8UmrZH1YR2BHDuGVgtrk+uL8ASXyct?= =?us-ascii?Q?DMRaT1ck9NQYbl3w/qZG371IX6jung2S9rh6WrcHK9P4Nbx0dVGcrLsI5LbD?= =?us-ascii?Q?sCW+oVwGVwI0oejOovhlTUUsiE2?= X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1647;6:CKECFbBrGgQaMgYD1rq32hqbE1I+iLl13ZMmXkGyoFchIc7UJ3B1fROvHYlo56rWAv8vozpa+VinbfW39y0KmLBU9bNVxmFkdRwXK7ojze2vgyj3Bvxzg6k896aSA6mhTy4O8ywSJqIZhsjkBiWyAA9MFVwtTzWhKluymj/QhCQ2MugeBb9zkPhUiGfh5HwQPGCcqPk7KCchr0BfvALZvhy7/u6LRsCAw1Zy5Km3iVXc0lTbUNArpiK0ahXo+s6v+GZDV7mqRyt+9kMjz0E6Ri9VgVXjJxFpJmTjwJ6t4Q7zcVXdMDnY1eCOyqXIFs7M99BPVpcRmiSmUW7egDah42xMBDJ+bn/IRGHBGtqWcWkFswA8O5IacP5cWtWNN8Yn40IqIXIZ6WO5m2/9IgNmrg==;5:D2mghUkwfjLKoqrr/TW0mju3FyCn3qmuSM9uBCmk3k6D0FC2RXuoHDggXAq3n/kd69TuWtNZ59PtujCHkTyyEnW2O0h/5POlkdRulXAnlDHfIgRtj7UhZys2GHm9o1fu1vbwUZa+p3w7JT2LMAHOgg==;24:72XXh0gXHgLwx+PLOxIJ8hvFrYeXFfeTp1C1KEBt/tl/zFnP/gMSPqj8aggo63q6ryIuwz6kSsIYWb3v15SuNtjKLDfuIRwDSvqdqZpGTDE= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;MWHPR15MB1647;7:Z7cCKlgcbtwT9rH4kSv4EPaU7i00/vUpD7mlbCJUiWswo8Bz5NkpLyZzOYU+R81CKBABpRaODw6QswEwi//7KF37gL6+OFv2vMGlkfll3U9TzACQhiLrsD/qiVueRnjHRe2ghI6W5QVNMc81VhPlsqApPVyBaFwKJG3cqF1D5I66yw1JjMwRuRwcXHocOgbD0x2DJBRVRdAs2Ss8NXIr3LGFAinIwF9hPKhz29ZuZCF7eWqaewT2YpeQE5QP+TQFk4fRiWhPPgMcm1YBdByWFaHdkamgQtL0zT2d0skZcIPr8LfwbuYe8k8RP0rDZT+s9MJGWHNFlVBOVRnP2AYDLg==;20:0Z5KXDyRunvxN8Pm9t+dDae0oSlWEYymspF1v1ItpHREyYq7GnUHEjqHiFckhw6Cnv+tYTe8syvRZWJ+SMFrP+Sjt9gix8GHw9gsaXKNN6R9c3hud8qfPmUbO8FArayhZrpllG0zvg0dDXMBZDlQrJWtMbuZSHBv89H+nJ3RuMo= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2017 11:51:09.2666 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR15MB1647 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-03-22_10:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1120 Lines: 24 On Wed, Mar 22, 2017 at 11:49:00AM +0100, Joerg Roedel wrote: > Hi Shaohua, > > On Tue, Mar 21, 2017 at 11:37:51AM -0700, Shaohua Li wrote: > > IOMMU harms performance signficantly when we run very fast networking > > workloads. This is a limitation in hardware based on our observation, so > > we'd like to disable the IOMMU force on, but we do want to use TBOOT and > > we can sacrifice the DMA security bought by IOMMU. I must admit I know > > nothing about TBOOT, but TBOOT guys (cc-ed) think not eabling IOMMU is > > totally ok. > > Can you elaborate a bit more on the setup where the IOMMU still harms > network performance? With the recent scalability improvements I measured > only a minimal impact on 10GBit networking. Hi, It's 40GB networking doing XDP test. Software overhead is almost unaware, but it's the IOTLB miss (based on our analysis) which kills the performance. We observed the same performance issue even with software passthrough (identity mapping), only the hardware passthrough survives. The pps with iommu (with software passthrough) is only about ~30% of that without it. Thanks, Shaohua