Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752118AbdDCTUB (ORCPT ); Mon, 3 Apr 2017 15:20:01 -0400 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:60602 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751732AbdDCTT7 (ORCPT ); Mon, 3 Apr 2017 15:19:59 -0400 Authentication-Results: suse.de; dkim=none (message not signed) header.d=none;suse.de; dmarc=none action=none header.from=fb.com; Date: Mon, 3 Apr 2017 12:19:28 -0700 From: Shaohua Li To: Joerg Roedel CC: , , , , , , , Subject: Re: [RFC] x86/tboot: add an option to disable iommu force on Message-ID: <20170403191927.GA35817@MacBook-Pro.local> References: <20170322104900.GE8329@suse.de> <20170322115055.GA35752@dhcp-172-20-162-56.dhcp.thefacebook.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <20170322115055.GA35752@dhcp-172-20-162-56.dhcp.thefacebook.com> User-Agent: Mutt/1.8.0 (2017-02-23) X-Originating-IP: [2620:10d:c090:180::1:848b] X-ClientProxiedBy: AM3PR07CA0093.eurprd07.prod.outlook.com (10.165.222.27) To BN6PR15MB1636.namprd15.prod.outlook.com (10.175.131.10) X-MS-Office365-Filtering-Correlation-Id: 72f5b93f-613f-48db-0e08-08d47ac665be X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(201703131423075)(201703031133081);SRVR:BN6PR15MB1636; X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1636;3:K9IO0NOp9SmpdFQ52hsMbMBigMAgV2XxXTa1WFencR/mrEB2nt9XVZwDBVHPCu2TdLKanwfy6O9eEJyjYo4BVweyzjkLPDOCVQfpu/TI+UcmjNTs5hMg5y9eAXKFLKXgt1MACw4/m2Qj06x4Odd2gev+uBzDNUJIW+yxlP824qDEbFTQMblyFXevRBNBzsByeWwqKhzhYPdQpsX8AWdVNHhfmgQCnF6REftA+yTNFJY7iMtpvN1dfqZX49Tl9nkdmZw23Y0Fkoih99tfnszIvZbbylHqoEhMq39PaRsZf4mBLP6q8eimy4mmKigOTMVA6k4Iz77LChgepftKn0QqKg==;25:/y8h91K2a8PKI2jf2QHDrA8e8/c2WWdXF1IYBt0pKyB3PlsiLhwk+GolYQ6Kzr+X9wA/5SJjvmk0JsetDuhsiDYKqe48pcbc7og6WB6CydsyOisHzGzmUFq9fkuZQT6ZWaX4LtEiCNeNdEmrESELu1Sskjxrhn4G1GZMF5MrI2gBx6LwURlf0KbYNuHIXP1FfMtF8wui0bpmDAFl+jWtz+ikvG2XJO/WGsORT0I+fCIvbf9ru5NDgXzDtR0HYeGKHZr92+PqOK4/kl4MmoqmJ7Q/syv8OR/3IM3GKPBLoi6VKw8O47oHuprJo8jk3bxTYmN7KoehkDTDvN0RASfGZRe3UoBqRLMrsXo5M/hImLTgOJMUQNikGTTVGalnVl1kkiBjDqbtiCo2f4SHApsU9AyiLsFQ9EmM50Wv49ticgN5fqvMIo5bJXlqku55YcJZ2K6ZvWqUoVc6/lfbG3haUg== X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1636;31:9DaTXV1E1yu5pMWKzxWBoAV8t7WFe550AABVwABdxGDjOVgJh8YjxvojA5u2xZ+d+GaRsGzRYEIUkOKDnaO0HXqygrUN8qfCpIP1jFjlKzA4Ikw0ZDvAI0LLt2cSwTAvrrrzoqP5Yf9W6gzLqVU4GlM4SVGXhV1QK51Slb0/9sPICSE3k/JgCMMsFOpRu7JWIH+RrxfhmaraHMgluBjRsFmR9ewMwigs36U8wMQn3A58Obpynrv7+3vaVXyYQmkg;20:4395wbHzEaUYsrw+BhIG/r8k5N7+mxrcOzKIO/0xhAyPhT0r1S9ZobcnBwrgC64p1mz/ASkGgXMu7PtB83D9jUjor/MAkK/k17wwpAlfAmMT8zUYuxcHIriFt1mxxQpzxnMFu8Bv1B/to8LHNQCxOwL/1GPw2KmrG5n9VFd2qRY4JSm9QZX+TQFms1iYWIPHC0f+Esbmb9j4XvK72rzbD4vuAIl+IlNEB+l27IgbhwheH2VRrOn/wFGIywJI5+1GcBBndX1WIFw9XdOwJwLbCTPx16FAjqSErXCQs95Povw4ZMokCLFdWV4uTjLmYn102z7l50763mo+lWUJqV/uuDujMJHtap7DGYgdHifeTTrcJjhTiOk9xj4jWPUUIps6mOX0tfHtjeJCG1r3jWnYggJkiN7bpQNFIW1e67Z/ORCKB2ayiWdcLNMPrhGXLp8rYuZxQN/l5ppTFdRzW/1LjrY9nR2RiFwRQf9HXEXA4e8e+4ULTdGrJNYneH2TobTL X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(192374486261705); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040450)(601004)(2401047)(8121501046)(5005006)(93006095)(93001095)(3002001)(10201501046)(6041248)(20161123562025)(20161123555025)(20161123560025)(201703131423075)(201702281528075)(201703061421075)(20161123564025)(6072148);SRVR:BN6PR15MB1636;BCL:0;PCL:0;RULEID:;SRVR:BN6PR15MB1636; X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1636;4:Vp2qSGmFRE1AnVGA0kyKC9dSC4JnqjyY6p7dKgpvCO9cY8yqg9chgZVei2AlhrrNiUdEM9BNyDeG5h50+GWZ3UPDA+pKHnUWwptlrYg93sD2FFNC4gClDIk6kDonjx7C5NRJXAXS7zkS+DPa8TlUSqdOEqvE2Oumz5cCb/kgZrINHannJ2dznqSdrIEwiUynWnrftpggn5xYKc9er8oHS1KV1ugkM0GIHps7nf6q7ITG3Su3a9pTHNRp/CGN9trXRrqdmKcjDyz9GrNXnAKH86wvSw7qHek4HH73yLtuZqhsFnwbXiPnbm4Zxgxz6ttdyd5+Di37BRvy4MmIjd9g3u/oUuZt/wBbFZ1Fki93ei5J/E/qDT4GswPCeXuNoqLNgTqA7mdFOZeW+mZGFGxDYGegq0oroAWVrx/xn/t1kLgAP90zQMsvTWsM88HKWq1bxewyaUyPXaEzcVcJs4bdsuaGgIjmB4+TBnMOiL1HyjHMFGjUE+504w+umSYLdoMPeyV4Si23tsNy5UUNeCmquQjXYgPyBsGCm/I48xvnIQ/VjYXOOtI05RssWtgAc0iizp+nDJPrSwf4Lk1l/5RGKMHLA2KF1BaSS8Zm7ug9SSQu/XTfTI5JyhsRamy7KPbwfsPMG0fiDgOmSWpLXQY0/HpSrD4UnKo2U2422+FS5hh2Ffx3Vo9CupJ1qBz57+9S+j1oGAfZ4tHO/VZg06azd9LuJryekAFmZo4G/ShV6wYWldEfNjo4VA8gjZJ/q7Py X-Forefront-PRVS: 0266491E90 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6009001)(39850400002)(39400400002)(39840400002)(39410400002)(39450400003)(24454002)(50986999)(7736002)(54356999)(6506006)(23726003)(76176999)(42186005)(2906002)(1076002)(33656002)(97756001)(47776003)(46406003)(81166006)(86362001)(189998001)(8676002)(305945005)(229853002)(50466002)(6916009)(6666003)(55016002)(25786009)(53936002)(4001350100001)(110136004)(2950100002)(38730400002)(6116002)(4326008)(54906002)(5660300001)(6246003)(9686003)(83506001)(18370500001);DIR:OUT;SFP:1102;SCL:1;SRVR:BN6PR15MB1636;H:MacBook-Pro.local;FPR:;SPF:None;MLV:sfv;LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;BN6PR15MB1636;23:C31VcbX4A8ArOWDHKUXl2FvUlHPJS25b0NsbcjuMR?= =?us-ascii?Q?NBsXB02+mNfg4BLo4GmjBhBMX7KafIVJZRpRn9DGuNfTSEib0r1BFFhO0GfK?= =?us-ascii?Q?cjWg0+9c5lSYjgcZW/aN9XD8o7cNAL1SJOT5I0mj1bTPdYLMrU5hWviOX+oY?= =?us-ascii?Q?Aq9MNBrj+bPNytf8+8U0z4q9WjpDggeq/0OjYGFFwzELOgxoF/HlmExMIpWo?= =?us-ascii?Q?2Bg0Oo8rs9J7N06Oa9J1IPbYSnbrSwSmhfF+vdAmx47aE3bDTF5JdsNcNeZh?= =?us-ascii?Q?zbAHFdq8YB7lgDBLHDAzfeCsr0nWGcauGtilxEu7FAoEpathUsK9MipGPOV3?= =?us-ascii?Q?AFHWQiaPIv7ic725XMjGJPTz6gpzuk9OuCeZfKGngXrO7Rj5tmkRtRr9IeFj?= =?us-ascii?Q?AU7C/44IhqLaEpTsuplJRev54oTfu7QopG9/VSBYqOLX4ooX1ykKKJNuHdHu?= =?us-ascii?Q?swtad211vj+cu5S1PeYrjvjqUBkhN5sFZqAV9dU6qIhhMfg97qtfCdqoHyFr?= =?us-ascii?Q?kMbzzoi9NDcDTIT6hzRkvHRu1jwDiiPlfIRauUgYs0y/DDG19WLpJFTANmAF?= =?us-ascii?Q?mDv5t96QfNRkpGta7KHopTcEg4eSukVs0y+aDBuzq0WtxOWSI8PW2miGoVqj?= =?us-ascii?Q?5kBSv2G+b+J++FKfG90nqivH5iNF+qg3NDFoenX7TTDEa9kUA+Ydjtruonj9?= =?us-ascii?Q?FJIxaTruUkiNmcM31x4X9sC34T0YklOiiY1hLfL7dKpsnhbiL0PZ7TShlcWE?= =?us-ascii?Q?ttx3q+Ohbq5+9WbQqQsH/Nj+98WSiwMp83kp4QvyFzWLQeIJP/4rcp8ywvhJ?= =?us-ascii?Q?GVSwFZl/FHw85jn97Cxnb5jrk76iydYLF9rhjGM+bFIlf61R+en98AkZ9wmp?= =?us-ascii?Q?rwm+goRr2iWDX+yJLJyY0xDsVeZ9swuVWVB7FAjSf+LU8lVoOkL3Oa2DUJ+z?= =?us-ascii?Q?SRlidi0GPLT0Y6K8YxuHuNra8fY1rrFRh7dof0UrI0tPDUIbsOrkNyBrJSbA?= =?us-ascii?Q?XeLV7/cQmSsY8Q41h9L10s8C4v4WGPm5HmZcgkwBuoeFS2d6jMfxHIFf5kmj?= =?us-ascii?Q?l2e80LusrKnevFBFa05JwXcioUfM8fIzjh9vq8Szhh+TnLieZC7lkBbU0MiH?= =?us-ascii?Q?TWcfPx2Nl7JDB95sNUraBAnv6YYKdPH?= X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1636;6:H8em1pRjZDIFSRMUPdhCpUKqzfVMbcsNyboWTZ/rVXR/n3MyvggxKkeeJQxRJTLBwg2i/UPWm1/xPhBxb5EXkvIlD8jcYKosSrjH0JgPVvwtSOGBGwyG4CJg9qacs536O4GPmo+j8E47yi7LRJzAi1Qy5u7JtwAl03b9AgLUn44PgDveBAB4ZlgnnHHoSO6lM1MQMMfWvZkMIoa9xY8lbXRAnJZr4vCLzIQO2Ul3L5lWLMxbZxqjr5SogOqk/3z2l8UPdfBjR+DlC3OQcQcE54jeDKKT/zzugZPpmvYQDtKLo7PgX+qV5b+4l08vRW1DMos8X6s6ljnwkpBdRPHYzW6H3jw1mk+QrdpnbWk0H44lPU88Viig+ufxB3JkiAnJQt3CuMfZ0QLG86P3fPu29w==;5:tuXMD+aAqDrg+2ZqSpXHnmfYsbyVUaGjTikymg101m/z3A9kFUrsNT3S9nXnSeefs5fObU3Pb6I2wf+hEv9QREfNfMbtEzgY/bw9heybOx9E5QaLwCp/lfIvs4F494DAn6EuTzT9DwBaHYfBDOB2PA==;24:foGixkNJ0HXq265sjdl78jtko/qpnjQj+rooTwYPtaMR6ouzHOS6QFNiEzT5K1daFPU+Nbyl+E4/CceU08I8dUDZhZUhGCezlJA2mwQEgZ8= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1636;7:nPqy5AOOaHlSOvqSiNPiB5DS40I7A1UO2roHM2R8qzuHK2RYy7usfta+LIcwWd7yYQXRm9Z/OqJsrqLP+GUBp5AuYSqzpK1gzM9JGQsac7hPOnIDJgKJz+IWSOITR7yE9YEqFtAMx747x/eyAKxLkT0imv46tBeNU6tRlQfjHd7cCZhtjijg9JAkfR+y+YKwg3go2Y7oPTDlc7j1YkmbeyulDWsQwCCTwbS3Y1nNNXeVHRh8H9qFBUvym86nv2ehKTT8j1SW81PnYFqOYHzfcpceqyixkMYQyDDiBYhUx8ebla2BYrFU/EfuQOUpMkUAiKuKxET64qyojKhfIDe5zA==;20:6Vw5VJRR7tC/8pVU/fs/b09agUYWvoCrCu7fTT8AFXzOdpviX8UIR2kM7kHrKO3280zHHnhuadN+CxuH38sIAPu62qYlC6JsukUMjsP34zT0yjeqts49al0mfO6e3gLagu46i1ObjNoU2cWUjzPjMURl2gJd4aiPPnfCxnPcT/Y= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Apr 2017 19:19:46.5024 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR15MB1636 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-04-03_16:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1248 Lines: 29 On Wed, Mar 22, 2017 at 07:50:55AM -0400, Shaohua Li wrote: > On Wed, Mar 22, 2017 at 11:49:00AM +0100, Joerg Roedel wrote: > > Hi Shaohua, > > > > On Tue, Mar 21, 2017 at 11:37:51AM -0700, Shaohua Li wrote: > > > IOMMU harms performance signficantly when we run very fast networking > > > workloads. This is a limitation in hardware based on our observation, so > > > we'd like to disable the IOMMU force on, but we do want to use TBOOT and > > > we can sacrifice the DMA security bought by IOMMU. I must admit I know > > > nothing about TBOOT, but TBOOT guys (cc-ed) think not eabling IOMMU is > > > totally ok. > > > > Can you elaborate a bit more on the setup where the IOMMU still harms > > network performance? With the recent scalability improvements I measured > > only a minimal impact on 10GBit networking. > Hi, > > It's 40GB networking doing XDP test. Software overhead is almost unaware, but > it's the IOTLB miss (based on our analysis) which kills the performance. We > observed the same performance issue even with software passthrough (identity > mapping), only the hardware passthrough survives. The pps with iommu (with > software passthrough) is only about ~30% of that without it. Hi, Any update on this? Thanks, Shaohua