Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1030564AbdD0Plz (ORCPT ); Thu, 27 Apr 2017 11:41:55 -0400 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:36665 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S937775AbdD0Plq (ORCPT ); Thu, 27 Apr 2017 11:41:46 -0400 Authentication-Results: suse.de; dkim=none (message not signed) header.d=none;suse.de; dmarc=none action=none header.from=fb.com; Date: Thu, 27 Apr 2017 08:41:20 -0700 From: Shaohua Li To: Joerg Roedel CC: Ingo Molnar , , , , , , , Subject: Re: [PATCH V2] x86/tboot: add an option to disable iommu force on Message-ID: <20170427154119.GA26498@dhcp-172-26-110-153.dhcp.thefacebook.com> References: <1c2cadcf5cd7d19cea93c56435610e61b551bd1e.1493223474.git.shli@fb.com> <20170427065142.lnsdegq7zwxacqo2@gmail.com> <20170427084207.GU5077@suse.de> <20170427144901.GA98840@dhcp-172-26-110-153.dhcp.thefacebook.com> <20170427151855.GW5077@suse.de> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <20170427151855.GW5077@suse.de> User-Agent: Mutt/1.8.0 (2017-02-23) X-Originating-IP: [2620:10d:c090:180::573f] X-ClientProxiedBy: DM5PR1601CA0030.namprd16.prod.outlook.com (10.174.111.43) To DM5PR15MB1643.namprd15.prod.outlook.com (10.175.106.23) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f5b91e75-f102-4835-d165-08d48d83e0ea X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(201703131423075)(201703031133081);SRVR:DM5PR15MB1643; X-Microsoft-Exchange-Diagnostics: 1;DM5PR15MB1643;3:cNuBxFZJE7tjpcG8ZASE/4dLza1os7b24abFcOtaZasUn4k8FGq9fpqv5iCDXNjRihCNsvv7ASsBG1Me9umbZngHDyGeYN1SdJu8EAjeddRaT4bJ+5GGBtffqo4SZVImcCCzynEVXvuif+7E8EhMcY6Z/slydu4mfccW5IsyyUpuegXOdYehBvVP6wFh29n6mEXH2wb/eSsV/UuAK4QFrKABdMj8v+g6V4zzcOFw6za5FDNbNCQpmottABbQenn6ggXiGUkAlHOtgqyiooDN5+La/igIKPoemyARYHXUXcxiQoNTgnW0bhw5Xr2M1/t7HL0V1N+yMH9n6R4bpkbrLw==;25:JYByZjwlvP9zuCJ6xAkfVaKIrir8m28OYqMy7ivgh7OL2spDXaraWN5DUYnObtSNJZx9KANWLvfFm9wlivkEsz8DimPto1QBnMAf3GGUn5BNfr/iqzb/F2npPQyhj24Wyos4B9Rbp1zGyslogVEuHQs6PmpIVL67NilmEtu/JfALdmkzEbsyvlW9VXWYxIgSvH9vvDnkRfDLgeZQeqSDgoQNHD6t2aRalhYp72LI3V8+ivR6mns27rQuK6SA1lwy+zbq4hWZEnOuYapSFqTaataphG6dOoveOsxXID0oL3W9AyetMkcKE3wEGikI44AtEBasa40B2xgeMXmgd5wfsygIoszyFZ9CHDbevOAgw3Zrp+tFukV9ruyfYW1ja2sAEaKuEQhsFupEO2BV9HAOkRG5BfISA0JQtjMxLM+DCtmKgo0PEfwE4pgoVjdqIckz4rOdKNWNSyRATtE3M+hQ3w== X-Microsoft-Exchange-Diagnostics: 1;DM5PR15MB1643;31:sveYN+mL0ZN417K65NrHeDZTMQRuZUtRhhj2GX+61eMtVCnALRa8hr/NsKIfXaZ8+LyCvcEpqT3DBEotgDrTqZpwmO0ZdG7Khnc0jFAnWLRyph3kcsWh3fZEooI0vHW5fiNKbI2qasNP+Q8ZiHtUVNiUuw+B6h/gIztcwMknGAHtDpP1abS+J9h2ShxLu84EmGgh9hW0yyUgm13J3su8v0SstNlI/mHt1Lrsp4EddqU=;20:2EaJYOg8UJVH2FNQ24lt3tg2N4hqwDsRQiz1EKCkCJUHMEA8c+pAyYL1zpXPaVqqWdh+uPC+BJhf+nzTa4gettVChZT0TC00AtX+N0cVQrJ8j81p6O6PtOft1j1HjeqqlLSg/IqmGnlAWC2f+3XZTd1RHlB3SPP2FXt7NZf+poIoKqhtMVUGIvVnxy3sGFz3JT8wgfHNOzlgUYgXZX3Vyv/UePBFRTYa9sutxmBNppz1KjvLpcs75ZLi8ZCLXbpZbch1h+IJQ3fbpz9zwp9LE7q5QPnBeU0woE/UIyCP0IQt6x5HEdiDSoIFVvYVYFtNh9luhZxYBhZuTJRKIhwnEAapdRxPgwzBAZd/hY8RfLmV0wUVKVdJ2RDqC0fWspe/8hQuDymdTwt0neGy1nQjMrcScmEXDYzMeNOMRZS+1/FoaTRM1O1yRfRHIgfy1DfTy1cso+nBGIroN6o93hZz4YkMg66+uVcbinpx7EOgFw4If246PQMWI2G9vOZTqGJ7 X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040450)(601004)(2401047)(8121501046)(5005006)(93006095)(93001095)(10201501046)(3002001)(6041248)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(20161123564025)(20161123562025)(20161123560025)(20161123555025)(6072148);SRVR:DM5PR15MB1643;BCL:0;PCL:0;RULEID:;SRVR:DM5PR15MB1643; X-Microsoft-Exchange-Diagnostics: 1;DM5PR15MB1643;4:tmcKA9dwLw9I5tfUxCY5RvLD4fpsUaIZbw3NkgnU/04Fkxnrqz5hCaMfAIzr6DQt4+HudONtICppzRjiaHRjuXTT1h4MmWJCv2FKu8PyRlKBycMY86/jiGY8dhaAP92fiVJsfKbqrMXx8fan0m+JsDrXvwsrWyb2W6S43rcTuHBg4OI3wDuDZS3iHtypmXnO0rYCq0Xc2iOlKi0F5HiuRo+ppznekhM3W44msnIZ+5n18EA1Skwu0FJbK8Lz3DAek69nRj5Wnn92tCILagPpJB+oQO06wSTCt9/iMqZIuLAl+F/t2conB5HDx2FU07arkD5k40ajQRXq1U1883//Lp89i8Mi2K6mhZV12jVVNhMF19UzUFLoaBiZI+XPim2mcOrSV8aPijxcg6WFgVNifTZydfWhUbMOoXbWGMC8S90lH8lHQlA1FaU4rq8e2dRd19YkcJZgRZm10igR53yQImJ/mJhwvett6AvzPGMYKQvN8DnZy0vhaMbMKIWRNF4HWv+3JtlTmZAMJZXh1rYBgnmWMJe3nc1UkScrYO9WxiQtzPJ43jfjHokhClFNpHhmOpkRmPIGJ82e2nsIpfTVwh8CfFCkfTUiwOjl2WwfJk2zyGZgSjHCb0Ley6pXTq+VPXF3nRmlBtQQQ6fFD8qyt9ZKxLhqhQLcGzx+oEaP80XIrKlzz4kST5TKBiEzlmFvaUYaa1vDbW8njqb/OJd5b1wRdtlRjDKd20nenWonJxJyhVYIz5NwH/lFbKfE/oHQFzY1SzDbTD1STSZYOSMivN0ViAGuXLKj2b4PWkxzbac= X-Forefront-PRVS: 029097202E X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6009001)(39450400003)(39850400002)(39410400002)(39400400002)(39840400002)(24454002)(2906002)(33656002)(7736002)(86362001)(47776003)(305945005)(6116002)(76176999)(23726003)(1076002)(4326008)(50986999)(54356999)(25786009)(42186005)(97756001)(189998001)(4001350100001)(50466002)(83506001)(2950100002)(46406003)(6666003)(6916009)(5660300001)(93886004)(229853002)(81166006)(6246003)(8676002)(55016002)(110136004)(6506006)(38730400002)(54906002)(53936002)(9686003)(18370500001)(102196002);DIR:OUT;SFP:1102;SCL:1;SRVR:DM5PR15MB1643;H:dhcp-172-26-110-153.dhcp.thefacebook.com;FPR:;SPF:None;MLV:sfv;LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;DM5PR15MB1643;23:aN9Zw3qR6RhjGfQxU7Vk6/ccn3uM4/4ercMRyqZ0Q?= =?us-ascii?Q?13c/fxnKAjlqJ9RoYGW9Mh9bfomUaM9+pi3bf06xoRJ91/72Oqj3Cbe6U6iv?= =?us-ascii?Q?4yyTaVLArnJm+ooHrZbXREDnbydJ8AF6GSqBvsX9YdLfLIuEmQz7WcV3u9zh?= =?us-ascii?Q?Oniy6+hY6ES6WjEx0xQ9c2jQz3TnDz6O6Vp9VAeFZGZOe61fL7KKRVWRTwIk?= =?us-ascii?Q?vQ5WLZI4bKaEbOuCizb7dLVrauw8V9aDoqYeqS72w3SN41XnO2PU3faR4C9t?= =?us-ascii?Q?I8WhpAH0PsONU78bO8D0DaS/U2pLwGaWFa52tbeYT+JB3XDfXiw8LqC162V/?= =?us-ascii?Q?dA80cL+W54JjSY7gh9W3VxN/YIotq/actHfr9iweHn5tLqKj3Zn6ptsevkcQ?= =?us-ascii?Q?4LBZZC++ydQB/Jq+SC3p8gPv5XkAhErI7bLi6kgx3diy2HJBHffaRhzYJ5V0?= =?us-ascii?Q?U2vr0smrDPgNvq3d3UNTx3pxn/VYBHztvYMSbuUzCBWcVYbKOeWYvIKqeHc/?= =?us-ascii?Q?cE2tg1t1IaEh+34p7/Muvj13Y35bUIUBfb8rnaKNovt0C1BhKfSWMsImfFJp?= =?us-ascii?Q?mzXer4+7BKKrJZyACeNmyrchtqfrMQMyvOE13z7Emmu4DUnMzBlMEERucUj6?= =?us-ascii?Q?Fifs5rfhRozzrRvMNwZup0iddtPZJ5jkpvkYCx3OQDg8Kc8Qt6IOBnlB5WWD?= =?us-ascii?Q?QGyC97RvJT3IPKt/mUn4ONoNln9Snw/1vxXi75ZovvDVRFYHILapfu7U2O+x?= =?us-ascii?Q?pUCUG9n/BKz2aw/RUWk3SUof52+2qFXtgsk2f2FtjRudIZTmhh2dV3xlFSnL?= =?us-ascii?Q?EtAjWg/+p77me/xVRXyjM9Z1rONN806HylrqKsE3/gXTDfYuY+x5iOosaASK?= =?us-ascii?Q?/bJL7xBZyyzJu+QcLr4ulJEV13f9p+K1CyW4S/1lvf3A4N3ohEQvWQrStZKl?= =?us-ascii?Q?ztog3qFLkbIfrwanAwCZvO+rfnkVBKt4JtbR7hF4u+2kPs9kuld8DQA0+yY0?= =?us-ascii?Q?S9UbZQc5espEUPOS2Yvg6yQ8qOKi4p55ExkGM731wkdrTkqgG3/Xxs3qW/5W?= =?us-ascii?Q?8qZv4bzya3MAy+pyQP1wbZm3ikcZhJ+YVanElBuwZTAOasfxlRBmUfltUzaU?= =?us-ascii?Q?zAj3PGp3aYPph9Wj0DoqACKC3jKrm8t8KCiucdj49UcVXI1cNG1W2AGAWens?= =?us-ascii?Q?SLAQN176lEtVG8=3D?= X-Microsoft-Exchange-Diagnostics: 1;DM5PR15MB1643;6:7PetFwVc0HPKU9mau3oc51k/NDhpn/ANSi/Bo/bDGhLOEevvUwjrTnkOUJrwoBWYDA7BlC4pwaf5jiNsG3EOHgy7cKDJdYTP4AnFJ6PiR79yqqQGxvK/NE77BXD6owYYIum+DbMMjjt4kjlkSoIZ+NdqnuHbYTkYSsjLht4e7fOYgkZ7Em6RirPHJr0buzxW7czxDUw5ARGnBM/U4Vcxt9ajE71xKo7eU7bab05XuI9Wwm5+H/FPkzh6qAp9ohkbBhAVFxizDZNmqAFrYHRRgEancs8o0LAYuqw3xonkk3Gud44//OnXr8bFYoeKbZHVmLVtlSkEjKcsAurBeiJXJoEKlg4svbRYDNEGz6l96NjQlPe7sH+0pLtK48Rj96InbcqT/XBG6wEzRW/lj3aDlCgA1BtxONM2k3QI810htaWMF2tHOikWUN6UW58SG4g//tg1C+KlqJivjBxNYAX+QGTF7Dpe0uVS/TwRIwSOLhNVCi6QjBoSsgQB8oDtM17xP55x7TDnSnanIWy1i+Q9Fg==;5:l0EPPCTwXp1lU9TSs1BFExWmg4GqKsIFKtJZ0b373X9mHSFN6skDLkDgMLykk+4mMJE7RLhnJdfbmKNeQkO6ssPyYQSkSrSDk+yJpHFBrapk4OLittL7JALbGLNa82/+LQhd4PP3LO9+ZfeeQlO3Qw==;24:FkZFxJ923y5NfUeP/NipzFVvVgQotSUt5bpI+xbNY07Aa8V9qEIzuiCydz2kHcMzRnthHoj5Amx2Nm6xm4d7kSZxO7108eIpG3XrXQOxbHc= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;DM5PR15MB1643;7:X7t+GiVcyBqLA8SMOoOpd7hEmJsAVdRXbGvqqI1dLaZCMhT3NWoIs5af3Hvr1MOCtJQqrm/4mPasUDSyWIAdaxwKJ8vpP0yKKLilwKYl26S4UpY6fQOX4XhhjVeX7DkR5TJn8X7zei/AJlBrqFcySbZbAqyzvvmY11lg4ykz/I8HWE6PxjP7g19GFK15izVkPktBzb/qeVeCqvB8X7wmRMpWK7j2l4l14+ny3IZ/pt34Np7N8kh8r3JyEBCngVjHSIoGP9p54JOh1t4JCbHysKbSI0oPWgVviAfua974EIso/fYhC6UEudo2HWksFrBrsykoh09VIAZ5SEGYgU27Wg==;20:9X1Ug/s5OctAhX1XJeCkeoTdhyLU8TZuGmmpJreL7FnEe5QzKoo8DDJN0fjVp3jEERrSYl3vlcVFcfqPzqvmTIOSZvJhr/J8QuPiSOF/7lpBqioEYP7RM75CU8zCqPsLEkfelzJOHL6R33/Kughs7y5twBnLaj79IGERZSEga70= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Apr 2017 15:41:30.9371 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR15MB1643 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-04-27_14:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1571 Lines: 29 On Thu, Apr 27, 2017 at 05:18:55PM +0200, Joerg Roedel wrote: > On Thu, Apr 27, 2017 at 07:49:02AM -0700, Shaohua Li wrote: > > This is exactly the usage for us. And please note, not everybody should > > sacrifice the DMA security. It is only required when the pcie device hits iommu > > hardware limitation. In our enviroment, normal network workloads (as high as > > 60k pps) are completely ok with iommu enabled. Only the XDP workload, which can > > do around 200k pps, is suffering from the problem. So completely forcing iommu > > off for some workloads without the performance issue isn't good because of the > > DMA security. > > How big are the packets in your XDP workload? I also run pps tests for > performance measurement on older desktop-class hardware > (Xeon E5-1620 v2 and AMD FX 6100) and 10GBit network > hardware, and easily get over the 200k pps mark with IOMMU enabled. The > Intel system can receive >900k pps and the AMD system is still at > ~240k pps. > > But my tests only send IPv4/UDP packets with 8bytes of payload, so that > is probably different to your setup. Sorry, I wrote the wrong data. With iommu the pps is 6M pps, and without it, we can get around 20M pps. XDP is much faster than normal network workloads. The test uses 64 bytes. We tried other sizes in the machine (not 8 bytes though), but pps doesn't change significantly. With different package size, the peek pps is around 7M with iommu, then the NIC starts to drop package. CPU util is very low as I said before. Without iommu, the peek pps is around 22M. Thanks, Shaohua