Date: Thu, 27 Apr 2017 08:41:20 -0700
From: Shaohua Li <shli@fb.com>
To: Joerg Roedel <jroedel@suse.de>
CC: Ingo Molnar <mingo@kernel.org>, <linux-kernel@vger.kernel.org>,
        <gang.wei@intel.com>, <hpa@linux.intel.com>, <kernel-team@fb.com>,
        <ning.sun@intel.com>, <srihan@fb.com>, <alex.eydelberg@intel.com>
Subject: Re: [PATCH V2] x86/tboot: add an option to disable iommu force on
Message-ID: <20170427154119.GA26498@dhcp-172-26-110-153.dhcp.thefacebook.com>
References: <1c2cadcf5cd7d19cea93c56435610e61b551bd1e.1493223474.git.shli@fb.com>
 <20170427065142.lnsdegq7zwxacqo2@gmail.com>
 <20170427084207.GU5077@suse.de>
 <20170427144901.GA98840@dhcp-172-26-110-153.dhcp.thefacebook.com>
 <20170427151855.GW5077@suse.de>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20170427151855.GW5077@suse.de>
User-Agent: Mutt/1.8.0 (2017-02-23)
SpamDiagnosticOutput: 1:99
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Apr 2017 15:41:30.9371 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR15MB1643
X-OriginatorOrg: fb.com
X-Proofpoint-Spam-Reason: safe
X-FB-Internal: Safe
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-04-27_14:,,
 signatures=0
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 1571
Lines: 29

On Thu, Apr 27, 2017 at 05:18:55PM +0200, Joerg Roedel wrote:
> On Thu, Apr 27, 2017 at 07:49:02AM -0700, Shaohua Li wrote:
> > This is exactly the usage for us. And please note, not everybody should
> > sacrifice the DMA security. It is only required when the pcie device hits iommu
> > hardware limitation. In our enviroment, normal network workloads (as high as
> > 60k pps) are completely ok with iommu enabled. Only the XDP workload, which can
> > do around 200k pps, is suffering from the problem. So completely forcing iommu
> > off for some workloads without the performance issue isn't good because of the
> > DMA security.
> 
> How big are the packets in your XDP workload? I also run pps tests for
> performance measurement on older desktop-class hardware
> (Xeon E5-1620 v2 and AMD FX 6100) and 10GBit network
> hardware, and easily get over the 200k pps mark with IOMMU enabled. The
> Intel system can receive >900k pps and the AMD system is still at
> ~240k pps.
> 
> But my tests only send IPv4/UDP packets with 8bytes of payload, so that
> is probably different to your setup.

Sorry, I wrote the wrong data. With iommu the pps is 6M pps, and without it, we
can get around 20M pps. XDP is much faster than normal network workloads. The
test uses 64 bytes. We tried other sizes in the machine (not 8 bytes though),
but pps doesn't change significantly. With different package size, the peek pps
is around 7M with iommu, then the NIC starts to drop package. CPU util is very
low as I said before. Without iommu, the peek pps is around 22M.

Thanks,
Shaohua