Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754364AbdCFQdC (ORCPT ); Mon, 6 Mar 2017 11:33:02 -0500 Received: from foss.arm.com ([217.140.101.70]:37808 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754370AbdCFQc7 (ORCPT ); Mon, 6 Mar 2017 11:32:59 -0500 Subject: Re: [PATCH 1/4] net: thunderx: Fix IOMMU translation faults To: Sunil Kovvuri References: <1488538070-12549-1-git-send-email-sunil.kovvuri@gmail.com> <1488538070-12549-2-git-send-email-sunil.kovvuri@gmail.com> <20170303.095635.355325147613813698.davem@davemloft.net> Cc: David Miller , Linux Netdev List , Sunil Goutham , LKML , LAKML From: Robin Murphy Message-ID: <9a809c6e-3b01-4deb-e69e-65aa62d1e5e1@arm.com> Date: Mon, 6 Mar 2017 16:32:54 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1309 Lines: 36 On 06/03/17 12:57, Sunil Kovvuri wrote: >>> >>> We are seeing a 0.75Mpps drop with IP forwarding rate due to that. >>> Hence I have restricted calling DMA interfaces to only when IOMMU is enabled. >> >> What's 0.07Mpps as a percentage of baseline? On a correctly configured >> coherent arm64 system, in the absence of an IOMMU, dma_map_*() is >> essentially just virt_to_phys() behind a function call or two, so I'd be >> interested to know where any non-trivial overhead might be coming from. > > It's a 5% drop and yes device is configured as coherent. > And the drop is due to additional function calls. OK, interesting - sounds like there's potential for some optimisation there as well. AFAICS the callchain goes: dma_map_single_attrs (inline) - ops->map_page (__swiotlb_map_page) - swiotlb_map_page - phys_to_dma (inline) - dma_capable (inline) Do you happen to have a breakdown of where the time goes? If it's mostly just in the indirect branch our options are limited (I'm guessing ThunderX doesn't have a particularly fancy branch predictor, if it's not even got a data prefetcher), but if it's in the SWIOTLB code then there's certainly room for improvement (which will hopefully tie in with some DMA ops work I'm planning to do soon anyway). Thanks, Robin. > > Thanks, > Sunil. >