Received-SPF: pass (google.com: domain of linux-kernel+bounces-174602-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <f207da41-5222-4d1d-897d-9b288e33a547@intel.com>
Date: Thu, 9 May 2024 16:43:00 +0200
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v6 2/7] dma: avoid redundant calls for sync operations
To: Robin Murphy <robin.murphy@arm.com>, Steven Price <steven.price@arm.com>,
	Christoph Hellwig <hch@lst.de>
CC: Eric Dumazet <edumazet@google.com>, Jakub Kicinski <kuba@kernel.org>,
	Marek Szyprowski <m.szyprowski@samsung.com>, Joerg Roedel <joro@8bytes.org>,
	Will Deacon <will@kernel.org>, "Rafael J. Wysocki" <rafael@kernel.org>,
	Magnus Karlsson <magnus.karlsson@intel.com>,
	<nex.sw.ncis.osdt.itp.upstreaming@intel.com>, <bpf@vger.kernel.org>,
	<netdev@vger.kernel.org>, <iommu@lists.linux.dev>,
	<linux-kernel@vger.kernel.org>
References: <20240507112026.1803778-1-aleksander.lobakin@intel.com>
 <20240507112026.1803778-3-aleksander.lobakin@intel.com>
 <010686f5-3049-46a1-8230-7752a1b433ff@arm.com>
 <5d6a26f5-5acb-4c5d-aa11-724399d1348b@arm.com>
From: Alexander Lobakin <aleksander.lobakin@intel.com>
Content-Language: en-US
In-Reply-To: <5d6a26f5-5acb-4c5d-aa11-724399d1348b@arm.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?L001ZTBoN0tWVjRpS3E4VTRnRUNsOVpWOXI5ckZSYjNHK1pYSjNydUNlckpv?=
 =?utf-8?B?OEVOb3NwQ2RnMG5mcWFHcnhXTEtWcWphN1hFVUR0SmtRNGJ2RzBiM2ZUT2gr?=
 =?utf-8?B?SXRDRm5SaWpabERZMmdWcnRGRU95Sm42NTBJcVBVUmRTNTJXZ01CTFBsR0po?=
 =?utf-8?B?R21ma2xOREcvZ0NUSjFpZmNKU2dJTXZVRWlFM2ZhVVJlaTNYbjd3d21vWXBz?=
 =?utf-8?B?S0FJY1BLNXdvSE1iSmlmV3Y0Q0w3YzdEdzEwR0pJYkRSSWVUdEtGR1FuS0RI?=
 =?utf-8?B?a09oekNpS25mUjIvcHVTeVBSM3ZjWHBTaXdtZi9tYWlVVUdyMXJlcW5VR2kz?=
 =?utf-8?B?K3dFQTZkQm9LQ1VpTTNSVDJVNTVGZWxkTEtzc3R4QThHNG1ZV1JXK291YUtE?=
 =?utf-8?B?UUtMN2tKQTkvRnBhNzlLajZGMFBzenJBL1FkMTBZYkUzVkhtZVhReFdzdmtt?=
 =?utf-8?B?OEh1Y3F3dnJhTHJpNFJJeUxJeEhqSmk2SS96K1VGU0FpQlgyeHZVT2g0eFhC?=
 =?utf-8?B?WlZBQmo3WUt6bUV1dDNjUkZGUHFtRzR4MjFvTyt3LzNVeUZ5RVRBeEcrV3pT?=
 =?utf-8?B?bC9SUzBNWC91djdCWXRNbjRRZklhZlROQ0xNc1hGMHBya1RhcjFSWmNsUHBu?=
 =?utf-8?B?aVZIMFJvVjlSOHNQWW9yeFdrY0s4WHlaU1d2Uk9jb3g0MVA0Y2YzaTNXWnVE?=
 =?utf-8?B?R1pFTUNFVExKSk9IR29NYVM4WmYrVGFpdWF5Z0hrc0l3WWRJUUg0dDdrcW00?=
 =?utf-8?B?WnBGclRnT0FvRjkvSm1uZ1FxVlRmb1pLTGlwajAyV3orcXlkQ1RwalUzRG9N?=
 =?utf-8?B?ME1RZVlYM3hEL3dzWkxNb2NaRkFwNnh4a3h0M3BwSzVEZ3RoT2ovOVY3V0U4?=
 =?utf-8?B?K3NqZGFza2dkaFhlbWpxdzFmRDh4Njl4cmZRbzI3K3F1Tlg2alRvMFZYU2ZP?=
 =?utf-8?B?eG1FV0p5T1NZT0NJaFNJT1VqQTU0dU04dEkyclpxbFlRa080WnFETzZoczYv?=
 =?utf-8?B?cVk5RzdsZndXV1BZWXk5ZStaM2RDZmV4U0lreXpOUVExQ1ZkeDRCYzNXMWhx?=
 =?utf-8?B?WGdNV3c3WDhXS3llZGtSdlBWa0taSVc5VStiZHRtd2lhdXpRS1d2TUp2YTJH?=
 =?utf-8?B?bzBEOFJrYUFWOG5pVzNIUkNmK1dLbTZXZGFjc0JmRGpjQVQ0U3dnQXA3M3hZ?=
 =?utf-8?B?d1pVcjYwdTVDdFk5RjdDNkN4cTYvNlRRU3FIMWZtZ043aVg5MU9rcWNCdXRs?=
 =?utf-8?B?NzBBaFdJeUswdUxpREVIK1hoMjNsOG9qb3pQY0kwVHBSQmhqL1NIVUlmVFQ4?=
 =?utf-8?B?ZjE4anBuakwwMklhbTFEUlpQK1hrYUo4Y0JGUTY1ejdobDlIL2xOUWpzNUN6?=
 =?utf-8?B?Qm9BQzNaTjh5OE1UVXFXaTJhZHMweG5aQzhORVd3dTRacHNHLzVMWGtBWFhv?=
 =?utf-8?B?MGQ4ZWVrczVOT1VEamZIWVZMZ2FJQUdDMEVIZkNHZks2U3VUZmZkY0FJb0xu?=
 =?utf-8?B?dUU1dzVuYUV0OGpoQWtpOFlHRGpaSW9pdUJqaFpVUUJuNFlpUmNKTGNDMFZS?=
 =?utf-8?B?RG9EUmpoS0ovUVV4YU5NdEtGZTBGU1pzSXRRenNCUzVjVUhvaGlyR0NaVzFR?=
 =?utf-8?B?VEtCMDBmNisrNkNlemZ5WHdQTm03eXQrVEpFeXZ6WkZMWU5yMnM0RENVYW5u?=
 =?utf-8?B?clFvVFI4SjNZRHRxNmFDRldZTlpuYStTNWZBQ2RKL3pubTIyK29VODgyREZH?=
 =?utf-8?B?b05jUzZseGdVVVlpTzNreVpWYm51QzVaai8vb1lTR1BLMzF6RzczSzhzWVZ0?=
 =?utf-8?B?SWVoOWRjN0V0bjh3bE9KRi9YMEs3RUpCUjV4TUlRWEllcWp5anBwbm8wOXo2?=
 =?utf-8?B?TmFWbWI1WWJaWWRvb2l6aUdkaGtCb3UrTUhXdlpjK2tvRk95OFVpK3RQRmNa?=
 =?utf-8?B?WHlZc2djT2dmRVFpQjFDd3NrWkpEaEhEMVp1OW9xY1BYTi9tbnpmSXhiOElN?=
 =?utf-8?B?U01nTmdPTnVnUk5CdGtiR3I2eUVUQzhMVCtQK2lTL1ZSL0wxQWxZYzlScTJH?=
 =?utf-8?B?NDlWWG80TXROQUF4MjQyNTRqRWdJdldkQUhVTDIzcW1VVVF3NVBpS2U1c0Iw?=
 =?utf-8?B?a24zaFAxUy90TFh1cmFYWjhRdzlUcGJHOVZES1dwaHBWWTB4cm1uUys4RjZa?=
 =?utf-8?B?YXc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 66c5bbfa-1e6d-4d1b-db36-08dc703666ef
X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB8718.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 May 2024 14:43:33.9879
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 5PaaxB6PVbH1rzb/F9UmMMZPOOn0pLAy2xbSWZxmIzfbDxkyK355azC1w8ZAk3++PrdkObFncPlFXbrrhRMwSQkPWnBqpTxjlDV4TQfr3mA=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB8009
X-OriginatorOrg: intel.com

From: Robin Murphy <robin.murphy@arm.com>
Date: Thu, 9 May 2024 15:33:13 +0100

> On 09/05/2024 2:43 pm, Steven Price wrote:
>> On 07/05/2024 12:20, Alexander Lobakin wrote:
>>> Quite often, devices do not need dma_sync operations on x86_64 at least.
>>> Indeed, when dev_is_dma_coherent(dev) is true and
>>> dev_use_swiotlb(dev) is false, iommu_dma_sync_single_for_cpu()
>>> and friends do nothing.
>>>
>>> However, indirectly calling them when CONFIG_RETPOLINE=y consumes about
>>> 10% of cycles on a cpu receiving packets from softirq at ~100Gbit rate.
>>> Even if/when CONFIG_RETPOLINE is not set, there is a cost of about 3%.
>>>
>>> Add dev->need_dma_sync boolean and turn it off during the device
>>> initialization (dma_set_mask()) depending on the setup:
>>> dev_is_dma_coherent() for the direct DMA, !(sync_single_for_device ||
>>> sync_single_for_cpu) or the new dma_map_ops flag, %DMA_F_CAN_SKIP_SYNC,
>>> advertised for non-NULL DMA ops.
>>> Then later, if/when swiotlb is used for the first time, the flag
>>> is reset back to on, from swiotlb_tbl_map_single().
>>>
>>> On iavf, the UDP trafficgen with XDP_DROP in skb mode test shows
>>> +3-5% increase for direct DMA.
>>>
>>> Suggested-by: Christoph Hellwig <hch@lst.de> # direct DMA shortcut
>>> Co-developed-by: Eric Dumazet <edumazet@google.com>
>>> Signed-off-by: Eric Dumazet <edumazet@google.com>
>>> Signed-off-by: Alexander Lobakin <aleksander.lobakin@intel.com>
>>
>> I've bisected a boot failure (on a Firefly RK3288) to this commit.
>> AFAICT the problem is that I have (at least) two drivers which don't
>> call dma_set_mask() and therefore never initialise the new dma_need_sync
>> variable.
>>
>> The specific drivers are "rockchip-drm" and "rk_gmac-dwmac". Is it a
>> requirement that all drivers engaging in DMA should call dma_set_mask()
>> - and therefore this has uncovered a bug in those drivers. Or is the
>> assumption that all drivers call dma_set_mask() faulty?
> 
> Historically it's long been documented (at least in DMA-API-HOWTO) that
> a 32-bit DMA mask is assumed by default, so as much as we would prefer
> to shift expectations, there are still going to be a great many drivers
> relying on that :(
> 
> Perhaps its time for dma-debug to start warning about implicit mask
> usage, maybe that might help push the agenda a bit?

I also thought of this, but currently don't know how to detect whether a
driver has called dma_set_mask*().

The fix will arrive in several minutes.

> 
> Thanks,
> Robin.

Thanks,
Olek