Received-SPF: pass (google.com: domain of linux-kernel+bounces-40367-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Message-ID: <7ff3cf5d-b3ff-4b52-9031-30a1cb71c0c9@intel.com>
Date: Fri, 26 Jan 2024 17:45:11 +0100
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH net-next 2/7] dma: avoid expensive redundant calls for
 sync operations
To: Robin Murphy <robin.murphy@arm.com>
CC: "David S. Miller" <davem@davemloft.net>, Eric Dumazet
	<edumazet@google.com>, Jakub Kicinski <kuba@kernel.org>, Paolo Abeni
	<pabeni@redhat.com>, Christoph Hellwig <hch@lst.de>, Marek Szyprowski
	<m.szyprowski@samsung.com>, Joerg Roedel <joro@8bytes.org>, Will Deacon
	<will@kernel.org>, Greg Kroah-Hartman <gregkh@linuxfoundation.org>, "Rafael
 J. Wysocki" <rafael@kernel.org>, Magnus Karlsson <magnus.karlsson@intel.com>,
	Maciej Fijalkowski <maciej.fijalkowski@intel.com>, Alexander Duyck
	<alexanderduyck@fb.com>, <bpf@vger.kernel.org>, <netdev@vger.kernel.org>,
	<iommu@lists.linux.dev>, <linux-kernel@vger.kernel.org>
References: <20240126135456.704351-1-aleksander.lobakin@intel.com>
 <20240126135456.704351-3-aleksander.lobakin@intel.com>
 <0f6f550c-3eee-46dc-8c42-baceaa237610@arm.com>
Content-Language: en-US
From: Alexander Lobakin <aleksander.lobakin@intel.com>
In-Reply-To: <0f6f550c-3eee-46dc-8c42-baceaa237610@arm.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?N09ObXFOdzFyak9XamVqMmR2dHl6VmZkWVlEQkpPWnkzdkhHa29ybDI2SVU4?=
 =?utf-8?B?dHJEWTVzZktxcSsrUm5QZGJCVGs0TUxoM2xDS1VoUmhWNlR5YytVeWt4cXB3?=
 =?utf-8?B?a29HSDcxaVBqdmxtN2VYV1JBeURib2t0MFByL3c4ck90UWR1YlU4UUp4cnZn?=
 =?utf-8?B?ZGV1RXM1d2Y5RURFZ2lUVW9maTFxc1ZFTUVpdHhYMFZlVCtXZFljQ3Jra1VV?=
 =?utf-8?B?MDM0MmtXSElWbllOb2pKN3lBeHVNYlJzak1TSElHU09IWEJWc2ErSFZLMDhl?=
 =?utf-8?B?bVhPNUxLOG5NYmdGVGVnYk9ZMWdsL3ZlUlpqbUpHUktWdWcvU2llSFlBdnFV?=
 =?utf-8?B?eHI1Vm40T1QrTkZOejBOMStDMmk3Q0UrMUI5T0svZHhiZ1JxVEZSblRPWWYz?=
 =?utf-8?B?UDJpS2x6TnRVcFJtWmJRd3IrQnE0OUgva2JtSjExQkxPT3NZbmtrTUt5V0Ey?=
 =?utf-8?B?SXpxL216V281dVAzU0NjSUptYzRqYjRDSjVvSlJQN3dSSjlSVFVnTVZpdkh1?=
 =?utf-8?B?Slc0bUZneTAxcWg4M2JhQjVSdFpsZXd4UmdSQllWcEN2dDNuT01BaUdObFlv?=
 =?utf-8?B?eitvQzUxOTNyOHFxNlFxeGhiVDY4VERQbVV2VGdaSExhYURUQlhlRnVWTUp3?=
 =?utf-8?B?dDRQKzRMcUhGS3d1SW50MExnRFlSU3kzQThhT2VNL2VZenA2ei9TcnZmc1cw?=
 =?utf-8?B?RWt5RHluZWxhcXUyOE93b3IzVTloSExkcXZWZVFGREpkR01VRUk0YzExekV3?=
 =?utf-8?B?cmpIU3EwbjNiS01MTisvV3JQbGNDUlNhbXlhaFZCSVJuN1E3bENJL1JZNGti?=
 =?utf-8?B?aGpaaTd1QTVhbVV2SHdTaFA5R0lFNUNSNFB0M0EranVvYWxyS1QrVWF0dnpn?=
 =?utf-8?B?U0g1NUNKaldGcWtRQjc4b2htWlZ0SDhRL1NSbUZoNWJ4eTBtS0VTYWpMbE16?=
 =?utf-8?B?dkJ2NUpoVHVJRFhBWVYzM3pQa1RVb21MQ0tNeVZZM0JYblY2TlpUcG56L2lX?=
 =?utf-8?B?aEV0azJEanQ3c0NzV2I3ZEhHYWQ3RmM4TXNQVGo3U0dUR1BnNDVJV05LcVJV?=
 =?utf-8?B?UHpnQldtc2hDeVozV2tWS1Nxa3JiWEdwcmRCOFAzN1E5RWd2cmh4N29iNVZL?=
 =?utf-8?B?bjdvTUJuMys2d1V6OGZ3TGcrTWJJUTFKQWtYU1BHbEgwY1Z5OS9hWEs4K3Vu?=
 =?utf-8?B?K0VFenNVSVJFL1lRVzdmUVRxVzhwckY4UEE4V2VSUmpHYlpRYTN0UVloMWk0?=
 =?utf-8?B?dnZtMWJhVXpUa0E1bkZldi80cDJPYnZBSkh1bnNFaE1STEZMb2RMaTZKQjNi?=
 =?utf-8?B?eWJjRSs4T1NHQU1sakYxYnZ3ME9kQUF0M25zeWN6aWtHNjVHMytMSW5JNWN6?=
 =?utf-8?B?YjhiUkR4K3N6dEJHNFl3aDRkNWxVYkJuK1VpajBDNENpNTNNNkJtTHdzRTBW?=
 =?utf-8?B?Mi8rSTEwUGVPaU5QVFJHYytRVUdWQ1YxNG96cENHNHNXd2RtWVFjaitRR1hk?=
 =?utf-8?B?QXJ0NEpYZ2ZpY1JzODN6dzdORjVIK2dLOVdsSFgzVEZqQmdvS1BRcHlPS01i?=
 =?utf-8?B?eEFoWkFzb3RqMXUzeHNOWXlLZGNOSjB6U1B1YkZZb1E2dTdXVmRYeFprd2pr?=
 =?utf-8?B?dytYWFRJNEpkQ3RvVUxTL0J6ZjE5eCtINmplNHNrMGFQcEhUc1VTMUdTdEFC?=
 =?utf-8?B?Y1R1Qk1HY3BDbjRxTVNNTWl1NEhOenFjR2FFbEQyRzFPTzRCbjE4cGo2NE9W?=
 =?utf-8?B?RTRqR1ZTRGExYzRBZy9LRGw4UFJKVjQzVW9DTlBSQTRHemdxTDJJSGRjcWZ1?=
 =?utf-8?B?R0lDRFNEU0FXYnZQbUg4WXpKcVkrVkN0NXVKZDJKNWk3Zm1pSmNHc2pKcCtJ?=
 =?utf-8?B?Z0tlbEdSOWJ0SEZ6K2dOVmhwSzBrYXQ0VGJKT1VKWFNFUXBwT1F4ZjhLNzg3?=
 =?utf-8?B?L21SRXZtbWNuc2xrS1ZiZW0xNlovYzVDMmNGQnpnL3NaMzM3eERkczhhaFNj?=
 =?utf-8?B?ek5TbFhBMWNVZmN0YzhXWWpvWStTMW1lVzQ0T1ppcVY5bmN1M1Z5cG91UWxm?=
 =?utf-8?B?djVNNU0wRmZxc0xScU5lREt3L1F3L0kxZ3RuZ1ArVmUzMkwwVm5walBsdzlp?=
 =?utf-8?B?Yk1aSE1qYmhUUzNoTzdtS0I2cjFzZWxERGc1amhna2ZRaXFpM1pOQkdZekYr?=
 =?utf-8?B?M1E9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 192237bd-7d4d-4057-ebf2-08dc1e8e2e55
X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB8718.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jan 2024 16:45:19.4411
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: VDD6rduNondL+j6BmlsfBOPHXB+CWMsUueaW++my07KaOlmKzOikXI6I4fo+8IejDViwgHG8xlbY8ihhgEJ4O/z4h017ZBgLNlKBT0MkHPk=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB5918
X-OriginatorOrg: intel.com

From: Robin Murphy <robin.murphy@arm.com>
Date: Fri, 26 Jan 2024 15:48:54 +0000

> On 26/01/2024 1:54 pm, Alexander Lobakin wrote:
>> From: Eric Dumazet <edumazet@google.com>
>>
>> Quite often, NIC devices do not need dma_sync operations on x86_64
>> at least.
>> Indeed, when dev_is_dma_coherent(dev) is true and
>> dev_use_swiotlb(dev) is false, iommu_dma_sync_single_for_cpu()
>> and friends do nothing.
>>
>> However, indirectly calling them when CONFIG_RETPOLINE=y consumes about
>> 10% of cycles on a cpu receiving packets from softirq at ~100Gbit rate.
>> Even if/when CONFIG_RETPOLINE is not set, there is a cost of about 3%.
>>
>> Add dev->skip_dma_sync boolean which is set during the device
>> initialization depending on the setup: dev_is_dma_coherent() for direct
>> DMA, !(sync_single_for_device || sync_single_for_cpu) or positive result
>> from the new callback, dma_map_ops::can_skip_sync for non-NULL DMA ops.
>> Then later, if/when swiotlb is used for the first time, the flag
>> is turned off, from swiotlb_tbl_map_single().
> 
> I think you could probably just promote the dma_uses_io_tlb flag from
> SWIOTLB_DYNAMIC to a general SWIOTLB thing to serve this purpose now.

Nice catch!

> 
> Similarly I don't think a new op is necessary now that we have
> dma_map_ops.flags. A simple static flag to indicate that sync may be> skipped under the same conditions as implied for dma-direct - i.e.
> dev_is_dma_coherent(dev) && !dev->dma_use_io_tlb - seems like it ought
> to suffice.

In my initial implementation, I used a new dma_map_ops flag, but then I
realized different DMA ops may require or not require syncing under
different conditions, not only dev_is_dma_coherent().
Or am I wrong and they would always be the same?

> 
> Thanks,
> Robin.

Thanks,
Olek