Received-SPF: pass (google.com: domain of linux-kernel+bounces-178480-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
From: Yang Yang <yang.yang@vivo.com>
To: Alasdair Kergon <agk@redhat.com>,
	Mike Snitzer <snitzer@kernel.org>,
	Mikulas Patocka <mpatocka@redhat.com>,
	dm-devel@lists.linux.dev,
	linux-kernel@vger.kernel.org
Cc: Yang Yang <yang.yang@vivo.com>
Subject: [PATCH 0/5] dm: empty flush optimization
Date: Tue, 14 May 2024 17:04:39 +0800
Message-Id: <20240514090445.2847-1-yang.yang@vivo.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?CwmoEReDlZaCydgVHFHAvVlpTKRd/5SUzqVZOKiGLKljZDPvKXvxXQLAk+Xu?=
 =?us-ascii?Q?Grpq1CdBh+mSQEa9b026CWBq7uExw21SmpRrbB1YU+esQnN7y7ZO13te5V7y?=
 =?us-ascii?Q?K//2/9THdcinqcZu8RCNEts+YGHBzKU34KiMNhYFUr0dhDg+2WDfjHv1VeE2?=
 =?us-ascii?Q?b3rLEn+gmb14pX5ZrLTgA6eJ2706PbahKGV9kPx6D1rhXm/t+rbX14t8mCew?=
 =?us-ascii?Q?d5sPAwc+Y4/FSqG5Qnd5dvZY37WJ//+E0WBnsxi9eZqZxVHbTBfBm0uTcbfm?=
 =?us-ascii?Q?TobU/uMz7OhJ/i/yX/kuRvFWXcS+SY0kx5oYkn6XFcSWksdvpFTpAEN70EGn?=
 =?us-ascii?Q?3bGlakA94O7t7NM/Ym16lZkm0imy2cQuJvbfNfCNZeNElBxq+9s8H6UPClkT?=
 =?us-ascii?Q?4ZON6Fhoj9YvvPuHl2FTyqcmMJA07oLK+1K+uChk6N7anjSZNxV2Hlr5TRnJ?=
 =?us-ascii?Q?HPY3Lic5DbvVkqgC84XK9BM4gVubmVrBGWZ5s3NbXhEnd10Xo/gUR/DDNuIn?=
 =?us-ascii?Q?MJPKyE7JR3MVfaerwOZ1Sf1LZiyoghGONcoLNOrDLvZY6uXrqXny7KxtjwH7?=
 =?us-ascii?Q?vNxYzJ+J35UJtd4Rjl+mzmFrV9UE8JJI0j4QWV9j3kGfYdecd3o5Oa1UVnEU?=
 =?us-ascii?Q?fT4AVNluDP7sbeI/r7uGkSqotdXCw3KJZuIO1J5SC2qTBNPC7QOF/YyQsAhI?=
 =?us-ascii?Q?HYWmhykI7AQuHYM24kphkHVpzwUaf8uG1Mc3Ig6hEBTzpOABCAR0af7GqVfb?=
 =?us-ascii?Q?nmK2quK0Rckfj4y4qGpTopq1FaHm5ylv1CjtHKbOXiYWi5M7ozzR7jxXPSjN?=
 =?us-ascii?Q?P+KW5oHnmzhFbseBn2RxvgQRpRC5QRRnTsHjxqSu//gENT7wpuk6JcPsAU7Q?=
 =?us-ascii?Q?GLW+D7GBz9Y54D2vB+0d9GajiQhahOflgIv1PGny6MNcux7i8Qut8rgKMaQc?=
 =?us-ascii?Q?l8PjDwwIJ1ex/PGetatOqV2l+Z/5OtWO4XSRC64ox6U/aUpZIJI5tTxdX1o+?=
 =?us-ascii?Q?j9jV/gbbPxKZ+Tpj8hxvq1BvSgsPBfuqP/mWPNr1UhZZE3P9ZadyNhBJ6qFd?=
 =?us-ascii?Q?fpnMmlqgfcNuYj2tztGNelZzh1zCLQbrjEnDZiXCmZNjfoDi81Hzu6+VFsOo?=
 =?us-ascii?Q?QqJCBz1MfIXUoyLdep+WmAcYER2qe4jBfpMXRHw0w4pFtAGkA/S7H1Vy94x0?=
 =?us-ascii?Q?tQH1abyhVFzB1yH8QyntdgdW+0X78GA5FiVtkIanUOquMaBjNM5etikMPZ5m?=
 =?us-ascii?Q?mNGz4hE4cQLH1c+uuJCh3boqBZym5gmJeLAllD1jiACNyFn0CTZMru9i4Q/f?=
 =?us-ascii?Q?l8A/cgdbrcUHlxfVTKHrfE0LwE1CmxsUB0aaU5LcxsuPopkC/nBdVV6sIjWe?=
 =?us-ascii?Q?4AfxW0AmowEJQixNMOV+53rqX1/6xUW++6utZgkDHm6PfLGM+jUWbPFshaQp?=
 =?us-ascii?Q?1fabfx7+X5VFCer5PGuDXLKbWHd1HQrGMNZHOe39uqJ23IUBHM2pFZgFjImo?=
 =?us-ascii?Q?hkSPGYWmcf+V1pJoTLrnzStVgyv5viJSacAtlLmOdS+C3DsfJCI4rwCbfL30?=
 =?us-ascii?Q?o029Vjg2hYfL7VFUIC82ssZ7YlcX4SfgXCjzK28b?=
X-OriginatorOrg: vivo.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d1326706-5b1b-4440-71fb-08dc73f50973
X-MS-Exchange-CrossTenant-AuthSource: SEYPR06MB6398.apcprd06.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 May 2024 09:05:44.4552
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 923e42dc-48d5-4cbe-b582-1a797a6412ed
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: E2NdhLQMuV9wt4G0+UcdqnWvGwCPEs3XRk3u1H/GtjSdF5ZkIjW9ISvK49JNXoXJP5F//UtDi+0oVFnoceHTuQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PUZPR06MB6102

__send_empty_flush() sends empty flush bios to every target in the
dm_table. However, if the num_targets exceeds the number of block
devices in the dm_table's device list, it could lead to multiple
invocations of __send_duplicate_bios() for the same block device.
Typically, a single thread sending numerous empty flush bios to one
block device is redundant, as these bios are likely to be merged by the
flush state machine. In scenarios where num_targets significantly
outweighs the number of block devices, such behavior may result in a
noteworthy decrease in performance.

This is a real-world scenario that we have encountered:
1) Call fallocate(file_fd, 0, 0, SZ_8G)
2) Call ioctl(file_fd, FS_IOC_FIEMAP, fiemap). In situations of severe
file system fragmentation, fiemap->fm_mapped_extents may exceed 1000.
3) Create a dm-linear device based on fiemap->fm_extents
4) Create a snapshot-cow device based on the dm-linear device 

Perf diff of fio test:
  fio --group_reporting --name=benchmark --filename=/dev/mapper/example \
      --ioengine=sync --invalidate=1 --numjobs=16 --rw=randwrite \
      --blocksize=4k --size=2G --time_based --runtime=30 --fdatasync=1

Scenario one:
  for i in {0..1023}; do
    echo $((8000*$i)) 8000 linear /dev/sda2 $((16384*$i))
  done | sudo dmsetup create example

  Before: bw=857KiB/
  After:  bw=30.8MiB/s    +3580%

Scenario two:
  for i in {0..1023}; do
    if [[ $i -gt 511 ]]; then
      echo $((8000*$i)) 8000 linear /dev/nvme0n1p6 $((16384*$i))
    else
      echo $((8000*$i)) 8000 linear /dev/sda2 $((16384*$i))
    fi
  done | sudo dmsetup create example

  Before: bw=1470KiB/
  After:  bw=33.9MiB/s    +2261%

Any comments are welcome!

V2:
-- Split into smaller pieces that are easier to review
-- Add flush_pass_around, suggested by Mikulas Patocka
-- Handling different target types separately

Yang Yang (5):
  dm: introduce flush_pass_around flag
  dm: add __send_empty_flush_bios() helper
  dm: support retrieving struct dm_target from struct dm_dev
  dm: Avoid sending redundant empty flush bios to the same block device
  dm linear: enable flush optimization function

 drivers/md/dm-core.h          |  3 +++
 drivers/md/dm-ioctl.c         |  4 ++++
 drivers/md/dm-linear.c        |  1 +
 drivers/md/dm-table.c         | 39 +++++++++++++++++++++++++++++++++++
 drivers/md/dm.c               | 37 +++++++++++++++++++++++++--------
 include/linux/device-mapper.h |  8 +++++++
 6 files changed, 83 insertions(+), 9 deletions(-)

-- 
2.34.1