From: Namjae Jeon <[email protected]>
This patch series introduces 2 new ioctls for ext4.
Truncate_block_range ioctl truncates blocks from source file.
Transfer_block_range ioctl transfers data blocks from source file
and append them at the end of destination file.
Ioctl1: EXT4_IOC_TRUNCATE_BLOCK_RANGE:
This ioctl truncates a range of data blocks from file.
It is useful to remove easily and quickly the garbage data
at the middle of file.
e.g. we have a movie file and there is long advertisement in movie file.
user want to remove only advertisement range.
1) Movie file (8GB), There is the adverisement of 500MB size.
____________________________________________________________________
| | | |
| a) range | b) Advertisement | c) range |
| | (unneeded data) | |
|_____________________|___________________|_________________________|
2) Currently if user want to remove portion b), the conventional way
would be to copy a) and c) (7.5GB) to new file by reading data from
original file and writing to new file, followed up by delete original
file and rename new file. It will take long time.
When we measure time, it takes around 3 minutes.
3) If we use EXT4_IOC_TRUNCATE_BLOCK_RANGE, we can have garbage data removed
in less than a second. Also, no need to perform deletion and rename.
_______________________________________________
| | |
| a) range | c) range |
| | |
|_____________________|________________________|
#define EXT4_IOC_TRUNCATE_BLOCK_RANGE _IOW('f', 18, struct truncate_range)
struct truncate_range {
__u32 start_block;
__u32 length;
};
example =>
Originally the file "abc" has the below extent tree:
debugfs: ex abc
Level Entries Logical Physical Length Flags
0/ 0 1/ 3 0 - 4 33615 - 33619 5
0/ 0 2/ 3 5 - 9 33855 - 33859 5
0/ 0 3/ 3 10 - 14 69657 - 69661 5
ls -lh abc
-rw-r--r-- 1 root 0 60.0K Jan 1 00:01 abc
du -h abc
60.0K abc
e4_truncate_block_range abc 2 10
Return:
: Success
After executing truncate_block_range ioctl, the extent tree:
ex abc
Level Entries Logical Physical Length Flags
0/ 0 1/ 2 0 - 1 33615 - 33616 2
0/ 0 2/ 2 2 - 4 69659 - 69661 3
ls -lh abc
-rw-r--r-- 1 root 0 20.0K Jan 1 00:08 abc
du -h abc
20.0K abc
This ioctl works in 2 parts:
1) remove _only_ data blocks that resides within specified range.
If the entire range is a hole than nothing is removed.
2) update file's logical block offsets ranging from block number
"start_block + length" to last logical block of file such that
lblk_number = lblk_number - length;
This is done by updating starting block of all the extents that
resides within the range.
If "start_block + length" is already equal to the last block of file
than no block is updated. This case is similar to convential truncate.
In the above example:
The data blocks ranging from [2 - 11] have been removed
and the logical offsets of the file beyond block number 12 till last block
of file are updated by subtracting length from each of logical numbers.
This gives a contiguous logical space to the file.
Also, the logical size and disksize of the file are updated accordingly.
Ioctl2: EXT4_IOC_TRANSFER_BLOCK_RANGE:
This ioctl transfers a range of data blocks from source file and append
them at the end of the destination file.
This is not actual data transfer but only metadata is moved.
____________________________________________________________________
| | | |
| a) range | b) range | c) range |
| | | |
|_____________________|___________________|_________________________|
If user does not want b) in the orig file but wants to make a new file
comprising only b) OR wants b) at the end of an already existing file,
the conventional way of doing it would be to:
1) Copy b) to new file
2) Copy c) to temp file
3) Truncate orig file to a)
4) Copy c) from temp file to the end of orig file.
5) Delete temp file.
After this operations =>
orig_file:
__________________________________________
| | |
| a) range | c) range |
| | |
|_____________________|___________________|
new_file:
_______________________
| |
| b) range |
| |
|_____________________|
Again, this operation would take a long time (depending on the sizes of range)
if done using conventional way while using transfer_block_range ioctl reduces
the time within a second.
#define EXT4_IOC_TRANSFER_BLOCK_RANGE _IOW('f', 19, struct transfer_range)
struct transfer_range {
__u32 dest_fd;
__u32 start_block;
__u32 length;
};
example=>
debugfs: ex source
Level Entries Logical Physical Length Flags
0/ 1 1/ 1 0 - 24 32809 25
1/ 1 1/ 5 0 - 4 4071 - 4075 5
1/ 1 2/ 5 5 - 9 4081 - 4085 5
1/ 1 3/ 5 10 - 14 4091 - 4095 5
1/ 1 4/ 5 15 - 19 4101 - 4105 5
1/ 1 5/ 5 20 - 24 4151 - 4155 5
debugfs: ex dest
Level Entries Logical Physical Length Flags
0/ 0 1/ 3 0 - 4 32825 - 32829 5
0/ 0 2/ 3 5 - 9 33545 - 33549 5
0/ 0 3/ 3 10 - 14 33615 - 33619 5
ls -lh source
-rw-r--r-- 1 root 0 100.0K Jan 1 00:01 source
ls -lh dest
-rw-r--r-- 1 root 0 60.0K Jan 1 00:01 dest
du -h source
104.0K source
du -h dest
60.0K dest
e4_transfer_block_range source dest 2 10
Return:
: Success
debugfs: ex source
Level Entries Logical Physical Length Flags
0/ 1 1/ 1 0 - 24 32809 25
1/ 1 1/ 4 0 - 1 4071 - 4072 2
1/ 1 2/ 4 12 - 14 4093 - 4095 3
1/ 1 3/ 4 15 - 19 4101 - 4105 5
1/ 1 4/ 4 20 - 24 4151 - 4155 5
debugfs: ex dest
Level Entries Logical Physical Length Flags
0/ 1 1/ 1 0 - 24 32835 25
1/ 1 1/ 6 0 - 4 32825 - 32829 5
1/ 1 2/ 6 5 - 9 33545 - 33549 5
1/ 1 3/ 6 10 - 14 33615 - 33619 5
1/ 1 4/ 6 15 - 17 4073 - 4075 3
1/ 1 5/ 6 18 - 22 4081 - 4085 5
1/ 1 6/ 6 23 - 24 4091 - 4092 2
ls -lh source
-rw-r--r-- 1 root 0 100.0K Jan 1 00:04 source
ls -lh dest
-rw-r--r-- 1 root 0 100.0K Jan 1 00:04 dest
du -h source
64.0K source
du -h dest
104.0K dest
The data blocks lying between [start_block to start_block + length) are appended
contiguously at the end of destination file.
The block transfer leaves a hole in the source file.
If any hole is encountered in the range, it is ommited.
This ioctl does not change the logical size of the source file hence
leaves a hole in place of transfered range.
If user want contiguous logical space for source file,
it can truncate the hole by calling truncate_range_ioctl for source file.
Example for above "source" file:
e4_truncate_block_range source 2 10
Return:
: Success
debugfs: ex source
Level Entries Logical Physical Length Flags
0/ 1 1/ 1 0 - 14 32809 15
1/ 1 1/ 4 0 - 1 4071 - 4072 2
1/ 1 2/ 4 2 - 4 4093 - 4095 3
1/ 1 3/ 4 5 - 9 4101 - 4105 5
1/ 1 4/ 4 10 - 14 4151 - 4155 5
Namjae Jeon (3):
ext4: Add EXT4_IOC_TRUNCATE_BLOCK_RANGE ioctl
ext4: make mext_next_extent non static and move get_ext_path
ext4: Add EXT4_IOC_TRANSFER_BLOCK_RANGE ioctl
--
1.7.9.5
Hi.
I attached testcases for these ioctls.
Thanks.
2013/6/23 Namjae Jeon <[email protected]>:
> From: Namjae Jeon <[email protected]>
>
> This patch series introduces 2 new ioctls for ext4.
>
> Truncate_block_range ioctl truncates blocks from source file.
> Transfer_block_range ioctl transfers data blocks from source file
> and append them at the end of destination file.
>
> Ioctl1: EXT4_IOC_TRUNCATE_BLOCK_RANGE:
> This ioctl truncates a range of data blocks from file.
> It is useful to remove easily and quickly the garbage data
> at the middle of file.
>
> e.g. we have a movie file and there is long advertisement in movie file.
> user want to remove only advertisement range.
>
> 1) Movie file (8GB), There is the adverisement of 500MB size.
> ____________________________________________________________________
> | | | |
> | a) range | b) Advertisement | c) range |
> | | (unneeded data) | |
> |_____________________|___________________|_________________________|
>
> 2) Currently if user want to remove portion b), the conventional way
> would be to copy a) and c) (7.5GB) to new file by reading data from
> original file and writing to new file, followed up by delete original
> file and rename new file. It will take long time.
> When we measure time, it takes around 3 minutes.
>
> 3) If we use EXT4_IOC_TRUNCATE_BLOCK_RANGE, we can have garbage data removed
> in less than a second. Also, no need to perform deletion and rename.
> _______________________________________________
> | | |
> | a) range | c) range |
> | | |
> |_____________________|________________________|
>
>
> #define EXT4_IOC_TRUNCATE_BLOCK_RANGE _IOW('f', 18, struct truncate_range)
> struct truncate_range {
> __u32 start_block;
> __u32 length;
> };
>
> example =>
> Originally the file "abc" has the below extent tree:
> debugfs: ex abc
> Level Entries Logical Physical Length Flags
> 0/ 0 1/ 3 0 - 4 33615 - 33619 5
> 0/ 0 2/ 3 5 - 9 33855 - 33859 5
> 0/ 0 3/ 3 10 - 14 69657 - 69661 5
>
> ls -lh abc
> -rw-r--r-- 1 root 0 60.0K Jan 1 00:01 abc
>
> du -h abc
> 60.0K abc
>
> e4_truncate_block_range abc 2 10
> Return:
> : Success
>
> After executing truncate_block_range ioctl, the extent tree:
> ex abc
> Level Entries Logical Physical Length Flags
> 0/ 0 1/ 2 0 - 1 33615 - 33616 2
> 0/ 0 2/ 2 2 - 4 69659 - 69661 3
>
> ls -lh abc
> -rw-r--r-- 1 root 0 20.0K Jan 1 00:08 abc
>
> du -h abc
> 20.0K abc
>
> This ioctl works in 2 parts:
> 1) remove _only_ data blocks that resides within specified range.
> If the entire range is a hole than nothing is removed.
>
> 2) update file's logical block offsets ranging from block number
> "start_block + length" to last logical block of file such that
> lblk_number = lblk_number - length;
> This is done by updating starting block of all the extents that
> resides within the range.
>
> If "start_block + length" is already equal to the last block of file
> than no block is updated. This case is similar to convential truncate.
>
> In the above example:
> The data blocks ranging from [2 - 11] have been removed
> and the logical offsets of the file beyond block number 12 till last block
> of file are updated by subtracting length from each of logical numbers.
> This gives a contiguous logical space to the file.
> Also, the logical size and disksize of the file are updated accordingly.
>
> Ioctl2: EXT4_IOC_TRANSFER_BLOCK_RANGE:
> This ioctl transfers a range of data blocks from source file and append
> them at the end of the destination file.
> This is not actual data transfer but only metadata is moved.
>
> ____________________________________________________________________
> | | | |
> | a) range | b) range | c) range |
> | | | |
> |_____________________|___________________|_________________________|
>
> If user does not want b) in the orig file but wants to make a new file
> comprising only b) OR wants b) at the end of an already existing file,
> the conventional way of doing it would be to:
> 1) Copy b) to new file
> 2) Copy c) to temp file
> 3) Truncate orig file to a)
> 4) Copy c) from temp file to the end of orig file.
> 5) Delete temp file.
>
> After this operations =>
> orig_file:
> __________________________________________
> | | |
> | a) range | c) range |
> | | |
> |_____________________|___________________|
>
> new_file:
> _______________________
> | |
> | b) range |
> | |
> |_____________________|
>
> Again, this operation would take a long time (depending on the sizes of range)
> if done using conventional way while using transfer_block_range ioctl reduces
> the time within a second.
>
> #define EXT4_IOC_TRANSFER_BLOCK_RANGE _IOW('f', 19, struct transfer_range)
> struct transfer_range {
> __u32 dest_fd;
> __u32 start_block;
> __u32 length;
> };
>
> example=>
> debugfs: ex source
> Level Entries Logical Physical Length Flags
> 0/ 1 1/ 1 0 - 24 32809 25
> 1/ 1 1/ 5 0 - 4 4071 - 4075 5
> 1/ 1 2/ 5 5 - 9 4081 - 4085 5
> 1/ 1 3/ 5 10 - 14 4091 - 4095 5
> 1/ 1 4/ 5 15 - 19 4101 - 4105 5
> 1/ 1 5/ 5 20 - 24 4151 - 4155 5
>
> debugfs: ex dest
> Level Entries Logical Physical Length Flags
> 0/ 0 1/ 3 0 - 4 32825 - 32829 5
> 0/ 0 2/ 3 5 - 9 33545 - 33549 5
> 0/ 0 3/ 3 10 - 14 33615 - 33619 5
>
> ls -lh source
> -rw-r--r-- 1 root 0 100.0K Jan 1 00:01 source
> ls -lh dest
> -rw-r--r-- 1 root 0 60.0K Jan 1 00:01 dest
>
> du -h source
> 104.0K source
> du -h dest
> 60.0K dest
>
> e4_transfer_block_range source dest 2 10
> Return:
> : Success
>
> debugfs: ex source
> Level Entries Logical Physical Length Flags
> 0/ 1 1/ 1 0 - 24 32809 25
> 1/ 1 1/ 4 0 - 1 4071 - 4072 2
> 1/ 1 2/ 4 12 - 14 4093 - 4095 3
> 1/ 1 3/ 4 15 - 19 4101 - 4105 5
> 1/ 1 4/ 4 20 - 24 4151 - 4155 5
> debugfs: ex dest
> Level Entries Logical Physical Length Flags
> 0/ 1 1/ 1 0 - 24 32835 25
> 1/ 1 1/ 6 0 - 4 32825 - 32829 5
> 1/ 1 2/ 6 5 - 9 33545 - 33549 5
> 1/ 1 3/ 6 10 - 14 33615 - 33619 5
> 1/ 1 4/ 6 15 - 17 4073 - 4075 3
> 1/ 1 5/ 6 18 - 22 4081 - 4085 5
> 1/ 1 6/ 6 23 - 24 4091 - 4092 2
>
> ls -lh source
> -rw-r--r-- 1 root 0 100.0K Jan 1 00:04 source
> ls -lh dest
> -rw-r--r-- 1 root 0 100.0K Jan 1 00:04 dest
>
> du -h source
> 64.0K source
> du -h dest
> 104.0K dest
>
> The data blocks lying between [start_block to start_block + length) are appended
> contiguously at the end of destination file.
> The block transfer leaves a hole in the source file.
> If any hole is encountered in the range, it is ommited.
>
> This ioctl does not change the logical size of the source file hence
> leaves a hole in place of transfered range.
> If user want contiguous logical space for source file,
> it can truncate the hole by calling truncate_range_ioctl for source file.
>
> Example for above "source" file:
> e4_truncate_block_range source 2 10
> Return:
> : Success
> debugfs: ex source
> Level Entries Logical Physical Length Flags
> 0/ 1 1/ 1 0 - 14 32809 15
> 1/ 1 1/ 4 0 - 1 4071 - 4072 2
> 1/ 1 2/ 4 2 - 4 4093 - 4095 3
> 1/ 1 3/ 4 5 - 9 4101 - 4105 5
> 1/ 1 4/ 4 10 - 14 4151 - 4155 5
>
> Namjae Jeon (3):
> ext4: Add EXT4_IOC_TRUNCATE_BLOCK_RANGE ioctl
> ext4: make mext_next_extent non static and move get_ext_path
> ext4: Add EXT4_IOC_TRANSFER_BLOCK_RANGE ioctl
>
> --
> 1.7.9.5
>
On 2013-06-23, at 0:07, Namjae Jeon <[email protected]> wrote:
> From: Namjae Jeon <[email protected]>
>
> This patch series introduces 2 new ioctls for ext4.
>
> Truncate_block_range ioctl truncates blocks from source file.
How is this different from fallocate(FALLOC_FL_PUNCH_HOLE)? That is already in existing kernels, and portable across multiple filesystems.
> Transfer_block_range ioctl transfers data blocks from source file
> and append them at the end of destination file.
There is already a similar ioctl for defragmenting files. Is it possible to use that, or does it have different semantics?
> Ioctl1: EXT4_IOC_TRUNCATE_BLOCK_RANGE:
> This ioctl truncates a range of data blocks from file.
> It is useful to remove easily and quickly the garbage data
> at the middle of file.
>
> e.g. we have a movie file and there is long advertisement in movie file.
> user want to remove only advertisement range.
While this works in theory, there is very little chance that the movie data will align exactly to filesystem block boundaries.
Cheers, Andreas
> 1) Movie file (8GB), There is the adverisement of 500MB size.
> ____________________________________________________________________
> | | | |
> | a) range | b) Advertisement | c) range |
> | | (unneeded data) | |
> |_____________________|___________________|_________________________|
>
> 2) Currently if user want to remove portion b), the conventional way
> would be to copy a) and c) (7.5GB) to new file by reading data from
> original file and writing to new file, followed up by delete original
> file and rename new file. It will take long time.
> When we measure time, it takes around 3 minutes.
>
> 3) If we use EXT4_IOC_TRUNCATE_BLOCK_RANGE, we can have garbage data removed
> in less than a second. Also, no need to perform deletion and rename.
> _______________________________________________
> | | |
> | a) range | c) range |
> | | |
> |_____________________|________________________|
>
>
> #define EXT4_IOC_TRUNCATE_BLOCK_RANGE _IOW('f', 18, struct truncate_range)
> struct truncate_range {
> __u32 start_block;
> __u32 length;
> };
>
> example =>
> Originally the file "abc" has the below extent tree:
> debugfs: ex abc
> Level Entries Logical Physical Length Flags
> 0/ 0 1/ 3 0 - 4 33615 - 33619 5
> 0/ 0 2/ 3 5 - 9 33855 - 33859 5
> 0/ 0 3/ 3 10 - 14 69657 - 69661 5
>
> ls -lh abc
> -rw-r--r-- 1 root 0 60.0K Jan 1 00:01 abc
>
> du -h abc
> 60.0K abc
>
> e4_truncate_block_range abc 2 10
> Return:
> : Success
>
> After executing truncate_block_range ioctl, the extent tree:
> ex abc
> Level Entries Logical Physical Length Flags
> 0/ 0 1/ 2 0 - 1 33615 - 33616 2
> 0/ 0 2/ 2 2 - 4 69659 - 69661 3
>
> ls -lh abc
> -rw-r--r-- 1 root 0 20.0K Jan 1 00:08 abc
>
> du -h abc
> 20.0K abc
>
> This ioctl works in 2 parts:
> 1) remove _only_ data blocks that resides within specified range.
> If the entire range is a hole than nothing is removed.
>
> 2) update file's logical block offsets ranging from block number
> "start_block + length" to last logical block of file such that
> lblk_number = lblk_number - length;
> This is done by updating starting block of all the extents that
> resides within the range.
>
> If "start_block + length" is already equal to the last block of file
> than no block is updated. This case is similar to convential truncate.
>
> In the above example:
> The data blocks ranging from [2 - 11] have been removed
> and the logical offsets of the file beyond block number 12 till last block
> of file are updated by subtracting length from each of logical numbers.
> This gives a contiguous logical space to the file.
> Also, the logical size and disksize of the file are updated accordingly.
>
> Ioctl2: EXT4_IOC_TRANSFER_BLOCK_RANGE:
> This ioctl transfers a range of data blocks from source file and append
> them at the end of the destination file.
> This is not actual data transfer but only metadata is moved.
>
> ____________________________________________________________________
> | | | |
> | a) range | b) range | c) range |
> | | | |
> |_____________________|___________________|_________________________|
>
> If user does not want b) in the orig file but wants to make a new file
> comprising only b) OR wants b) at the end of an already existing file,
> the conventional way of doing it would be to:
> 1) Copy b) to new file
> 2) Copy c) to temp file
> 3) Truncate orig file to a)
> 4) Copy c) from temp file to the end of orig file.
> 5) Delete temp file.
>
> After this operations =>
> orig_file:
> __________________________________________
> | | |
> | a) range | c) range |
> | | |
> |_____________________|___________________|
>
> new_file:
> _______________________
> | |
> | b) range |
> | |
> |_____________________|
>
> Again, this operation would take a long time (depending on the sizes of range)
> if done using conventional way while using transfer_block_range ioctl reduces
> the time within a second.
>
> #define EXT4_IOC_TRANSFER_BLOCK_RANGE _IOW('f', 19, struct transfer_range)
> struct transfer_range {
> __u32 dest_fd;
> __u32 start_block;
> __u32 length;
> };
>
> example=>
> debugfs: ex source
> Level Entries Logical Physical Length Flags
> 0/ 1 1/ 1 0 - 24 32809 25
> 1/ 1 1/ 5 0 - 4 4071 - 4075 5
> 1/ 1 2/ 5 5 - 9 4081 - 4085 5
> 1/ 1 3/ 5 10 - 14 4091 - 4095 5
> 1/ 1 4/ 5 15 - 19 4101 - 4105 5
> 1/ 1 5/ 5 20 - 24 4151 - 4155 5
>
> debugfs: ex dest
> Level Entries Logical Physical Length Flags
> 0/ 0 1/ 3 0 - 4 32825 - 32829 5
> 0/ 0 2/ 3 5 - 9 33545 - 33549 5
> 0/ 0 3/ 3 10 - 14 33615 - 33619 5
>
> ls -lh source
> -rw-r--r-- 1 root 0 100.0K Jan 1 00:01 source
> ls -lh dest
> -rw-r--r-- 1 root 0 60.0K Jan 1 00:01 dest
>
> du -h source
> 104.0K source
> du -h dest
> 60.0K dest
>
> e4_transfer_block_range source dest 2 10
> Return:
> : Success
>
> debugfs: ex source
> Level Entries Logical Physical Length Flags
> 0/ 1 1/ 1 0 - 24 32809 25
> 1/ 1 1/ 4 0 - 1 4071 - 4072 2
> 1/ 1 2/ 4 12 - 14 4093 - 4095 3
> 1/ 1 3/ 4 15 - 19 4101 - 4105 5
> 1/ 1 4/ 4 20 - 24 4151 - 4155 5
> debugfs: ex dest
> Level Entries Logical Physical Length Flags
> 0/ 1 1/ 1 0 - 24 32835 25
> 1/ 1 1/ 6 0 - 4 32825 - 32829 5
> 1/ 1 2/ 6 5 - 9 33545 - 33549 5
> 1/ 1 3/ 6 10 - 14 33615 - 33619 5
> 1/ 1 4/ 6 15 - 17 4073 - 4075 3
> 1/ 1 5/ 6 18 - 22 4081 - 4085 5
> 1/ 1 6/ 6 23 - 24 4091 - 4092 2
>
> ls -lh source
> -rw-r--r-- 1 root 0 100.0K Jan 1 00:04 source
> ls -lh dest
> -rw-r--r-- 1 root 0 100.0K Jan 1 00:04 dest
>
> du -h source
> 64.0K source
> du -h dest
> 104.0K dest
>
> The data blocks lying between [start_block to start_block + length) are appended
> contiguously at the end of destination file.
> The block transfer leaves a hole in the source file.
> If any hole is encountered in the range, it is ommited.
>
> This ioctl does not change the logical size of the source file hence
> leaves a hole in place of transfered range.
> If user want contiguous logical space for source file,
> it can truncate the hole by calling truncate_range_ioctl for source file.
>
> Example for above "source" file:
> e4_truncate_block_range source 2 10
> Return:
> : Success
> debugfs: ex source
> Level Entries Logical Physical Length Flags
> 0/ 1 1/ 1 0 - 14 32809 15
> 1/ 1 1/ 4 0 - 1 4071 - 4072 2
> 1/ 1 2/ 4 2 - 4 4093 - 4095 3
> 1/ 1 3/ 4 5 - 9 4101 - 4105 5
> 1/ 1 4/ 4 10 - 14 4151 - 4155 5
>
> Namjae Jeon (3):
> ext4: Add EXT4_IOC_TRUNCATE_BLOCK_RANGE ioctl
> ext4: make mext_next_extent non static and move get_ext_path
> ext4: Add EXT4_IOC_TRANSFER_BLOCK_RANGE ioctl
>
> --
> 1.7.9.5
>
On Jun 23, 2013, at 12:01 PM, Andreas Dilger <[email protected]> wrote:
> On 2013-06-23, at 0:07, Namjae Jeon <[email protected]> wrote:
>
>> From: Namjae Jeon <[email protected]>
>>
>> This patch series introduces 2 new ioctls for ext4.
>>
>> Truncate_block_range ioctl truncates blocks from source file.
>
> How is this different from fallocate(FALLOC_FL_PUNCH_HOLE)? That is already in existing kernels, and portable across multiple filesystems.
>
Same question. Punch hole should do this already...
>> Transfer_block_range ioctl transfers data blocks from source file
>> and append them at the end of destination file.
>
> There is already a similar ioctl for defragmenting files. Is it possible to use that, or does it have different semantics?
>
>> Ioctl1: EXT4_IOC_TRUNCATE_BLOCK_RANGE:
>> This ioctl truncates a range of data blocks from file.
>> It is useful to remove easily and quickly the garbage data
>> at the middle of file.
>>
>> e.g. we have a movie file and there is long advertisement in movie file.
>> user want to remove only advertisement range.
>
> While this works in theory, there is very little chance that the movie data will align exactly to filesystem block boundaries.
>
Or more importantly on compression codec boundaries. Wouldn't this look like corruption at playback time?
Eric
> Cheers, Andreas
>
>> 1) Movie file (8GB), There is the adverisement of 500MB size.
>> ____________________________________________________________________
>> | | | |
>> | a) range | b) Advertisement | c) range |
>> | | (unneeded data) | |
>> |_____________________|___________________|_________________________|
>>
>> 2) Currently if user want to remove portion b), the conventional way
>> would be to copy a) and c) (7.5GB) to new file by reading data from
>> original file and writing to new file, followed up by delete original
>> file and rename new file. It will take long time.
>> When we measure time, it takes around 3 minutes.
>>
>> 3) If we use EXT4_IOC_TRUNCATE_BLOCK_RANGE, we can have garbage data removed
>> in less than a second. Also, no need to perform deletion and rename.
>> _______________________________________________
>> | | |
>> | a) range | c) range |
>> | | |
>> |_____________________|________________________|
>>
>>
>> #define EXT4_IOC_TRUNCATE_BLOCK_RANGE _IOW('f', 18, struct truncate_range)
>> struct truncate_range {
>> __u32 start_block;
>> __u32 length;
>> };
>>
>> example =>
>> Originally the file "abc" has the below extent tree:
>> debugfs: ex abc
>> Level Entries Logical Physical Length Flags
>> 0/ 0 1/ 3 0 - 4 33615 - 33619 5
>> 0/ 0 2/ 3 5 - 9 33855 - 33859 5
>> 0/ 0 3/ 3 10 - 14 69657 - 69661 5
>>
>> ls -lh abc
>> -rw-r--r-- 1 root 0 60.0K Jan 1 00:01 abc
>>
>> du -h abc
>> 60.0K abc
>>
>> e4_truncate_block_range abc 2 10
>> Return:
>> : Success
>>
>> After executing truncate_block_range ioctl, the extent tree:
>> ex abc
>> Level Entries Logical Physical Length Flags
>> 0/ 0 1/ 2 0 - 1 33615 - 33616 2
>> 0/ 0 2/ 2 2 - 4 69659 - 69661 3
>>
>> ls -lh abc
>> -rw-r--r-- 1 root 0 20.0K Jan 1 00:08 abc
>>
>> du -h abc
>> 20.0K abc
>>
>> This ioctl works in 2 parts:
>> 1) remove _only_ data blocks that resides within specified range.
>> If the entire range is a hole than nothing is removed.
>>
>> 2) update file's logical block offsets ranging from block number
>> "start_block + length" to last logical block of file such that
>> lblk_number = lblk_number - length;
>> This is done by updating starting block of all the extents that
>> resides within the range.
>>
>> If "start_block + length" is already equal to the last block of file
>> than no block is updated. This case is similar to convential truncate.
>>
>> In the above example:
>> The data blocks ranging from [2 - 11] have been removed
>> and the logical offsets of the file beyond block number 12 till last block
>> of file are updated by subtracting length from each of logical numbers.
>> This gives a contiguous logical space to the file.
>> Also, the logical size and disksize of the file are updated accordingly.
>>
>> Ioctl2: EXT4_IOC_TRANSFER_BLOCK_RANGE:
>> This ioctl transfers a range of data blocks from source file and append
>> them at the end of the destination file.
>> This is not actual data transfer but only metadata is moved.
>>
>> ____________________________________________________________________
>> | | | |
>> | a) range | b) range | c) range |
>> | | | |
>> |_____________________|___________________|_________________________|
>>
>> If user does not want b) in the orig file but wants to make a new file
>> comprising only b) OR wants b) at the end of an already existing file,
>> the conventional way of doing it would be to:
>> 1) Copy b) to new file
>> 2) Copy c) to temp file
>> 3) Truncate orig file to a)
>> 4) Copy c) from temp file to the end of orig file.
>> 5) Delete temp file.
>>
>> After this operations =>
>> orig_file:
>> __________________________________________
>> | | |
>> | a) range | c) range |
>> | | |
>> |_____________________|___________________|
>>
>> new_file:
>> _______________________
>> | |
>> | b) range |
>> | |
>> |_____________________|
>>
>> Again, this operation would take a long time (depending on the sizes of range)
>> if done using conventional way while using transfer_block_range ioctl reduces
>> the time within a second.
>>
>> #define EXT4_IOC_TRANSFER_BLOCK_RANGE _IOW('f', 19, struct transfer_range)
>> struct transfer_range {
>> __u32 dest_fd;
>> __u32 start_block;
>> __u32 length;
>> };
>>
>> example=>
>> debugfs: ex source
>> Level Entries Logical Physical Length Flags
>> 0/ 1 1/ 1 0 - 24 32809 25
>> 1/ 1 1/ 5 0 - 4 4071 - 4075 5
>> 1/ 1 2/ 5 5 - 9 4081 - 4085 5
>> 1/ 1 3/ 5 10 - 14 4091 - 4095 5
>> 1/ 1 4/ 5 15 - 19 4101 - 4105 5
>> 1/ 1 5/ 5 20 - 24 4151 - 4155 5
>>
>> debugfs: ex dest
>> Level Entries Logical Physical Length Flags
>> 0/ 0 1/ 3 0 - 4 32825 - 32829 5
>> 0/ 0 2/ 3 5 - 9 33545 - 33549 5
>> 0/ 0 3/ 3 10 - 14 33615 - 33619 5
>>
>> ls -lh source
>> -rw-r--r-- 1 root 0 100.0K Jan 1 00:01 source
>> ls -lh dest
>> -rw-r--r-- 1 root 0 60.0K Jan 1 00:01 dest
>>
>> du -h source
>> 104.0K source
>> du -h dest
>> 60.0K dest
>>
>> e4_transfer_block_range source dest 2 10
>> Return:
>> : Success
>>
>> debugfs: ex source
>> Level Entries Logical Physical Length Flags
>> 0/ 1 1/ 1 0 - 24 32809 25
>> 1/ 1 1/ 4 0 - 1 4071 - 4072 2
>> 1/ 1 2/ 4 12 - 14 4093 - 4095 3
>> 1/ 1 3/ 4 15 - 19 4101 - 4105 5
>> 1/ 1 4/ 4 20 - 24 4151 - 4155 5
>> debugfs: ex dest
>> Level Entries Logical Physical Length Flags
>> 0/ 1 1/ 1 0 - 24 32835 25
>> 1/ 1 1/ 6 0 - 4 32825 - 32829 5
>> 1/ 1 2/ 6 5 - 9 33545 - 33549 5
>> 1/ 1 3/ 6 10 - 14 33615 - 33619 5
>> 1/ 1 4/ 6 15 - 17 4073 - 4075 3
>> 1/ 1 5/ 6 18 - 22 4081 - 4085 5
>> 1/ 1 6/ 6 23 - 24 4091 - 4092 2
>>
>> ls -lh source
>> -rw-r--r-- 1 root 0 100.0K Jan 1 00:04 source
>> ls -lh dest
>> -rw-r--r-- 1 root 0 100.0K Jan 1 00:04 dest
>>
>> du -h source
>> 64.0K source
>> du -h dest
>> 104.0K dest
>>
>> The data blocks lying between [start_block to start_block + length) are appended
>> contiguously at the end of destination file.
>> The block transfer leaves a hole in the source file.
>> If any hole is encountered in the range, it is ommited.
>>
>> This ioctl does not change the logical size of the source file hence
>> leaves a hole in place of transfered range.
>> If user want contiguous logical space for source file,
>> it can truncate the hole by calling truncate_range_ioctl for source file.
>>
>> Example for above "source" file:
>> e4_truncate_block_range source 2 10
>> Return:
>> : Success
>> debugfs: ex source
>> Level Entries Logical Physical Length Flags
>> 0/ 1 1/ 1 0 - 14 32809 15
>> 1/ 1 1/ 4 0 - 1 4071 - 4072 2
>> 1/ 1 2/ 4 2 - 4 4093 - 4095 3
>> 1/ 1 3/ 4 5 - 9 4101 - 4105 5
>> 1/ 1 4/ 4 10 - 14 4151 - 4155 5
>>
>> Namjae Jeon (3):
>> ext4: Add EXT4_IOC_TRUNCATE_BLOCK_RANGE ioctl
>> ext4: make mext_next_extent non static and move get_ext_path
>> ext4: Add EXT4_IOC_TRANSFER_BLOCK_RANGE ioctl
>>
>> --
>> 1.7.9.5
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sun, Jun 23, 2013 at 08:32:32PM -0400, Eric Sandeen wrote:
> On Jun 23, 2013, at 12:01 PM, Andreas Dilger <[email protected]> wrote:
>
> > On 2013-06-23, at 0:07, Namjae Jeon <[email protected]> wrote:
> >
> >> From: Namjae Jeon <[email protected]>
> >>
> >> This patch series introduces 2 new ioctls for ext4.
> >>
> >> Truncate_block_range ioctl truncates blocks from source file.
> >
> > How is this different from fallocate(FALLOC_FL_PUNCH_HOLE)? That is already in existing kernels, and portable across multiple filesystems.
> >
> Same question. Punch hole should do this already...
Hole punch doesn't change the offsets of subsequent extents in the
file - it leaves a hole. This doesn't leave a hole at all - all the
extents above the range are shifted down to offset where the extents
being punched out started.
> >> Transfer_block_range ioctl transfers data blocks from source file
> >> and append them at the end of destination file.
> >
> > There is already a similar ioctl for defragmenting files. Is it possible to use that, or does it have different semantics?
> >
> >> Ioctl1: EXT4_IOC_TRUNCATE_BLOCK_RANGE:
> >> This ioctl truncates a range of data blocks from file.
> >> It is useful to remove easily and quickly the garbage data
> >> at the middle of file.
> >>
> >> e.g. we have a movie file and there is long advertisement in movie file.
> >> user want to remove only advertisement range.
> >
> > While this works in theory, there is very little chance that the movie data will align exactly to filesystem block boundaries.
> >
> Or more importantly on compression codec boundaries. Wouldn't this look like corruption at playback time?
Not necessarily. Video codecs are encapsulated in a container that
can be used to link key frames together so you can do this sort of
manipulation of the file contents and just change an
offset-to-next-keyframe value in the container and it all just
works. Non linear editting (NLE) software has been doing this
manually for 15 years by copying data around - this just optimises
the operation by manipulating the extent mapping rather than needing
to physically copy the data.
FWIW, I've heard persistent rumors going back several years of
various DVR companies shipping equivalent ioctl-based functionality
for XFS filesystems to both insert and remove chunks in video
streams, but I've never been able to find the code for it anywhere.
Hence, at minimum, this should be a fallocate() operation, not a ext4
specific ioctl as it is relatively trivial to implement on most
extent based filesystems.
However, My conditions for merging such functionality into fallocate
are:
1. it needs xfs_io support [to provide]
2. comprehensive xfstests coverage, similar to the current
hole punch coverage we have.
> >> #define EXT4_IOC_TRUNCATE_BLOCK_RANGE _IOW('f', 18, struct truncate_range)
> >> struct truncate_range {
> >> __u32 start_block;
> >> __u32 length;
> >> };
And have 64 bit file size support, please!
Also FALLOC_FL_COLLAPSE_RANGE is probably a better name for the
operation being done ;)
Cheers,
Dave.
--
Dave Chinner
[email protected]
On Mon, Jun 24, 2013 at 12:44:59PM +1000, Dave Chinner wrote:
>
> Hence, at minimum, this should be a fallocate() operation, not a ext4
> specific ioctl as it is relatively trivial to implement on most
> extent based filesystems.
The fallocate() uses a units of bytes for the offset and length; would
a FALLOC_FL_COLLAPSE_RANGE be guaranteed to work on any arbitrary
offset and length? Or would it only work if the offset and length are
multiples of the file system blocksize?
The the EXT4_IOC_TRUNCATE_BLOCK_RANGE interface solves this problem by
using units of file system blocks (i.e., __u32 start_block), but that
raises another issue, which is it forces the user space program to
somehow figure out the file system block size, which seems a bit nasty.
- Ted
2013/6/24, Andreas Dilger <[email protected]>:
> On 2013-06-23, at 0:07, Namjae Jeon <[email protected]> wrote:
>
>> From: Namjae Jeon <[email protected]>
>>
>> This patch series introduces 2 new ioctls for ext4.
>>
>> Truncate_block_range ioctl truncates blocks from source file.
>
Hi. Andreas.
> How is this different from fallocate(FALLOC_FL_PUNCH_HOLE)? That is already
> in existing kernels, and portable across multiple filesystems.
Dave chinner already answered it.
>
>> Transfer_block_range ioctl transfers data blocks from source file
>> and append them at the end of destination file.
>
> There is already a similar ioctl for defragmenting files. Is it possible to
> use that, or does it have different semantics?
For defrag ioctl:
struct move_extent {
__u32 reserved; /* should be zero */
__u32 donor_fd; /* donor file descriptor */
__u64 orig_start; /* logical start offset in block for orig */
__u64 donor_start; /* logical start offset in block for donor */
__u64 len; /* block length to be moved */
__u64 moved_len; /* moved block length */
};
For this ioctl to work, there are some pre conditions:
1) Contiguous blocks for donor_fd are allocated in user space using fallocate.
2) orig_start and donor_start should be same.
The fundamental difference between these 2 ioctls is that in defrag
ioctl data is copied, (page by page?) from the extents of source file
to the newly allocated extents of donor file. However, in transfer
block range ioctl the extents are transfered from the source file and
appended at the end of donor file. There is only metadata movement.
Thanks.
>
>> Ioctl1: EXT4_IOC_TRUNCATE_BLOCK_RANGE:
>> This ioctl truncates a range of data blocks from file.
>> It is useful to remove easily and quickly the garbage data
>> at the middle of file.
>>
>> e.g. we have a movie file and there is long advertisement in movie file.
>> user want to remove only advertisement range.
>
> While this works in theory, there is very little chance that the movie data
> will align exactly to filesystem block boundaries.
>
> Cheers, Andreas
>
>> 1) Movie file (8GB), There is the adverisement of 500MB size.
>> ____________________________________________________________________
On Mon, Jun 24, 2013 at 12:44:59PM +1000, Dave Chinner wrote:
> various DVR companies shipping equivalent ioctl-based functionality
> for XFS filesystems to both insert and remove chunks in video
> streams, but I've never been able to find the code for it anywhere.
Samsung does this, also for UDF. Namjae, could you please submit these
two as well, including a proper VFS interface?
On Sun, Jun 23, 2013 at 11:12:35PM -0400, Theodore Ts'o wrote:
> On Mon, Jun 24, 2013 at 12:44:59PM +1000, Dave Chinner wrote:
> >
> > Hence, at minimum, this should be a fallocate() operation, not a ext4
> > specific ioctl as it is relatively trivial to implement on most
> > extent based filesystems.
>
> The fallocate() uses a units of bytes for the offset and length; would
> a FALLOC_FL_COLLAPSE_RANGE be guaranteed to work on any arbitrary
> offset and length? Or would it only work if the offset and length are
> multiples of the file system blocksize?
There's nothing stopping us from restricting the offset/len to
specific alignments if the operation cannot be done on arbitrary
byte ranges. We do that for direct IO, and the sky hasn't fallen
yet.
> The the EXT4_IOC_TRUNCATE_BLOCK_RANGE interface solves this problem by
> using units of file system blocks (i.e., __u32 start_block), but that
> raises another issue, which is it forces the user space program to
> somehow figure out the file system block size, which seems a bit nasty.
Yeah, exactly. We can do that internally very easily, and EINVAL can
be returned when the alignment is bad just like we do for direct
IO...
But, well, I pine for a generic XFS_IOC_DIOINFO interface so the
filesystem can tell users about alignment restrictions....
Cheers,
Dave.
--
Dave Chinner
[email protected]
2013/6/24, Christoph Hellwig <[email protected]>:
> On Mon, Jun 24, 2013 at 12:44:59PM +1000, Dave Chinner wrote:
>> various DVR companies shipping equivalent ioctl-based functionality
>> for XFS filesystems to both insert and remove chunks in video
>> streams, but I've never been able to find the code for it anywhere.
>
Hi. Christoph.
> Samsung does this, also for UDF. Namjae, could you please submit these
> two as well, including a proper VFS interface?
Yes, Sure.
Intially, we also thought of adding a new flag to fallocate instead of ioctls.
But as Ted rightly pointed, fallocate works on byte granularity and we
have block aligned restrictions, so once we decided to go for ioctls
and hear other people opninons.
Currently, we can try implementing dave's suggesstion of introducing a
new flag FALLOC_FL_COLLAPSE_RANGE for falloctae instead of individual
ioctls for both XFS and EXT4.
Thanks.
>
>
On 24.06.2013 13:36, Namjae Jeon wrote:
> Currently, we can try implementing dave's suggesstion of introducing a
> new flag FALLOC_FL_COLLAPSE_RANGE for falloctae instead of individual
> ioctls for both XFS and EXT4. Thanks.
Hi,
Currently PUNCH_HOLE requires KEEP_SIZE to be set as well. I think there
is no need to invent COLLAPSE_RANGE, but instead fallocate should
support PUNCH_HOLE without KEEP_SIZE. However I'm not sure that putting
block alignment restriction is a right way to go.
Regards,
Andrey.
On Mon, Jun 24, 2013 at 10:37:57AM +0000, Sidorov, Andrei wrote:
> On 24.06.2013 13:36, Namjae Jeon wrote:
> > Currently, we can try implementing dave's suggesstion of introducing a
> > new flag FALLOC_FL_COLLAPSE_RANGE for falloctae instead of individual
> > ioctls for both XFS and EXT4. Thanks.
>
> Hi,
>
> Currently PUNCH_HOLE requires KEEP_SIZE to be set as well. I think there
> is no need to invent COLLAPSE_RANGE, but instead fallocate should
> support PUNCH_HOLE without KEEP_SIZE. However I'm not sure that putting
> block alignment restriction is a right way to go.
PUNCH_HOLE without KEEP_SIZE makes sense to me.
Regards,
- Zheng
On Mon, Jun 24, 2013 at 10:37:57AM +0000, Sidorov, Andrei wrote:
> On 24.06.2013 13:36, Namjae Jeon wrote:
> > Currently, we can try implementing dave's suggesstion of introducing a
> > new flag FALLOC_FL_COLLAPSE_RANGE for falloctae instead of individual
> > ioctls for both XFS and EXT4. Thanks.
>
> Hi,
>
> Currently PUNCH_HOLE requires KEEP_SIZE to be set as well. I think there
> is no need to invent COLLAPSE_RANGE, but instead fallocate should
> support PUNCH_HOLE without KEEP_SIZE. However I'm not sure that putting
> block alignment restriction is a right way to go.
No, it doesn't make sense. Punching a hole *leaves a hole* in the
file, not that all the data on the high side of the region being
punched out is *moved* into the hole that was punched out. i.e.
PUNCH_HOLE does not affect data outside the range of the holebeing
punched, while COLLAPSE_RANGE significantly affects data in the
file outside the range being removed. Hence that are not the same
operation at all.
FYI, what KEEP_SIZE means for punching a hole is that preserve the
file size is explicitly required by the operation. It's
documentation of the expected behaviour more than anything. i.e if
the hole is at the end of the file, punching a hole won't -truncate-
the file. Also, you can punch holes *beyond EOF* (so you can punch
blocks allocated beyond EOF) and KEEP_SIZE means that it doesn't
change the file size when it does this.
Cheers,
Dave.
--
Dave Chinner
[email protected]