Hi All,
Here is the latest version of the punch hole patch for ext4. There have
been a lot of revisions and tests since the last patch that was sent.
It looks like some parts of it may conflict with the "[PATCH RFC 2/3]
ext4:Add two functions splitting an extent" patch, so we will still need
to work that part out. But I wanted to keep people updated and get some
feed back. Here is a quick list of the big changes since v4:
- The first patch adds a flag to ext4_has_free_blocks which enables the
use of reserved blocks.
Since the extents may need to be split during the punch hole process, it
is possible that it may require more blocks than are available. This
new flag will allow a punch hole to proceed even if the disk is full.
- The code for splitting an extent has been simplified into a new
ext4_split_unwritten_extents flag.
The ext4_split_extents function from the first patch in v4 has been removed.
- The code for finding and punching out blocks in an extent has been
moved in the the ext4_ext_map_blocks function, and invoked with a new
punch hole flag.
- The code that converts the extent to uninitialized has also been moved
into the ext4_ext_map_blocks function with the new punch hole code,
so the ext4_ext_convert_blocks_uninit function from the first patch in
v4 has been removed
Also, here is a list of test cases that were done to help find any bugs.
These test cases were initially created to be used with a debug patch to
help ensure that the new code paths behave as expected, but we plan to
turn them in new test cases for xfstests and ltp after we have
the punch hole code wrapped up.
Small Hole Test
---------------------------------------------------------------
A hole is punched in a 10K file from bytes 8180 to 16216 resulting in
one block being removed,
and the non block aligned data to be cleared.
This test is a general purpose punch hole test to verify that the code
can successfully
split the extents, and remove the affected blocks, as well as clearing
the associated pages
This test successful when the following conditions are met:
- ls shows that the file now occupies one less block
- df shows there is one more block available
- File frag shows that the second block is absent from the extents
- The test file contains zeros from bytes 8180 to 16216
Cold Cache Test
---------------------------------------------------------------
This test is similar to the Small Hole Test, but the drive is unmounted
and remounted before
the hole is punched out to make sure that there are no pages in the
cache. This test verifies
that the hole contains zeros even when the pages are cold.
This test is successful when the same conditions as the Small Hole Test
are met.
Dirty Data Test
---------------------------------------------------------------
This test is similar to the Small Hole Test, but the file is not copied
to the test area. Instead
a new file is created, and the data is written into the file. This test
verifies
that the hole contains zeros even when the pages are dirty.
This test is successful when the same conditions as the Small Hole Test
are met.
Big Hole Test
---------------------------------------------------------------
A hole large hole is punched in a large file (exact file size=638169088
bytes, exact hole size = 638150422 bytes, offset = 6144 bytes),
resulting in all but 5 blocks being punched out (2 in the front, 3 in
the back). This test case verifies that the code can properly
punch out a hole covering multiple extents.
This test is successful when the following conditions are met:
- File frag shows extents only for the first two blocks and the last 3
blocks
- The test file contains zeros from bytes 6144 to 638156566
(* ls and df is not measured here because some blocks will still be
reserved as index
blocks causing the consumed space to be appear larger)
Big Hole Removal Test
---------------------------------------------------------------
This test is similar to the Big Hole Test, but the file is deleted after
the hole is punched out.
This test is successful when the following conditions are met:
- df shows that the number of available blocks is the same before the
test file is copied
into the file system, and after the file is deleted
First Byte Hole Test
---------------------------------------------------------------
This test cases punches a hole in 10K file from bytes 0 to 8180
resulting in one block being removed at the
beginning of the file. This test verifies that the code appropriately
handles punching holes appearing at the
beginning of the file.
This test is successful when the following conditions are met:
- File frag shows that the first extent starts at block one
- ls shows that the file now occupies one less block
- df shows there is one more block available
- The first 8180 bytes of the file contains zeros
Last Block Hole Test
---------------------------------------------------------------
This test cases punches a hole in 10K file from bytes 32768 to 40960
resulting in a hole that extends to the end
of the file. This test verifies that the code appropriately handles
removing blocks at the end of the file
This test is successful when the following conditions are met:
- File frag shows that the last extent only extends to block 8
- ls shows that the file now occupies two less blocks
- df shows there are two more block available
- The file contains zeros between bytes 32768 to 40960
Extended Last Block Hole Test
---------------------------------------------------------------
This test cases punches a hole in 10K file from bytes 32768 to 45056
resulting in a hole that extends beyond
the end of the file. This test verifies that the code appropriately
handles removing blocks beyond the end of the file
This test is successful when the following conditions are met:
- File frag shows that the last extent only extends to block 8
- ls shows that the file now occupies two less blocks
- df shows there are two more block available
-The file contains zeros between bytes 32768 to 40960
One block hole test
---------------------------------------------------------------
This test punches a hole in a file that is only one block in size from
byte 1024 to 3072
This test is successful when the following conditions are met:
-ls, df, and filefrag show no changes before and after the hole punch
-The file contains zeros between bytes 1024 and 3072
Off Edge Hole Test
---------------------------------------------------------------
This test case punched a hole that is beyond the edge of the file.
This test is successful when ls, df, filefrag and the file contents show
no changes
Delayed Allocation Hole Test
---------------------------------------------------------------
This test is similar to the Small Hole Test, but the file is not sync'd
to the disk before the hole is punched out.
This test is successful when the following conditions are met:
- ls shows that the file now occupies one less block
- df shows there is one more block available
- File frag shows that the second block is absent from the extents, and
the first and last extents are delayed
- The test file contains zeros from bytes 8180 to 16216
Pre allocation Hole Test
---------------------------------------------------------------
This test is similar to the Small Hole Test, but the file is
preallocated before the hole is punched out.
This test is successfull when the following conditions are met:
- ls shows that the file now occupies one less block
- df shows there is one more block available
- File frag shows that the second block is absent from the extents, and
the first and last extents are unwritten
- The test file contains zeros from bytes 8180 to 16216
Multi Hole Test
---------------------------------------------------------------
This test verifies the code ability to handle existing holes
The test punches out 3 consecutive overlapping holes in a 10K file:
Hole 1: offset 8192, length 8192
Hole 2: offset 8192, length 8192
Hole 3: offset 8180, length 8192
Hole 4: offset 8204, length 8192
Hole 5: offset 12288, length 12288
After each hole punch, output from ls, df and file frag are shown.
This test is successful when the following conditions are met:
- Hole 1 shows the following changes:
- ls shows that the file now occupies two less blocks
- df shows there is two more blocks available
- File frag shows that the second and third blocks are absent from the
extents
- The test file contains zeros from bytes 8192 to 16384
- Hole 2 shows no changes in ls, df, filefrag or file contents.
- Hole 3 shows no changes in ls, df, filefrag but the file contents
should be zeroed out between 8192 and 16372
- Hole 5 shows no changes in ls, df, filefrag but the file contents
should be zeroed out between 8180 and 16396
- Hole 5 shows that the hole has increased by two blocks for ls, df, and
filefrag.
The file contents should be zeroed out between 8180 and 24576
Multi Hole Test Front
---------------------------------------------------------------
This test verifies the code ability to handle existing holes at the
front of the file.
The test punches two 8192 byte holes starting at byte 0.
This test is successful when the following conditions are met:
- Hole 1 shows the following changes:
- ls shows that the file now occupies two less blocks
- df shows there are two more block available
- File frag shows that the first extent starts at block 2
- The test file contains zeros from bytes 0 to 8192
- Hole 2 shows no changes in ls, df, filefrag or file contents.
Multi Hole Delayed Test
---------------------------------------------------------------
This test verifies the code ability to handle existing holes in delayed
blocks.
This test is similar to the Delayed Allocation Hole Test, but the hole
is punched twice
This test is successful when the following conditions are met:
- ls shows that the file now occupies one less block
- df shows there is one more block available
- File frag shows that the second block is absent from the extents, and
the first and last extents are delayed
- The test file contains zeros from bytes 8180 to 16216
ENOSPC Hole Test
---------------------------------------------------------------
This test case was created to show that the punch hole can proceed even
if the disk is full. This test case fills the file system as much as
possible, and then proceeds to punch out every other two blocks of the
largest file, refilling the file system after each hole. With out the
ext4_has_free_blocks
flag to enable the use of reserved blocks, the ENOSPC error was
encountered after about 333 iterations. This test case is considered
passed after 5000 successful hole punches.
FSX Stress Test
---------------------------------------------------------------
To help track down any unidentified bugs, a copy of the fsx stress code
was obtained from xfstests, and modified to include hole punching as
part of it
test operation. This test is considered complete after 100000
successful file operations