Hi,
There are the updated patches for ext4 online defrag.
I have addressed the comments that I had got about defrag V1
(http://marc.info/?l=linux-ext4&m=123329594919001&w=3).
Changes of this version are as follows:
- Change function name from ext4_defrag_xxx to ext4_mext_xxx
and also change the source file name from defrag.c to move_extent.c.
- Disallow swapfile defrag.
- Support uninitialized extent defrag.
- Disallow data write out for uninitalized extent.
- Call fget() and fput() instead of fget_light() and fput_light()
for module compile.
- Acquire mutex and semaphore of orig and donor inodes by i_ino order.
- Localized semaphore acquisition/release.
- Add ext4_mext_inode_double_lock/unlock() for i_mutex lock/unlock,
since inode_double_lock/unlock() have been removed.
- Replace random printk() with ext4_debug().
- Some cleanups.
Ext4 online defrag uses EXT4_IOC_MOVE_EXT ioctl and move_extent structure.
#define EXT4_IOC_MOVE_EXT _IOWR('f', 15, struct move_extent)
struct move_extent {
int orig_fd; /* original file descriptor */
int donor_fd; /* donor file descriptor */
__u64 orig_start; /* logical start offset in block for orig */
__u64 donor_start; /* logical start offset in block for donor */
__u64 len; /* block length to be moved */
__u64 moved_len; /* moved block length */
};
Ext4 online defrag is done in the following steps:
(U:User space K:Kernel space)
1:U Create donor inode and then call unlink() to it.
2:U Allocate contiguous blocks to donor inode with fallocate().
3:U Call the FS_IOC_FIEMAP ioctl to get the extents information of donor inode.
And check the extents of donor inode are less than the defrag target inode's.
4:U Call the EXT4_IOC_MOVE_EXT ioctl extent unit.
5:K In kernel space, the EXT4_IOC_MOVE_EXT ioctl calls ext4_mext_move_extent()
to exchange the blocks between orig_fd and donor_fd.
Then write the file data of orig_fd to donor_fd.
6:U Close donor inode, then it will be deleted.
Note: More tests may be needed without journal,
so I recommend that e4defrag is run on ext4 with journal.
Summary of patch series:
* The patch can be applied to the ext4 patch queue (2.6.30-rc6).
[PATCH 1/3] Add EXT4_IOC_DEFRAG ioctl and defrag functions
- Add EXT4_IOC_DEFRAG ioctl and related functions.
- For each page, exchange the extents between original inode
and donor inode, and then write the file data of
the original inode to donor inode.
[PATCH 2/3] Ext4 online defrag command
- Usage is as follows:
- Defrag for a single file.
# e4defrag file, directory, device
- Check the file fragments on ext4.
# e4defrag -c file, directory, device
[PATCH 3/3] Manpage of e4defrag
- Ext4 online defrag command manpage
# nroff -man e4defrag.8
Best regards,
Akira Fujita
Ted / Akira,
What's the status of this. I certainly like the ABI much better now
than on the previous try.
I don't know the ext4 code internally so I can't speak to the
implementation quality, but I know this has been floating around for a
long time and seems to just be getting better and better.
Any chance of it being merged soon?
Thanks
Greg
2009/5/22 Akira Fujita <[email protected]>:
> Hi,
>
> There are the updated patches for ext4 online defrag.
> I have addressed the comments that I had got about defrag V1
> (http://marc.info/?l=linux-ext4&m=123329594919001&w=3).
> Changes of this version are as follows:
>
> - Change function name from ext4_defrag_xxx to ext4_mext_xxx
> ?and also change the source file name from defrag.c to move_extent.c.
> - Disallow swapfile defrag.
> - Support uninitialized extent defrag.
> - Disallow data write out for uninitalized extent.
> - Call fget() and fput() instead of fget_light() and fput_light()
> ?for module compile.
> - Acquire mutex and semaphore of orig and donor inodes by i_ino order.
> - Localized semaphore acquisition/release.
> - Add ext4_mext_inode_double_lock/unlock() for i_mutex lock/unlock,
> ?since inode_double_lock/unlock() have been removed.
> - Replace random printk() with ext4_debug().
> - Some cleanups.
>
> Ext4 online defrag uses EXT4_IOC_MOVE_EXT ioctl and move_extent structure.
>
> #define EXT4_IOC_MOVE_EXT ? ? ? ? ? ? ? _IOWR('f', 15, struct move_extent)
>
> struct move_extent {
> ? ? ? ?int orig_fd; ? ? ? ? ? ?/* original file descriptor */
> ? ? ? ?int donor_fd; ? ? ? ? ? /* donor file descriptor */
> ? ? ? ?__u64 orig_start; ? ? ? /* logical start offset in block for orig */
> ? ? ? ?__u64 donor_start; ? ? ?/* logical start offset in block for donor */
> ? ? ? ?__u64 len; ? ? ? ? ? ? ?/* block length to be moved */
> ? ? ? ?__u64 moved_len; ? ? ? ?/* moved block length */
> };
>
>
> Ext4 online defrag is done in the following steps:
> (U:User space K:Kernel space)
>
> 1:U ?Create donor inode and then call unlink() to it.
>
> 2:U ?Allocate contiguous blocks to donor inode with fallocate().
>
> 3:U ?Call the FS_IOC_FIEMAP ioctl to get the extents information of donor inode.
> ? ? And check the extents of donor inode are less than the defrag target inode's.
>
> 4:U ?Call the EXT4_IOC_MOVE_EXT ioctl extent unit.
>
> 5:K ?In kernel space, the EXT4_IOC_MOVE_EXT ioctl calls ext4_mext_move_extent()
> ? ? to exchange the blocks between orig_fd and donor_fd.
> ? ? Then write the file data of orig_fd to donor_fd.
>
> 6:U ?Close donor inode, then it will be deleted.
>
> Note: More tests may be needed without journal,
> ? ? ?so I recommend that e4defrag is run on ext4 with journal.
>
> Summary of patch series:
> * The patch can be applied to the ext4 patch queue (2.6.30-rc6).
>
> [PATCH 1/3] Add EXT4_IOC_DEFRAG ioctl and defrag functions
> - Add EXT4_IOC_DEFRAG ioctl and related functions.
> - For each page, exchange the extents between original inode
> ?and donor inode, and then write the file data of
> ?the original inode to donor inode.
>
> [PATCH 2/3] Ext4 online defrag command
> - Usage is as follows:
> ?- Defrag for a single file.
> ? ?# e4defrag file, directory, device
> ?- Check the file fragments on ext4.
> ? ?# e4defrag -c file, directory, device
>
> [PATCH 3/3] Manpage of e4defrag
> - ?Ext4 online defrag command manpage
> ? ?# nroff -man e4defrag.8
>
> Best regards,
> Akira Fujita
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to [email protected]
> More majordomo info at ?http://vger.kernel.org/majordomo-info.html
>
--
Greg Freemyer
Head of EDD Tape Extraction and Processing team
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
First 99 Days Litigation White Paper -
http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf
The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com
Maybe my question is stupid, but I'll post it anyway. Where can I get the
patches for online defrag?
Thanks,
Andreas
--
Andreas Philipp
Institute for Mathematics and Scientific Computing
University of Graz
Heinrichstrasse 36
8010 Graz
Austria
phone: ++43 316 380 5155
e-mail: [email protected]
web site: http://www.uni-graz.at/~philippa/
On Friday 12 June 2009 01:03:32 Greg Freemyer wrote:
> Ted / Akira,
>
> What's the status of this. I certainly like the ABI much better now
> than on the previous try.
>
> I don't know the ext4 code internally so I can't speak to the
> implementation quality, but I know this has been floating around for a
> long time and seems to just be getting better and better.
>
> Any chance of it being merged soon?
>
> Thanks
> Greg
>
> 2009/5/22 Akira Fujita <[email protected]>:
> > Hi,
> >
> > There are the updated patches for ext4 online defrag.
> > I have addressed the comments that I had got about defrag V1
> > (http://marc.info/?l=linux-ext4&m=123329594919001&w=3).
> > Changes of this version are as follows:
> >
> > - Change function name from ext4_defrag_xxx to ext4_mext_xxx
> > and also change the source file name from defrag.c to move_extent.c.
> > - Disallow swapfile defrag.
> > - Support uninitialized extent defrag.
> > - Disallow data write out for uninitalized extent.
> > - Call fget() and fput() instead of fget_light() and fput_light()
> > for module compile.
> > - Acquire mutex and semaphore of orig and donor inodes by i_ino order.
> > - Localized semaphore acquisition/release.
> > - Add ext4_mext_inode_double_lock/unlock() for i_mutex lock/unlock,
> > since inode_double_lock/unlock() have been removed.
> > - Replace random printk() with ext4_debug().
> > - Some cleanups.
> >
> > Ext4 online defrag uses EXT4_IOC_MOVE_EXT ioctl and move_extent
> > structure.
> >
> > #define EXT4_IOC_MOVE_EXT _IOWR('f', 15, struct
> > move_extent)
> >
> > struct move_extent {
> > int orig_fd; /* original file descriptor */
> > int donor_fd; /* donor file descriptor */
> > __u64 orig_start; /* logical start offset in block for orig
> > */ __u64 donor_start; /* logical start offset in block for donor */
> > __u64 len; /* block length to be moved */
> > __u64 moved_len; /* moved block length */
> > };
> >
> >
> > Ext4 online defrag is done in the following steps:
> > (U:User space K:Kernel space)
> >
> > 1:U Create donor inode and then call unlink() to it.
> >
> > 2:U Allocate contiguous blocks to donor inode with fallocate().
> >
> > 3:U Call the FS_IOC_FIEMAP ioctl to get the extents information of donor
> > inode. And check the extents of donor inode are less than the defrag
> > target inode's.
> >
> > 4:U Call the EXT4_IOC_MOVE_EXT ioctl extent unit.
> >
> > 5:K In kernel space, the EXT4_IOC_MOVE_EXT ioctl calls
> > ext4_mext_move_extent() to exchange the blocks between orig_fd and
> > donor_fd.
> > Then write the file data of orig_fd to donor_fd.
> >
> > 6:U Close donor inode, then it will be deleted.
> >
> > Note: More tests may be needed without journal,
> > so I recommend that e4defrag is run on ext4 with journal.
> >
> > Summary of patch series:
> > * The patch can be applied to the ext4 patch queue (2.6.30-rc6).
> >
> > [PATCH 1/3] Add EXT4_IOC_DEFRAG ioctl and defrag functions
> > - Add EXT4_IOC_DEFRAG ioctl and related functions.
> > - For each page, exchange the extents between original inode
> > and donor inode, and then write the file data of
> > the original inode to donor inode.
> >
> > [PATCH 2/3] Ext4 online defrag command
> > - Usage is as follows:
> > - Defrag for a single file.
> > # e4defrag file, directory, device
> > - Check the file fragments on ext4.
> > # e4defrag -c file, directory, device
> >
> > [PATCH 3/3] Manpage of e4defrag
> > - Ext4 online defrag command manpage
> > # nroff -man e4defrag.8
> >
> > Best regards,
> > Akira Fujita
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> > the body of a message to [email protected]
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, Jun 12, 2009 at 3:47 AM, Andreas
Philipp<[email protected]> wrote:
> Maybe my question is stupid, but I'll post it anyway. Where can I get the
> patches for online defrag?
>
> Thanks,
> Andreas
>
> --
> Andreas Philipp
Akira Fujita posted them last month.
http://markmail.org/search/?q=Akira+Fujita+#query:Akira Fujita date%3A200905
Greg
On Thu, Jun 11, 2009 at 07:03:32PM -0400, Greg Freemyer wrote:
> Ted / Akira,
>
> What's the status of this. I certainly like the ABI much better now
> than on the previous try.
>
> I don't know the ext4 code internally so I can't speak to the
> implementation quality, but I know this has been floating around for a
> long time and seems to just be getting better and better.
>
> Any chance of it being merged soon?
Sorry, I've been pretty busy trying to fix some rather critical and
nasty ext4 bugs, and then getting things ready for the merge window.
I've dropped Akira's patch into unstable portion of the ext4 patch
queue for testing. I want to take a closer look at the ABI, but I
agree it is much better. If the ABI is good, I'm likely to try
merging it this merge window, on the grounds that any implementation
issues can be cleaned up later, since it is a relatively
self-contained patch.
- Ted
Greg Freemyer wrote:
> On Fri, Jun 12, 2009 at 3:47 AM, Andreas
> Philipp<[email protected]> wrote:
>
>> Maybe my question is stupid, but I'll post it anyway. Where can I get the
>> patches for online defrag?
>>
>> Thanks,
>> Andreas
>>
>> --
>> Andreas Philipp
>>
>
> Akira Fujita posted them last month.
>
> http://markmail.org/search/?q=Akira+Fujita+#query:Akira Fujita date%3A200905
>
> Greg
>
I sucessfully applied them on top of 2.6.30-git3. Please don't hesitate
to contact me, if you need anyone running some tests or something like that.
Andreas
On Fri, Jun 19, 2009 at 10:53 AM, Andreas
Philipp<[email protected]> wrote:
> Greg Freemyer wrote:
>> On Fri, Jun 12, 2009 at 3:47 AM, Andreas
>> Philipp<[email protected]> wrote:
>>
>>> Maybe my question is stupid, but I'll post it anyway. Where can I get the
>>> patches for online defrag?
>>>
>>> Thanks,
>>> Andreas
>>>
>>> --
>>> Andreas Philipp
>>>
>>
>> Akira Fujita posted them last month.
>>
>> http://markmail.org/search/?q=Akira+Fujita+#query:Akira Fujita date%3A200905
>>
>> Greg
>>
>
> I sucessfully applied them on top of 2.6.30-git3. Please don't hesitate
> to contact me, if you need anyone running some tests or something like that.
>
> Andreas
Ted also pushed them as part of the 2.6.31 merge window, so testers
will be able to test with 2.6.30-git14 or newer without having to
apply the patches separately..
Greg
--
Greg Freemyer
Head of EDD Tape Extraction and Processing team
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
First 99 Days Litigation White Paper -
http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf
The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com
On Fri, Jun 19, 2009 at 11:31:20AM -0400, Greg Freemyer wrote:
>
> Ted also pushed them as part of the 2.6.31 merge window, so testers
> will be able to test with 2.6.30-git14 or newer without having to
> apply the patches separately..
You may need a slightly modified userspace utility from what Akira-san
posted, which you can get here:
http://repo.or.cz/w/ext4-patch-queue.git?a=blob_plain;f=online-defrag-cmd;h=7d42ad5cdb6813d262201d6806531f189d3d1f57;hb=35e9fa546ac349cb9a7c4ee9e45ff9747edd9335
This reflects a change that I made in the structure passed in the new
EXT4_IOC_MOVE_EXT ioctl.
Note that I have not done a lot of stress testing of the defrag
functionality, so please be careful using it on production systems.
Please make sure you backup your files before trying to use it.
- Ted