Hello all,
We are pleased to announce the first official functional release of Online
Hierarchical Storage Manager (OHSM v1.2). This is a RFC release and
not yet aimed at mainline inclusion.
OHSM is a tool to manage and move data across various class of storage.
It can help users to selectively place and move data across tiers such
as SSD, Raid 10, Raid 6 based on the attributes of the data. OHSM
supports background movement of data
without any visible change in a files namespace to users and user applications.
OHSM is built as an external module with recompilation required for
ext4. The current version of OHSM is based upon kernel 2.6.32.2
This release includes 3 core kernel patches:
1) An initial ext4 patch to adjust ext4's block allocation to use a
preferred block range per Ted
Tso's Dec 2008 write-up
(http://markmail.org/message/qp7zjhhdzxum7rfn). This patch is not
ohsm specific.
2) A ohsm specific ext4 patch to provide callouts to the ohsm module.
3) The ohsm module itself
The source code for OHSM v1.2 is freely distributable under GPL.
The latest stable OHSM v1.2 is available at :
http://sourceforge.net/projects/ohsm/files/OHSMv1.2.tar.gz/download
Sources are available as git repository at :
git://ohsm.git.sourceforge.net/gitroot/ohsm/ohsm
The following list describes the features of OHSM
* Support for Ext4.
* File System callouts
* Policy based File Migration
* Per inode information maintainer in user xattr area
* XML based policy files
* Support for RAID
* DTD based XML validation
- Automatic Relocation engine
* Selective Relocation of mountpoint/Subdirectories
* Selection Relocation based on specific rule
* File Placement based on:
+ UID
+ GID
+ File Type
+ Directory Name
* File Migration based on
+ UID
+ GID
+ FILE SIZE
+ ACCESS AGE
+ MODIFICATION AGE
- File Access Temperatue
- File I/O Temperature
+
(*=current features, -= on our todo list, +=support already present)
Here are some simple examples :-
$ ohsm enable sample/ohsm.xml
OHSM enabled successfully on '/tmp/mnt'
$ ohsm relocate -m /tmp/mnt
Relocation performed successfully.
$ ohsm status /tmp/mnt
OHSM is enabled on '/tmp/mnt'
$ ohsm disable /tmp/mnt
OHSM disabled successfully on '/tmp/mnt'
$ ohsm
Usage: ohsm subcommand [optstring ..] [parameters ..]
Subcommands :
enable - Enable OHSM
disable - Disable OHSM
status - Show status of OHSM
relocate -Trigger relocation on filesystem
update - Update OHSM instance parameters
help -Show usage
*Note* that the code for OHSM is still in early stages
Feedback, bug reports and patches, as always, are welcomed.
([email protected])
--
Thanks -
OHSM team
Manish Katiyar <[email protected]> writes:
> Hello all,
> We are pleased to announce the first official functional release of Online
> Hierarchical Storage Manager (OHSM v1.2). This is a RFC release and
> not yet aimed at mainline inclusion.
>
> OHSM is a tool to manage and move data across various class of storage.
> It can help users to selectively place and move data across tiers such
> as SSD, Raid 10, Raid 6 based on the attributes of the data. OHSM
> supports background movement of data
> without any visible change in a files namespace to users and user applications.
It does sounds like btrfs multy-device support. Can you please
what it the differance?
> OHSM is built as an external module with recompilation required for
> ext4. The current version of OHSM is based upon kernel 2.6.32.2
>
> This release includes 3 core kernel patches:
>
> 1) An initial ext4 patch to adjust ext4's block allocation to use a
> preferred block range per Ted
> Tso's Dec 2008 write-up
> (http://markmail.org/message/qp7zjhhdzxum7rfn). This patch is not
> ohsm specific.
> 2) A ohsm specific ext4 patch to provide callouts to the ohsm module.
> 3) The ohsm module itself
>
> The source code for OHSM v1.2 is freely distributable under GPL.
> The latest stable OHSM v1.2 is available at :
> http://sourceforge.net/projects/ohsm/files/OHSMv1.2.tar.gz/download
> Sources are available as git repository at :
> git://ohsm.git.sourceforge.net/gitroot/ohsm/ohsm
>
> The following list describes the features of OHSM
> * Support for Ext4.
> * File System callouts
> * Policy based File Migration
> * Per inode information maintainer in user xattr area
> * XML based policy files
> * Support for RAID
> * DTD based XML validation
> - Automatic Relocation engine
> * Selective Relocation of mountpoint/Subdirectories
> * Selection Relocation based on specific rule
> * File Placement based on:
> + UID
> + GID
> + File Type
> + Directory Name
> * File Migration based on
> + UID
> + GID
> + FILE SIZE
> + ACCESS AGE
> + MODIFICATION AGE
> - File Access Temperatue
> - File I/O Temperature
> +
> (*=current features, -= on our todo list, +=support already present)
>
> Here are some simple examples :-
> $ ohsm enable sample/ohsm.xml
> OHSM enabled successfully on '/tmp/mnt'
>
> $ ohsm relocate -m /tmp/mnt
> Relocation performed successfully.
>
> $ ohsm status /tmp/mnt
> OHSM is enabled on '/tmp/mnt'
>
> $ ohsm disable /tmp/mnt
> OHSM disabled successfully on '/tmp/mnt'
>
> $ ohsm
> Usage: ohsm subcommand [optstring ..] [parameters ..]
> Subcommands :
> enable - Enable OHSM
> disable - Disable OHSM
> status - Show status of OHSM
> relocate -Trigger relocation on filesystem
> update - Update OHSM instance parameters
> help -Show usage
>
> *Note* that the code for OHSM is still in early stages
> Feedback, bug reports and patches, as always, are welcomed.
> ([email protected])
On Tue, Feb 23, 2010 at 2:14 AM, Dmitry Monakhov <[email protected]> wrote:
> Manish Katiyar <[email protected]> writes:
>
>> Hello all,
>> ? ?We are pleased to announce the first official functional release of Online
>> Hierarchical Storage Manager (OHSM v1.2). ?This is a RFC release and
>> not yet aimed at mainline inclusion.
>>
>> OHSM is a tool to manage and move data across various class of storage.
>> It can help users to selectively place and move data across tiers such
>> as SSD, Raid 10, Raid 6 based on the attributes of the data. OHSM
>> supports background movement of data
>> without any visible change in a files namespace to users and user applications.
> It does sounds like btrfs multy-device support. Can you please
> what it the differance?
Dmitry,
Per my understanding of btrfs multi-device, ohsm is totally different.
Btrfs multi-device I believe provides functionality similar to DM
(device mapper) and/or mdraid.
Ohsm is about cost effectively managing multiple storage tiers within
a single filesystem and leverages having DM and mdraid available to
build on, but makes no effort to duplicate their functionality. DM in
particular is a mandatory part of a ohsm environment.
For example:
Assume I have an enterprise app that needs 10 TB of storage, but 90%
of the data is of limited use most of the time. The other 10% is
heavily used and high performance is paramount. The trouble is that
from time-to-time the designation of the 10% changes as business needs
change. ie. Normally I really need database abc to be fast, but at
the end of the month I need database xyz to be as fast as possible.
One real world solution without OHSM is to create a SSD raid-1 that
holds the 10%, (1TB), and a SATA raid 6 that holds the less critical
90% (9TB).
Then, as just one example, if I need to accelerate a database for a
few days / weeks I simply move the database tables from the
low-performance file system to the high-performance file system. And
I see my database speed drastically accelerate.
There are 2 big downsides to the above:
1) The full path name to the database tables will change as they are
moved between file systems, so it is a admin hassle to update
references to the tables as they move around.
2) The tables can not be in use as they are moved, so if I really am
moving a TB of data between sata and SSD that could take some number
of hours where I can't be actively using the files. A definite
downtime issue.
Now with OHSM, we would build a DM logical volume composed of a sata
raid array and a ssd raid array. Thus blocks 0-x would be on the sata
array and blocks x-end would be on SSD.
We would then use ohsm to manage which block range was used by the
various files. The goal being that low-performance files would be
stored on sata devices and files needing high-performance would be
stored on SSD.
And as the performance needs of the files changed, the files could be
moved between the tiers. The files blocks are moved via
ext4_ioc_move_ext() so the path'ing is not changed and the file can be
open and in use as it is moved.
In a sense we are performing a defrag action on the file where the
destination blocks of the file are in a different storage tier than
the original data blocks.
fyi: One of the ways we track desired storage tier for a file is via
subtrees, thus the interest of ohsm in your subtree implementation.
HTH
Greg
Hello Manish,
> We are pleased to announce the first official functional release of Online
> Hierarchical Storage Manager (OHSM v1.2). This is a RFC release and
> not yet aimed at mainline inclusion.
>
> OHSM is a tool to manage and move data across various class of storage.
> It can help users to selectively place and move data across tiers such
> as SSD, Raid 10, Raid 6 based on the attributes of the data. OHSM
> supports background movement of data
> without any visible change in a files namespace to users and user applications.
> OHSM is built as an external module with recompilation required for
> ext4. The current version of OHSM is based upon kernel 2.6.32.2
>
> This release includes 3 core kernel patches:
>
> 1) An initial ext4 patch to adjust ext4's block allocation to use a
> preferred block range per Ted
> Tso's Dec 2008 write-up
> (http://markmail.org/message/qp7zjhhdzxum7rfn). This patch is not
> ohsm specific.
> 2) A ohsm specific ext4 patch to provide callouts to the ohsm module.
> 3) The ohsm module itself
>
> The source code for OHSM v1.2 is freely distributable under GPL.
> The latest stable OHSM v1.2 is available at :
> http://sourceforge.net/projects/ohsm/files/OHSMv1.2.tar.gz/download
> Sources are available as git repository at :
> git://ohsm.git.sourceforge.net/gitroot/ohsm/ohsm
Looking at the code, the code deciding where to put a given file isn't
really tied to ext4, is it?
Using pathnames to decide where to put files is ... natural but it has
its downsides. For example your code in build_path() is racy because the
directory hiearchy can change while you climb the tree - not mentioning the
fact that the static buffer of 4096 for pathname is ugly and implementing
that simple walk recursively isn't good either. Kernel already has a d_path
function for constructing path in at least somewhat reliable way.
Anyway I suggest you have a look at Tomoyo which implements pathname based
security module so that you can avoid mistakes they did...
Honza
On Tue, Feb 23, 2010 at 7:57 PM, Jan Kara <[email protected]> wrote:
> ?Hello Manish,
>
>> ? ?We are pleased to announce the first official functional release of Online
>> Hierarchical Storage Manager (OHSM v1.2). ?This is a RFC release and
>> not yet aimed at mainline inclusion.
>>
>> OHSM is a tool to manage and move data across various class of storage.
>> It can help users to selectively place and move data across tiers such
>> as SSD, Raid 10, Raid 6 based on the attributes of the data. OHSM
>> supports background movement of data
>> without any visible change in a files namespace to users and user applications.
>> OHSM is built as an external module with recompilation required for
>> ext4. The current version of OHSM is based upon kernel 2.6.32.2
>>
>> This release includes 3 core kernel patches:
>>
>> 1) An initial ext4 patch to adjust ext4's block allocation to use a
>> preferred block range per Ted
>> Tso's Dec 2008 write-up
>> (http://markmail.org/message/qp7zjhhdzxum7rfn). ?This patch is not
>> ohsm specific.
>> 2) A ohsm specific ext4 patch to provide callouts to the ohsm module.
>> 3) The ohsm module itself
>>
>> The source code for OHSM v1.2 is freely distributable under GPL.
>> The latest stable OHSM v1.2 is available at :
>> http://sourceforge.net/projects/ohsm/files/OHSMv1.2.tar.gz/download
>> Sources are available as git repository at :
>> git://ohsm.git.sourceforge.net/gitroot/ohsm/ohsm
> ?Looking at the code, the code deciding where to put a given file isn't
> really tied to ext4, is it?
Hi Jan,
ext4 was our default choice to get started, because of the ioctls
provided by Akira and since e4defrag already exists it was much easier
for us to refer. Though we have tried to keep the OHSM code as
independant as possible so that it would be possible to support other
filesystems in future too.
> ?Using pathnames to decide where to put files is ... natural but it has
> its downsides. For example your code in build_path() is racy because the
> directory hiearchy can change while you climb the tree - not mentioning the
> fact that the static buffer of 4096 for pathname is ugly and implementing
> that simple walk recursively isn't good either. Kernel already has a d_path
> function for constructing path in at least somewhat reliable way.
Thanks a lot for your feedback. We have fixed some bugs recently to
get rid of static allocations so they will go away. I will have a look
at d_path().
> ?Anyway I suggest you have a look at Tomoyo which implements pathname based
> security module so that you can avoid mistakes they did...
Sure, will do. Definitely it will help us learn more and make the code
base better.
Thanks -
Manish
>
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Honza
>
>
--
Thanks -
Manish
==================================
[$\*.^ -- I miss being one of them
==================================