Hello all,
hope I am right here.
We have some questions concerning the ext2 filesystem:
When looking for free blocks and inodes in function ext2_statfs the
overhead, free inodes and free blocks are newly calculated by iterating
through all
inodes and blocks and look if they are free or not.
On the other hand free inodes and free blocks are held in the
structure-members s_freeinode_count
and s_free_block_count of the superblock info.
Now my questions:
Why is the overhead calculated ?
Isn't it is constant since filling the superblock ?
Wy are free blocks and free inodes are calculated every time,
when they are held in s_freeinode_count and s_free_block_count ?
Isn't it is possible to use these variables directly ?
By the way: s_freeinode_count is not incremented in function
ext2_release_inode.
The background for these questions is as it always is:
We have a problem using 320 GB harddrives instead of 80 GB harddrives with
ext2.
Before writing to the harddrive our software looks for enough space
to make sure the data can be stored safely.
This leads to heavy performance problems when using larger disks.
We can reduce the check for enough space to every 10th or 100th write
operation,
but at some point our software has to check and this may lead to a los of
data
because the system is busy calculating the free inodes and blocks.
Our solution will be to create a patch using the structure members
s_freeinode_count, s_free_block_count
and create a new member s_overhead_count and use them directly.
s_overhead_count is calculated once when filling the superblock.
Is this a good solution or are there circumstances which can lead to a
misbehavior ?
Thanks in advance,
Bj?rn Slotkowski
--
imc Messsysteme GmbH
Bereich Entwicklung / Dept. Development
Voltastra?e 5
D-13355 Berlin
Germany
Tel: +49 30 46 70 90 23
Fax: +49 30 46 31 57 6
mailto:[email protected]
Homepage: http://www.imc-berlin.de
Sitz der Gesellschaft : Berlin
Handelsregister : Berlin-Charlottenburg HRB 28778
Gesch?ftsf?hrer : Dr.-Ing. Dietmar Sprenger
On 2010-12-22, at 05:14, Bjoern Slotkowski wrote:
> When looking for free blocks and inodes in function ext2_statfs the
> overhead, free inodes and free blocks are newly calculated by iterating
> through all inodes and blocks and look if they are free or not.
>
> On the other hand free inodes and free blocks are held in the
> structure-members s_freeinode_count and s_free_block_count of the
> superblock info.
>
>
> Now my questions:
>
> Why is the overhead calculated ?
> Isn't it is constant since filling the superblock ?
> Wy are free blocks and free inodes are calculated every time,
> when they are held in s_freeinode_count and s_free_block_count ?
They are not calculated every time ext2_statfs() is called. This
overhead is only calculated if sbi->s_blocks_last does not match
the total number of blocks the last time the function was called.
Normally this means the overhead is computed once when the filesystem
is first mounted, and never computed again unless the filesystem is
resized online.
> Isn't it is possible to use these variables directly ?
No, because the overhead is unrelated to the "free" blocks count,
but rather the difference between the "free" and "avail" blocks.
> The background for these questions is as it always is:
> We have a problem using 320 GB harddrives instead of 80 GB harddrives
> with ext2.
>
> Before writing to the harddrive our software looks for enough space
> to make sure the data can be stored safely.
> This leads to heavy performance problems when using larger disks.
What version of the kernel are you using?
> We can reduce the check for enough space to every 10th or 100th write
> operation, but at some point our software has to check and this may
> lead to a loss of data because the system is busy calculating the free
> inodes and blocks.
Isn't that really the filesystem's job, and not the job of your software?
The job of the software is to write until ENOSPC is returned by the
filesystem. There is no need to call statfs() at all before a write.
> Our solution will be to create a patch using the structure members
> s_freeinode_count, s_free_block_count
> and create a new member s_overhead_count and use them directly.
> s_overhead_count is calculated once when filling the superblock.
>
> Is this a good solution or are there circumstances which can lead to a
> misbehavior ?
Please see any newer kernel release. There is no scanning done anymore.
I see commit 2235219b7721b8e74de6841e79240936561a2b63 in 2007 implemented
this, which is included in vanilla 2.6.23.
Cheers, Andreas
Hello Andreas,
thanks a lot for your answers.
We have kernel version 2.6.14 thats really old I know.
When I looked at a newer version (2.6.31.8) the handling in ext2_statfs
seemed to be the same, but you are right I did not see all conditions which
lead to the calculation of the overhead.
My fault sorry.
But there is still one thing also in the newer kernel:
f_bfree and f_ffree are determined by functions "ext2_count_free_blocks" and
"ext2_count_free_inodes"
which iterate through s_groups_count which needs a lot of time.
When I look at ext3/super.c and ext4/super.c the member variables
s_freeblocks_counter and s_freeinodes_counter are used directly in functions
*_statfs.
Probably there is potential for an optimization.
When patching the handling of the newer linux version to our version,
increment s_freeinodes_counter in function "ext2_release_inode" and
use the optimization by using s_freeblocks_counter and s_freeinodes_counter
directly
your software works really fine.
Best regards,
Bjoern
--
imc Messsysteme GmbH
Bereich Entwicklung / Dept. Development
Voltastra?e 5
D-13355 Berlin
Germany
Tel: +49 30 46 70 90 23
Fax: +49 30 46 31 57 6
mailto:[email protected]
Homepage: http://www.imc-berlin.de
Sitz der Gesellschaft : Berlin
Handelsregister : Berlin-Charlottenburg HRB 28778
Geschaftsfuhrer : Dr.-Ing. Dietmar Sprenger
> -----Ursprungliche Nachricht-----
> Von: [email protected]
> [mailto:[email protected]]Im Auftrag von Andreas Dilger
> Gesendet: Mittwoch, 22. Dezember 2010 18:01
> An: Bjoern Slotkowski
> Cc: [email protected]
> Betreff: Re: Questions concerning ext2 filesystem: ext2_statfs
>
>
> On 2010-12-22, at 05:14, Bjoern Slotkowski wrote:
> > When looking for free blocks and inodes in function ext2_statfs the
> > overhead, free inodes and free blocks are newly calculated by iterating
> > through all inodes and blocks and look if they are free or not.
> >
> > On the other hand free inodes and free blocks are held in the
> > structure-members s_freeinode_count and s_free_block_count of the
> > superblock info.
> >
> >
> > Now my questions:
> >
> > Why is the overhead calculated ?
> > Isn't it is constant since filling the superblock ?
> > Wy are free blocks and free inodes are calculated every time,
> > when they are held in s_freeinode_count and s_free_block_count ?
>
> They are not calculated every time ext2_statfs() is called. This
> overhead is only calculated if sbi->s_blocks_last does not match
> the total number of blocks the last time the function was called.
>
> Normally this means the overhead is computed once when the filesystem
> is first mounted, and never computed again unless the filesystem is
> resized online.
>
> > Isn't it is possible to use these variables directly ?
>
> No, because the overhead is unrelated to the "free" blocks count,
> but rather the difference between the "free" and "avail" blocks.
>
> > The background for these questions is as it always is:
> > We have a problem using 320 GB harddrives instead of 80 GB harddrives
> > with ext2.
> >
> > Before writing to the harddrive our software looks for enough space
> > to make sure the data can be stored safely.
> > This leads to heavy performance problems when using larger disks.
>
> What version of the kernel are you using?
>
> > We can reduce the check for enough space to every 10th or 100th write
> > operation, but at some point our software has to check and this may
> > lead to a loss of data because the system is busy calculating the free
> > inodes and blocks.
>
> Isn't that really the filesystem's job, and not the job of your software?
> The job of the software is to write until ENOSPC is returned by the
> filesystem. There is no need to call statfs() at all before a write.
>
> > Our solution will be to create a patch using the structure members
> > s_freeinode_count, s_free_block_count
> > and create a new member s_overhead_count and use them directly.
> > s_overhead_count is calculated once when filling the superblock.
> >
> > Is this a good solution or are there circumstances which can lead to a
> > misbehavior ?
>
> Please see any newer kernel release. There is no scanning done anymore.
> I see commit 2235219b7721b8e74de6841e79240936561a2b63 in 2007 implemented
> this, which is included in vanilla 2.6.23.
>
> Cheers, Andreas
>
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
On Mon, Jan 03, 2011 at 02:12:04PM +0100, Bjoern Slotkowski wrote:
>
> But there is still one thing also in the newer kernel:
> f_bfree and f_ffree are determined by functions "ext2_count_free_blocks" and
> "ext2_count_free_inodes"
> which iterate through s_groups_count which needs a lot of time.
>
> When I look at ext3/super.c and ext4/super.c the member variables
> s_freeblocks_counter and s_freeinodes_counter are used directly in functions
> *_statfs.
>
> Probably there is potential for an optimization.
Perhaps, but note that ext4 is fully backwards compatible with ext2.
That is, you can mount ext2 file systems using the ext4 file system
driver. (You can also mount ext2 file systems using the ext3 file
system, although you will have to add a journal to the file system
first.)
Ext2 is primarily useful as an working, example file system which
shows how to interface a file system to Linux. As such piling in too
many optimizations isn't necessarily consistent with its current role.
No one is really actively maintaining ext2, except to update it work
with the latest kernel interfaces, so it's not clear who would do the
optimization.
If you want a faster, more efficient file system, there are plenty of
alternatives, including ext3 and ext4.
- Ted