Hey,
i tried to switch my root partition to ext4 (from reiserfs) but i ran
into problems when copying over all the data that it says 'no space
left on the device'
i'm running 2.6.28-gentoo-r1 (based on 2.6.28.1 if i'm not mistaken)
i deleted quite a bit and then tried copying again, this time it still
gave the same error but more files (data wise didn't count if the
files were bigger) were copied. currently df shows
rootfs 5044188 4310972 476980 91% / <= file
system that is 'full'
/dev/root 5044188 4310972 476980 91% /
rc-svcdir 1024 132 892 13% /lib/rc/init.d
udev 10240 68 10172 1% /dev
cachedir 4096 4 4092 1% /lib/splash/cache
none 776940 4 776936 1% /dev/shm
/dev/hda3 24416008 22546040 1869968 93% /opt
/dev/hda6 11560764 9300396 1673112 85% /home
tmpfs 776940 1728 775212 1% /tmp
tmpfs 1331200 36 1331164 1% /var/tmp
tmpfs 11560764 9300396 1673112 85% /usr/portage/distfiles
i like to use that last 470MB if possible
i also inclunded my tune2fs info
fsck didn't help (tried that)
On Wed, Jan 21, 2009 at 03:19:28PM +0100, Victor Pelt wrote:
> Hey,
>
> i tried to switch my root partition to ext4 (from reiserfs) but i ran
> into problems when copying over all the data that it says 'no space
> left on the device'
> i'm running 2.6.28-gentoo-r1 (based on 2.6.28.1 if i'm not mistaken)
>
> rootfs 5044188 4310972 476980 91% / <= file
> system that is 'full'
>
> i like to use that last 470MB if possible
>
> fsck didn't help (tried that)
If this is the problem I think it is, it's fixed in patches that were
merged for 2.6.29 (in -rc1), and you can work around it via umounting
and remounting the filesystem (or in case of the root filesystem,
rebooting).
However, when you said "fsck didn't help", what exactly did you mean
by that? That fsck didn't show any problems, when you ran it while
the filesystem was mounted? Did you actually reboot and force an
fsck, and then found that you were still getting ENOSPC errors?
I assume you have tried writing as root, and that quotas are disabled?
- Ted
P.S. If it is the problem I think it is, upgrading 2.6.29-rcX should
fix the problem, as would using the ext4-stable branch in the ext4 git
tree, or applying the patches in the for-stable branch of the ext4 git
tree:
http://git.kernel.org/?p=linux/kernel/git/tytso/ext4.git
i set the number mounted counter to 31, which forced fsck to run on
the next reboot. I rebooted, fsck didn't show any erros, but i still
got the no space left error when i tried to copy files to the
partition.
i'll try the latest git kernel though.
On Wed, Jan 21, 2009 at 3:55 PM, Theodore Tso <[email protected]> wrote:
> On Wed, Jan 21, 2009 at 03:19:28PM +0100, Victor Pelt wrote:
>> Hey,
>>
>> i tried to switch my root partition to ext4 (from reiserfs) but i ran
>> into problems when copying over all the data that it says 'no space
>> left on the device'
>> i'm running 2.6.28-gentoo-r1 (based on 2.6.28.1 if i'm not mistaken)
>>
>> rootfs 5044188 4310972 476980 91% / <= file
>> system that is 'full'
>>
>> i like to use that last 470MB if possible
>>
>> fsck didn't help (tried that)
>
> If this is the problem I think it is, it's fixed in patches that were
> merged for 2.6.29 (in -rc1), and you can work around it via umounting
> and remounting the filesystem (or in case of the root filesystem,
> rebooting).
>
> However, when you said "fsck didn't help", what exactly did you mean
> by that? That fsck didn't show any problems, when you ran it while
> the filesystem was mounted? Did you actually reboot and force an
> fsck, and then found that you were still getting ENOSPC errors?
>
> I assume you have tried writing as root, and that quotas are disabled?
>
> - Ted
>
> P.S. If it is the problem I think it is, upgrading 2.6.29-rcX should
> fix the problem, as would using the ext4-stable branch in the ext4 git
> tree, or applying the patches in the for-stable branch of the ext4 git
> tree:
>
> http://git.kernel.org/?p=linux/kernel/git/tytso/ext4.git
>
On Wed, Jan 21, 2009 at 04:18:35PM +0100, Victor Pelt wrote:
> i set the number mounted counter to 31, which forced fsck to run on
> the next reboot. I rebooted, fsck didn't show any erros, but i still
> got the no space left error when i tried to copy files to the
> partition.
OK, if a reboot didn't help, I'll bet I know what happened. E2fsprogs
changed the default default inode ratio, which means that number of
inodes being created is half what it was previously. If you do a "df
-i", you'll probably see that you have exhausted the number of inodes
in the filesystem.
The default inode ratio controlled by /etc/mke2fs.conf, and is
currently 16k. That is, it assumes the average size of files on the
filesystem is at least 16k. (It previously was 8k.) For large
filesystems, this is not a problem; I'm guessing that you have a small
root filesystem, and probably are using a hard-coded /dev partition,
so the large number of (zero-length) device files is throwing off the
average. If you recreate the filesystem with mke2fs -I 8192, it
should allow you to copy over all of your files in the root
filesystem.
Finally, note that we made this change for all ext2/3/4 filesystems,
so this is not unique to ext4; it's just that you reformatted your
root filesystem for the first time since upgrading to a newer
e2fsprogs with the changed default, and you ran into this problem. If
there are enough people who are using small root filesystems, maybe
we'll need to have some adjusted hueristics. Right now we have
"floppy" for filesystems less than 3 megs, "small" for filesystems
less than 512 megs, and every thing else is default. Maybe we need to
have a "medium" for filesystems smaller than 10 gigs, perhaps, and use
a default inode ratio of 8192 for medium-sized filesystems....
- Ted
upgrading to 2.6.29-rc2-git1 did allow me to use most of the space but
i still can't quite use all of it. maybe this is to be expected.
(note: i'm just trying testing now, everything i used to have on my
old partition fits. i'm just trying to understand what exactly is
'full'. it seems the file system is 'holding something back'
i included a dumo of df, df -i and tune2fs -l /dev/hda2
(and partition that has ext4 is the / partition or /dev/hda2)
victor
On Wed, Jan 21, 2009 at 6:04 PM, Theodore Tso <[email protected]> wrote:
> On Wed, Jan 21, 2009 at 04:18:35PM +0100, Victor Pelt wrote:
>> i set the number mounted counter to 31, which forced fsck to run on
>> the next reboot. I rebooted, fsck didn't show any erros, but i still
>> got the no space left error when i tried to copy files to the
>> partition.
>
> OK, if a reboot didn't help, I'll bet I know what happened. E2fsprogs
> changed the default default inode ratio, which means that number of
> inodes being created is half what it was previously. If you do a "df
> -i", you'll probably see that you have exhausted the number of inodes
> in the filesystem.
>
> The default inode ratio controlled by /etc/mke2fs.conf, and is
> currently 16k. That is, it assumes the average size of files on the
> filesystem is at least 16k. (It previously was 8k.) For large
> filesystems, this is not a problem; I'm guessing that you have a small
> root filesystem, and probably are using a hard-coded /dev partition,
> so the large number of (zero-length) device files is throwing off the
> average. If you recreate the filesystem with mke2fs -I 8192, it
> should allow you to copy over all of your files in the root
> filesystem.
>
> Finally, note that we made this change for all ext2/3/4 filesystems,
> so this is not unique to ext4; it's just that you reformatted your
> root filesystem for the first time since upgrading to a newer
> e2fsprogs with the changed default, and you ran into this problem. If
> there are enough people who are using small root filesystems, maybe
> we'll need to have some adjusted hueristics. Right now we have
> "floppy" for filesystems less than 3 megs, "small" for filesystems
> less than 512 megs, and every thing else is default. Maybe we need to
> have a "medium" for filesystems smaller than 10 gigs, perhaps, and use
> a default inode ratio of 8192 for medium-sized filesystems....
>
> - Ted
>
On Wed, Jan 21, 2009 at 07:15:26PM +0100, Victor Pelt wrote:
> upgrading to 2.6.29-rc2-git1 did allow me to use most of the space but
> i still can't quite use all of it. maybe this is to be expected.
No, it's not to be expected. You should be able to use everything up
to 100% of what is reported by df. There is the 5% reserved blocks
which only root is allowed to use, but df takes that into account. So
if you can't use that last 62944k reported by df after a reboot, and
df -i reports enough inodes, that's definitely a bug in ext4.
Would you be willing to send me a compressed raw e2image file? You
can generate it as follows:
e2image -r /dev/hda2 - | bzip2 > /tmp/hda2.e2i.bz2
Ideally this should be done while the filesystem is unmounted, or
mounted read-only.
It will give me a copy of the filesystem metadata, but no data blocks.
That will allow me to try to replicate the problem on my system. It
will probably be still too large e-mail, so if you can put it up on
some web or ftp server, and send me the URL, that would be best.
Thanks!!
- Ted
On Wed 2009-01-21 12:04:23, Theodore Tso wrote:
> On Wed, Jan 21, 2009 at 04:18:35PM +0100, Victor Pelt wrote:
> > i set the number mounted counter to 31, which forced fsck to run on
> > the next reboot. I rebooted, fsck didn't show any erros, but i still
> > got the no space left error when i tried to copy files to the
> > partition.
>
> OK, if a reboot didn't help, I'll bet I know what happened. E2fsprogs
> changed the default default inode ratio, which means that number of
> inodes being created is half what it was previously. If you do a "df
> -i", you'll probably see that you have exhausted the number of inodes
> in the filesystem.
Actually it would be cool if df warned about low inode free space by
default. "df -i" is an old trap, but still effective to these days
:-).
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html