Hello.
I often create a Live CD.
When creating a Live CD, loopback mounted ext3 image files are used.
These image files are then compressed so that the ISO image will fit within
a CD-R's capacity (i.e. 700MB).
Compressing whole image files includes compressing deleted/unused bytes within
a block. This means that non-zero bytes in deleted/unused blocks affect
compression ratio.
I'm using the below program to improve compression ratio.
----------
#define _FILE_OFFSET_BITS 64
#define _LARGEFILE_SOURCE
#define _LARGEFILE64_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
int main(int argc, char *argv[]) {
int fd;
static char buffer[4096];
memset(buffer, 0, sizeof(buffer));
snprintf(buffer, sizeof(buffer) - 1, "%s/XXXXXX", argc > 1 ? argv[1] : "");
if ((fd = mkstemp(buffer)) != EOF) {
unlink(buffer);
memset(buffer, 255, sizeof(buffer));
while (write(fd, buffer, sizeof(buffer)) > 0);
sync();
close(fd);
}
return 0;
}
----------
But this program cannot clear some bytes within a block for file data (e.g.
offset from 1 to 511 of a data block used by a file with only 1 byte data)
and blocks for directory entries.
Any suggestions?
Regards.
On Nov 22, 2008 12:36 +0900, Tetsuo Handa wrote:
> Compressing whole image files includes compressing deleted/unused bytes within
> a block. This means that non-zero bytes in deleted/unused blocks affect
> compression ratio.
>
> static char buffer[4096];
> memset(buffer, 0, sizeof(buffer));
> snprintf(buffer, sizeof(buffer) - 1, "%s/XXXXXX", argc > 1 ? argv[1] : "");
> if ((fd = mkstemp(buffer)) != EOF) {
> unlink(buffer);
> memset(buffer, 255, sizeof(buffer));
> while (write(fd, buffer, sizeof(buffer)) > 0);
Why would you fill the buffer with 0xff instead of 0?
In fact no such program is needed, just "dd if=/dev/zero of=/{fs}/tmp"
and then delete the file.
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.
Hello.
Andreas Dilger wrote:
> Why would you fill the buffer with 0xff instead of 0?
> In fact no such program is needed, just "dd if=/dev/zero of=/{fs}/tmp"
> and then delete the file.
>
To avoid that the /{fs}/tmp is created as a sparse file.
I want to clear not only not-in-use blocks but also in-use blocks,
while keeping files in /{fs}/tmp intact.
Regards.
Tetsuo Handa wrote:
> Hello.
>
> Andreas Dilger wrote:
>> Why would you fill the buffer with 0xff instead of 0?
>> In fact no such program is needed, just "dd if=/dev/zero of=/{fs}/tmp"
>> and then delete the file.
>>
> To avoid that the /{fs}/tmp is created as a sparse file.
>
Most filesystems will not create a sparse file if zero-byte filled
blocks are written. To create a sparse file you normally have to seek
beyond the file end and then write blocks, leaving a hole in-between the
positions.
The information from stat can tell you if a file has been stored
sparsely, because the blocks used count will be less than the file size
suggests.
For example:
$ dd if=/dev/zero of=small-file bs=512 count=16
16+0 records in
16+0 records out
8192 bytes (8.2 kB) copied, 0.000187744 seconds, 43.6 MB/s
phillip@dylan:/tmp$ stat small-file
File: `small-file'
Size: 8192 Blocks: 16 IO Block: 4096 regular file
Device: 801h/2049d Inode: 847831 Links: 1
The zero filled file uses 16 blocks (16 * 512 bytes = 8K), and so we
know it isn't sparsely stored.
You can get dd to seek a number of blocks into the output file before
writing, this will create a sparse file...
$ dd if=/dev/zero of=small-file bs=512 count=16 seek=16
16+0 records in
16+0 records out
8192 bytes (8.2 kB) copied, 0.000212609 seconds, 38.5 MB/s
phillip@dylan:/tmp$ stat small-file
File: `small-file'
Size: 16384 Blocks: 16 IO Block: 4096 regular file
Device: 801h/2049d Inode: 847831 Links: 1
A 16K file is only using 16 blocks, and so 8K is stored sparsely.
Phillip
Tetsuo Handa wrote:
> Hello.
>
> I often create a Live CD.
> When creating a Live CD, loopback mounted ext3 image files are used.
> These image files are then compressed so that the ISO image will fit within
> a CD-R's capacity (i.e. 700MB).
>
You don't mention what you're using to compress the ext3 image file. If
you're using Squashfs then it's much better to zero-fill the blocks
rather than use 255, because Squashfs detects zero-filled blocks and
stores them sparsely. This not only gets slightly better compression
(than a compressed zero-filled block), but reading is also slightly faster.
Phillip
Hello.
Phillip Lougher wrote:
> Most filesystems will not create a sparse file if zero-byte filled
> blocks are written. To create a sparse file you normally have to seek
> beyond the file end and then write blocks, leaving a hole in-between the
> positions.
>
> The information from stat can tell you if a file has been stored
> sparsely, because the blocks used count will be less than the file size
> suggests.
>
Then, I can use "dd if=/dev/zero" without worrying sparse files.
Phillip Lougher wrote:
> You don't mention what you're using to compress the ext3 image file. If
> you're using Squashfs then it's much better to zero-fill the blocks
> rather than use 255, because Squashfs detects zero-filled blocks and
> stores them sparsely. This not only gets slightly better compression
> (than a compressed zero-filled block), but reading is also slightly faster.
>
Yes, I use squashfs.
What I wanted to do is "Zero-clearing *all zero-clearable* bytes".
Suppose blocks for inode are 8192 bytes and only 128 bytes are in use,
I think there are 8064 bytes I can zero-fill.
Suppose blocks for file data are 8192 bytes and a file's size is 1 byte,
I think there are 8191 bytes I can zero-fill.
Such bytes may have non-zero values because directory entries can be
unlink()ed and file data can be truncate()d.
I wished there is a utility to zero-fill such bytes.
Thanks a lot.
On Sun, 23 Nov 2008, Tetsuo Handa wrote:
> What I wanted to do is "Zero-clearing *all zero-clearable* bytes".
Other than acessing the fs directly with something fs-specific that knows
how to do it, the following trick comes to mind:
1. compress all files to a tar.bz2.
2. remove all files
3. zero fs using dd to a file (will zero all blocks except for the ones used
by the tar.bz2
4. unpack tar.bz2
5. remove tar.bz2
6. redo the dd trick. This will now zero all blocks that were in use by
the tar.bz2.
Of course, this only works if the (kernel, glibc, tar) are not writing
random junk to the unused parts of a fs block.
The cost is from 1 to 3 rm'ed inodes left behind. If you use two
filesystems (i.e. tar to outside the filesystem), you avoid that
possibility.
> I wished there is a utility to zero-fill such bytes.
So do I. And a IOCTL/syscall/whatever that we could use to sanitize (i.e.
fill with an user-supplied byte) half-used blocks, so that we could have it
on the most used filesystems (ext2, ext3, xfs, reiser...), and that we could
implement scrub-erasing of unused filesystem areas properly.
--
"One disk to rule them all, One disk to find them. One disk to bring
them all and in the darkness grind them. In the Land of Redmond
where the shadows lie." -- The Silicon Valley Tarot
Henrique Holschuh
Hi!
Am Sonntag 23 November 2008 schrieben Sie:
> On Sun, 23 Nov 2008, Tetsuo Handa wrote:
> > What I wanted to do is "Zero-clearing *all zero-clearable* bytes".
> Other than acessing the fs directly with something fs-specific that
> knows how to do it, the following trick comes to mind:
>
> 1. compress all files to a tar.bz2.
> 2. remove all files
> 3. zero fs using dd to a file (will zero all blocks except for the ones
> used by the tar.bz2
> 4. unpack tar.bz2
> 5. remove tar.bz2
> 6. redo the dd trick. This will now zero all blocks that were in use by
> the tar.bz2.
>
> Of course, this only works if the (kernel, glibc, tar) are not writing
> random junk to the unused parts of a fs block.
>
> The cost is from 1 to 3 rm'ed inodes left behind. If you use two
> filesystems (i.e. tar to outside the filesystem), you avoid that
> possibility.
>
> > I wished there is a utility to zero-fill such bytes.
>
> So do I. And a IOCTL/syscall/whatever that we could use to sanitize
> (i.e. fill with an user-supplied byte) half-used blocks, so that we
> could have it on the most used filesystems (ext2, ext3, xfs,
> reiser...), and that we could implement scrub-erasing of unused
> filesystem areas properly.
There are at least 2 tools - I think there was even another one. I did not
test them and do not know whether they work on singular inode basis.
Debian Package zerofree:
Description: zero free blocks from ext2/3 file-systems
Zerofree finds the unallocated, non-zeroed blocks in an ext2 or ext3
file-system and fills them with zeroes. This is useful if the device
on which this file-system resides is a disk image. In this case,
depending on the type of disk image, a secondary utility may be able
to reduce the size of the disk image after zerofree has been
run. Zerofree requires the file-system to be unmounted or mounted
read-only.
.
The usual way to achieve the same result (zeroing the unallocated
blocks) is to run "dd" do create a file full of zeroes that takes up
the entire free space on the drive, and then delete this file. This
has many disadvantages, which zerofree alleviates:
* it is slow;
* it makes the disk image (temporarily) grow to its maximal extent;
* it (temporarily) uses all free space on the disk, so other
concurrent write actions may fail.
.
Zerofree has been written to be run from GNU/Linux systems installed
as guest OSes inside a virtual machine. If this is not your case, you
almost certainly don't need this package.
Debian Package wipe2fs:
Package: wipe2fs
Priority: extra
Section: admin
Installed-Size: 112
Maintainer: Martin A. Godisch <[email protected]>
Architecture: i386
Version: 0.2.1-1
Depends: e2fslibs, libc6 (>= 2.7-1), libcomerr2 (>= 1.33-3)
Filename: pool/main/w/wipe2fs/wipe2fs_0.2.1-1_i386.deb
Size: 14150
MD5sum: da0c0d4319fda08a1f5fc7a92f5b2535
SHA1: 2f7d55fbfda34b328b67f17e5e2e26d8f853bdac
SHA256: 569bc452331b1067f49ba5aaf36dd7bb357036307b79156c4d7c5beaa2a17545
Description: Securely wipe unused space in ext2/3 filesystems
This users-pace program locates unused space in ext2/3 filesystems
and overwrites the space with zero. wipe2fs also reaches the slack
space at the end of files, it does not require kernel filesystem
support.
Homepage: http://web.cecs.pdx.edu/~cklin/wipe2fs/
Ciao,
--
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7
Hello.
Martin Steigerwald wrote:
> There are at least 2 tools - I think there was even another one. I did not
> test them and do not know whether they work on singular inode basis.
> Debian Package zerofree:
> Debian Package wipe2fs:
Great!
They must be what I wished.
Thank you.