hi,
ext2simg.c file was moved from android (system/extras) into e2fsprogs.
however it seems that it cannot be compiled outside of android (it
uses sparse/sparse.h). some distro used to build/package this utility
(e.g. debian has it in android-tools-fsutils).
what was the main reason why it was moved into e2fsprogs in the first
place, given that it cannot be compiled (easily) outside of AOSP?
Also, the commit that brought this file has the following reference:
From AOSP commit: c1b7d19958dc3dbe53810811ea3dcc4f04f85c73
However I don't think this commit in AOSP.
thanks!
On Sun, Oct 08, 2017 at 10:04:21PM +0200, Nicolas Dechesne wrote:
> hi,
>
> ext2simg.c file was moved from android (system/extras) into e2fsprogs.
> however it seems that it cannot be compiled outside of android (it
> uses sparse/sparse.h). some distro used to build/package this utility
> (e.g. debian has it in android-tools-fsutils).
>
> what was the main reason why it was moved into e2fsprogs in the first
> place, given that it cannot be compiled (easily) outside of AOSP?
I've been pushing a long-term effort to try to rid the world from
make_ext4fs, which only exists at all because there was a time when
there were people who greatly feared the GPLv2. So the idea was that
AOSP would never use any GPLv2 code, and AOSP would use make_ext4fs,
and Android devices would *never* need to do silly things like run
e2fsck. (This was a long time ago, but we've been still paying for
that design decision, with interest.)
Unfortunately, make_ext4fs was a clean room implementation of mke2fs,
which means it had gotten a number of things wrong. Worse, it was
originally meant to work (only) as part of the build process using
sparse files or later, libsparse, and so it assumes that an unwritten
block is all zeros.
This even mostly worked on flash devices, assuming that the device
actually bothered to honor the discard/trim "hint" --- but it was a
complete nightmare for devices using dm-crypt, since the trim/discard
would result in the underlying blocks to be zero, but then when
dm-crypt read back the blocks that were supposed to be zeroed, the
decryption would cause the block to have some random data. Hilarity
would then ensue --- but only in very random, edge cases.
I've lost track of the number of times I've been asked to assist when
make_ext4fs has screwed up in some wonderous fashion, and there has
been some internal bug, usually but not always involving a
user-initiated factory reset, that had stumped the good folks on the
Android team, and then I would get asked to assist.
Unfortunately, make_ext4fs and its associated tools have accumulated a
number of extra functionality (populating the empty file system,
adding SELinux attributes, etc.), so trying to make replacements that
don't depend on a clean-room implementation of e2fsprogs and its
library has been a long, and not completely finished, journey. Still,
thanks to a summer intern last year, and some assistance from folks in
the Android team, we're almost there.
Anyway, the version of ext2simg in e2fsprogs uses the ext2fs library,
and since the cutover from make_ext4fs to mke2fs has been done on a
device by device basis (to minimize the risks of breakage), it was
simpler to make the e2fsprogs-dependent version of ext2simg live in
the e2fsprogs git tree.
As far as why it's in the upstream e2fsprogs tree, I've been making an
effort to keep the upstream and AOSP versions of e2fsprogs more
closely aligned, so it's easier for AOSP to update to a newer version
of e2fsprogs.
You can now upgrade to a newer version of e2fsprogs by doing a "git
merge", which used to not work at all. Now, it might result in some
merge conflicts, but at least there is a common git ancestor so the
merge has a chance of working.
As far as it being hard to compile libsparse outside of AOSP, it would
be great if someone were to package libsparse and libsparse-dev for
Debian. If they do so, I'll be glad (as debian maintainer) to package
up the e2fsprogs variants of these tools. We'll probably need to give
them a different package name so that both versions of the packages
can exist at least in the package namespace, if not installed at the
same time on a Debian system. But we can deal with that problem when
we get there.
> Also, the commit that brought this file has the following reference:
>
> From AOSP commit: c1b7d19958dc3dbe53810811ea3dcc4f04f85c73
>
> However I don't think this commit in AOSP.
Hmm... I don't have easy access to AOSP at the moment, since I'm at
home and it's not on my upstream development laptop, but is it
possible that the contents of the git commit exists, but with a
different git commit ID? My understanding is that changes to
external/e2fsprogs, once code reviewed in Gerrit, do get pushed out to
the AOSP master branch right away. Android source trees have a
massively complex merging scheme, and the most likely explanation is
that I used the git commit ID belonging to some merge into a tree that
isn't published in AOSP. Either that, or the commit ID got somehow
corrupted while it was in my editor doing the cherry-pick into the
upstream tree.
Cheers,
- Ted
P.S. One of the other really nice things you can do as a result of a
lot of this effort is you can now build tools such as debugfs for
Android, built against bionic. So it becomes a lot easier to build a
userdebug android image that includes debugfs, which is helpful if you
need to try to debug some interesting new failure.
And using mke2fs instead of make_ext4fs means that if we do add some
new file system feature into ext4/e2fsprogs which Android wants to
depend upon, it will be a lot easier to keep things in sync with the
external upstream world.
hi Theo,
thanks for the very detailed answer!
On Sun, Oct 8, 2017 at 11:27 PM, Theodore Ts'o <[email protected]> wrote:
> On Sun, Oct 08, 2017 at 10:04:21PM +0200, Nicolas Dechesne wrote:
>> hi,
>>
>> ext2simg.c file was moved from android (system/extras) into e2fsprogs.
>> however it seems that it cannot be compiled outside of android (it
>> uses sparse/sparse.h). some distro used to build/package this utility
>> (e.g. debian has it in android-tools-fsutils).
>>
>> what was the main reason why it was moved into e2fsprogs in the first
>> place, given that it cannot be compiled (easily) outside of AOSP?
>
> I've been pushing a long-term effort to try to rid the world from
> make_ext4fs, which only exists at all because there was a time when
> there were people who greatly feared the GPLv2. So the idea was that
> AOSP would never use any GPLv2 code, and AOSP would use make_ext4fs,
> and Android devices would *never* need to do silly things like run
> e2fsck. (This was a long time ago, but we've been still paying for
> that design decision, with interest.)
>
> Unfortunately, make_ext4fs was a clean room implementation of mke2fs,
> which means it had gotten a number of things wrong. Worse, it was
> originally meant to work (only) as part of the build process using
> sparse files or later, libsparse, and so it assumes that an unwritten
> block is all zeros.
>
> This even mostly worked on flash devices, assuming that the device
> actually bothered to honor the discard/trim "hint" --- but it was a
> complete nightmare for devices using dm-crypt, since the trim/discard
> would result in the underlying blocks to be zero, but then when
> dm-crypt read back the blocks that were supposed to be zeroed, the
> decryption would cause the block to have some random data. Hilarity
> would then ensue --- but only in very random, edge cases.
>
> I've lost track of the number of times I've been asked to assist when
> make_ext4fs has screwed up in some wonderous fashion, and there has
> been some internal bug, usually but not always involving a
> user-initiated factory reset, that had stumped the good folks on the
> Android team, and then I would get asked to assist.
>
> Unfortunately, make_ext4fs and its associated tools have accumulated a
> number of extra functionality (populating the empty file system,
> adding SELinux attributes, etc.), so trying to make replacements that
> don't depend on a clean-room implementation of e2fsprogs and its
> library has been a long, and not completely finished, journey. Still,
> thanks to a summer intern last year, and some assistance from folks in
> the Android team, we're almost there.
>
> Anyway, the version of ext2simg in e2fsprogs uses the ext2fs library,
> and since the cutover from make_ext4fs to mke2fs has been done on a
> device by device basis (to minimize the risks of breakage), it was
> simpler to make the e2fsprogs-dependent version of ext2simg live in
> the e2fsprogs git tree.
>
> As far as why it's in the upstream e2fsprogs tree, I've been making an
> effort to keep the upstream and AOSP versions of e2fsprogs more
> closely aligned, so it's easier for AOSP to update to a newer version
> of e2fsprogs.
>
> You can now upgrade to a newer version of e2fsprogs by doing a "git
> merge", which used to not work at all. Now, it might result in some
> merge conflicts, but at least there is a common git ancestor so the
> merge has a chance of working.
I agree that getting ext2simg closer to upstream and ext2fs was the
right thing to do, i have faced issues with make_ext4fs as well, so I
am aware of what you are discussing here!
>
> As far as it being hard to compile libsparse outside of AOSP, it would
> be great if someone were to package libsparse and libsparse-dev for
> Debian. If they do so, I'll be glad (as debian maintainer) to package
> up the e2fsprogs variants of these tools. We'll probably need to give
> them a different package name so that both versions of the packages
> can exist at least in the package namespace, if not installed at the
> same time on a Debian system. But we can deal with that problem when
> we get there.
libsparse is already available in Debian, in android-libsparse-dev
package, but it's being packaged from the full system/core, not
standalone.
In fact, building the 'new' ext2simg is exactly what I needed to do,
and the reason why i started to investigate what happened to
ext2simg.. the real reason why i've been looking into that , is that I
have noticed that my file system gets corrupted when doing ext2simg ->
simg2img. we are building ext4 file system for small dev boards that
are meant to be flashed with fastboot, so our workflow is:
mkfs.ext4 xxx
-> fill in the image
ext2simg
What i've noticed is that if we upgrade our servers running Debian
Jessie , this no longer works and after doing simg2img (or flashing
with fasboot) the files in the file system are corrupted (md5sum
differ).
The problems appears as soon as I upgrade from e2fsprogs 1.42.12-2+b1
(the version in Jessie) to either the version in Stretch or the
version in jessie-backport. So it looks like e2fsprogrs 1.43, breaks
something with existing ext2simg (I am using ext2simg from
android-tools-fsutils, so the 'old' one).
if i use img2simg, then of course, there is no corruption.
>
>> Also, the commit that brought this file has the following reference:
>>
>> From AOSP commit: c1b7d19958dc3dbe53810811ea3dcc4f04f85c73
>>
>> However I don't think this commit in AOSP.
>
> Hmm... I don't have easy access to AOSP at the moment, since I'm at
> home and it's not on my upstream development laptop, but is it
> possible that the contents of the git commit exists, but with a
> different git commit ID? My understanding is that changes to
> external/e2fsprogs, once code reviewed in Gerrit, do get pushed out to
> the AOSP master branch right away. Android source trees have a
> massively complex merging scheme, and the most likely explanation is
> that I used the git commit ID belonging to some merge into a tree that
> isn't published in AOSP. Either that, or the commit ID got somehow
> corrupted while it was in my editor doing the cherry-pick into the
> upstream tree.
argh. too bad, we lost the back link..
>
> Cheers,
>
> - Ted
>
> P.S. One of the other really nice things you can do as a result of a
> lot of this effort is you can now build tools such as debugfs for
> Android, built against bionic. So it becomes a lot easier to build a
> userdebug android image that includes debugfs, which is helpful if you
> need to try to debug some interesting new failure.
>
> And using mke2fs instead of make_ext4fs means that if we do add some
> new file system feature into ext4/e2fsprogs which Android wants to
> depend upon, it will be a lot easier to keep things in sync with the
> external upstream world.
[1] https://packages.debian.org/sid/android-libsparse
On Mon, Oct 09, 2017 at 10:24:46AM +0200, Nicolas Dechesne wrote:
>
> The problems appears as soon as I upgrade from e2fsprogs 1.42.12-2+b1
> (the version in Jessie) to either the version in Stretch or the
> version in jessie-backport. So it looks like e2fsprogrs 1.43, breaks
> something with existing ext2simg (I am using ext2simg from
> android-tools-fsutils, so the 'old' one).
So *that* shouldn't have happened. I do try to keep the ABI/API of
the libext2fs shared library stable, even across major version
numbers. That being said, I suspect ext2simg was doing something that
made it highly dependent on the internal implementation detalis of
libext2fs. It would be interesting to know what in the world that
was, so I can try to avoid that in the future. But I've always viewed
many of these tools to be quite highly fragile, so I'm not sure
there's much we can do to fix this in terms of changing what libext2fs
is doing. But if I can fix/improve the backwards compatibility
problem from libext2fs's side, I'm certainly willing to entertain that
notion.
- Ted